Three-dimensional display method and device, electronic equipment and storage medium

Abstract

The application provides a three-dimensional display method and device, electronic equipment and storage medium, and relates to the technical field of virtual reality display, and the method comprises the steps of collecting real scene information behind a three-dimensional display; performing three-dimensional reconstruction on a real scene behind the three-dimensional display based on the real scene information behind the three-dimensional display, to obtain a real scene model and a virtual reality model of the real scene behind the three-dimensional display; processing the real scene model and the virtual reality model of the real scene behind the three-dimensional display based on parameters of the three-dimensional display, to obtain a dense view point image sequence; performing multi-view point coding on the dense view point image sequence, to obtain a coded image; and loading the coded image onto the three-dimensional display for three-dimensional display. The application can display stereoscopic images of the real scene model and the virtual reality model simultaneously, and realize background transparency, thereby improving the sense of immersion.

Description

Technical Field

[0001] This invention relates to the field of virtual reality display technology, and in particular to a three-dimensional display method, device, electronic device, and storage medium. Background Technology

[0002] 3D display is a fascinating cutting-edge technology field today. 3D display systems with full parallax, multi-angle capabilities, and the ability for multiple viewers to view simultaneously can be applied in military, advertising, and medical fields, giving viewers the feeling of viewing realistic scenes.

[0003] Currently, virtual reality display technology typically renders directly in space, failing to display the scene information behind the 3D display and thus hindering the audience's immersion. Summary of the Invention

[0004] This invention provides a three-dimensional display method, device, electronic device, and storage medium that can simultaneously display stereoscopic images of a real scene model and a virtual reality model, and achieve background transparency, thereby enhancing the sense of immersion.

[0005] This invention provides a three-dimensional display method, comprising:

[0006] Acquire information about the real scene behind the 3D display;

[0007] Based on the real scene information behind the 3D display, a 3D reconstruction of the real scene behind the 3D display is performed to obtain a real scene model and a virtual reality model behind the 3D display.

[0008] Based on the parameters of the 3D display, the real scene model and virtual reality model behind the 3D display are processed to obtain a dense viewpoint image sequence.

[0009] The dense viewpoint image sequence is multi-viewpoint encoded to obtain an encoded map;

[0010] The encoded image is loaded onto the 3D display for 3D display.

[0011] According to a three-dimensional display method provided by the present invention, the background real scene information includes the image coordinates of objects in the background real scene and the depth distribution information of the background real scene;

[0012] The step of performing 3D reconstruction of the real-world scene behind the 3D display based on the real-world scene information to obtain a real-world scene model and a virtual reality model behind the 3D display includes:

[0013] Based on the aforementioned real-world scene information, the model of the underlying real-world scene is reconstructed;

[0014] Based on the depth distribution information of the real scene behind it, a first virtual reality model is constructed;

[0015] Convert the image coordinates of the real-world objects in the background to the world coordinates of the real-world objects in the background.

[0016] Adjust the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the real scene object behind it to obtain the second virtual reality model;

[0017] The scale of the second virtual reality model is adjusted according to the scaling factor so that it fits the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in virtual space.

[0018] According to a three-dimensional display method provided by the present invention, the scaling factor is the product of the scaling factor coefficient correction value and the area ratio, and the area ratio is the ratio between the area occupied by the real scene object behind the three-dimensional display in the captured image and the area of ​​the captured image.

[0019] The step of adjusting the scale of the second virtual reality model according to a scaling factor to make it fit the real scene objects behind the 3D display, and obtaining the world coordinates of the virtual reality model in virtual space, includes:

[0020] The scaling factor is adjusted by continuously adjusting the scaling factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scaling factor until it fits the real scene objects behind the three-dimensional display, so as to obtain the world coordinates of the virtual reality model in the virtual space.

[0021] According to a three-dimensional display method provided by the present invention, the step of adjusting the scale factor by continuously adjusting the scale factor coefficient correction value, and adjusting the scale of the second virtual reality model according to the adjusted scale factor until it fits the real scene objects behind the three-dimensional display, includes:

[0022] The following steps are repeated until the second virtual reality model in the 3D display is aligned with the real-world objects behind the 3D display:

[0023] The second virtual reality model in the 3D display and the real scene objects behind the 3D display are captured simultaneously by a binocular monitoring camera.

[0024] The coordinates of the first checkerboard corner points of the second virtual reality model in the 3D display and the coordinates of the second checkerboard corner points of the real scene objects behind the 3D display are detected.

[0025] Determine whether the coordinates of the first chessboard corner point and the coordinates of the second chessboard corner point form a continuous straight line. If so, determine that the second virtual reality model in the 3D display is aligned with the real scene object behind the 3D display. Otherwise, adjust the scale factor coefficient correction value to adjust the scale factor, and adjust the scale of the second virtual reality model according to the adjusted scale factor.

[0026] According to a three-dimensional display method provided by the present invention, the parameters of the three-dimensional display include the actual size of the three-dimensional display, the viewing distance, and the display angle;

[0027] The process, based on the parameters of the 3D display, involves processing the real-world scene model and virtual reality model behind the 3D display to obtain a dense viewpoint image sequence, including:

[0028] The central coordinate position of the three-dimensional display in the world coordinate system is determined as the dense viewpoint shooting center;

[0029] Determine the mapping relationship between the actual size, viewing distance, and display angle of the 3D display and the dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in virtual space.

[0030] The number of viewpoints generated, the field of view, the spacing, and the pose are determined based on the mapping relationship; the spacing is equal to the ratio between the field of view and the number of viewpoints generated.

[0031] Based on the dense viewpoint shooting center, the real scene model and virtual reality model behind the 3D display are processed to generate the dense viewpoint image sequence that matches the number of viewpoints generated, the field of view, the spacing, and the pose.

[0032] According to a three-dimensional display method provided by the present invention, the step of acquiring information about the real scene behind the three-dimensional display includes:

[0033] The camera array captures information about the real scene behind the 3D display.

[0034] The step of converting the image coordinates of the real scene object behind the camera into the world coordinates of the real scene object behind the camera includes: taking the geometric center of the camera array as the origin of the world coordinate system, and converting the image coordinates of the real scene object behind the camera into the world coordinates of the real scene object behind the camera.

[0035] According to a three-dimensional display method provided by the present invention, the step of acquiring information about the real scene behind the three-dimensional display includes:

[0036] The real scene information behind the 3D display is captured by a single camera;

[0037] The step of converting the image coordinates of the real scene objects behind the camera into the world coordinates of the real scene objects behind the camera includes: taking the position of the single camera as the origin of the world coordinate system, and converting the image coordinates of the real scene objects behind the camera into the world coordinates of the real scene objects behind the camera.

[0038] The present invention also provides a three-dimensional display device, comprising:

[0039] The acquisition module is used to acquire information about the real scene behind the 3D display.

[0040] The reconstruction module is used to perform three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind it, so as to obtain the real scene model behind the three-dimensional display and the virtual reality model.

[0041] The processing module is used to process the real scene model and virtual reality model behind the 3D display based on the parameters of the 3D display to obtain a dense viewpoint image sequence.

[0042] The encoding module is used to perform multi-viewpoint encoding on the dense viewpoint image sequence to obtain an encoded map;

[0043] The display module is used to load the encoded image onto the 3D display for 3D display.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional display method as described above.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional display method as described above.

[0046] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional display method as described above.

[0047] The three-dimensional display method, apparatus, electronic device, and storage medium provided by this invention first acquire information about the real-world scene behind the three-dimensional display; based on this information, a three-dimensional reconstruction of the real-world scene behind the three-dimensional display is performed to obtain a real-world scene model and a virtual reality model; then, based on the parameters of the three-dimensional display, the real-world scene model and the virtual reality model are processed to obtain a dense viewpoint image sequence; the dense viewpoint image sequence is then multi-viewpoint encoded to obtain an encoded image; finally, the encoded image is loaded onto the three-dimensional display for three-dimensional display, allowing simultaneous display of stereoscopic images of the real-world scene model and the virtual reality model with a transparent background. Therefore, this invention can simultaneously display stereoscopic images of the real-world scene model and the virtual reality model, achieving background transparency and thus enhancing immersion. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the three-dimensional display method provided in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of a binocular monitoring camera provided in an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram showing how the coordinates of the first and second corner points of the chessboard form a continuous straight line, as provided in an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram showing that the coordinates of the first and second corner points of the chessboard grid do not form a continuous straight line, as provided in an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram of the structure of the three-dimensional display device provided in an embodiment of the present invention.

[0054] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The following is combined Figure 1 The three-dimensional display method of the present invention is described.

[0057] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the three-dimensional display method provided in an embodiment of the present invention. Figure 1 As shown, the method may include the following steps 101-105.

[0058] Step 101: Collect information about the real scene behind the 3D display.

[0059] Specifically, information about the real scene behind the 3D display can be acquired using a camera array or a single camera. The camera array can be a color camera array, the single camera can be a depth camera, and the acquisition method can be a time-series acquisition method; this embodiment is not limited to these.

[0060] Step 102: Based on the real scene information behind the 3D display, perform 3D reconstruction of the real scene behind the 3D display to obtain the real scene model behind the 3D display and the virtual reality model.

[0061] In one embodiment, the background real scene information includes the image coordinates of objects in the background real scene and the depth distribution information of the background real scene; step 102 includes steps 1021-1025.

[0062] Step 1021: Reconstruct the real scene model based on the real scene information.

[0063] Specifically, methods such as truncated signed distance function (TSDF), multi-view stereo (MVS), and neural radiance fields (NERF) can be used to reconstruct the real scene model based on the real scene information. This embodiment is not limited to these methods.

[0064] Step 1022: Construct the first virtual reality model based on the depth distribution information of the real scene behind it.

[0065] Specifically, a first virtual reality model is constructed based on the depth distribution information of the real scene behind it, and the occlusion and perspective relationships between objects in the scene behind it are restored.

[0066] Step 1023: Convert the image coordinates of the real scene objects behind the scenes into the world coordinates of the real scene objects behind the scenes.

[0067] In one embodiment, step 101 includes: acquiring information about the real scene behind the 3D display through a camera array; step 1023 includes: converting the image coordinates of the real scene objects behind the display into the world coordinates (x, y, z) of the real scene objects behind the display, with the geometric center of the camera array as the origin (0, 0, 0) of the world coordinate system.

[0068] In another embodiment, step 101 includes: acquiring information about the real scene behind the 3D display using a single camera; step 1023 includes: converting the image coordinates of the real scene objects behind the display into the world coordinates (x, y, z) of the real scene objects behind the display, with the position of the single camera as the origin (0, 0, 0) of the world coordinate system.

[0069] Step 1024: Adjust the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the real scene object behind it, to obtain the second virtual reality model.

[0070] Specifically, the rotation angle of the first virtual reality model is continuously adjusted until the geometric center of the rotated virtual reality model coincides with the geometric center of the real scene object behind it, thus obtaining the second virtual reality model.

[0071] Step 1025: Adjust the scale of the second virtual reality model according to the scaling factor so that it fits the real scene objects behind the 3D display, and obtain the world coordinates of the virtual reality model in the virtual space.

[0072] Optionally, the scaling factor is the product of the scaling factor coefficient correction value and the area ratio, where the area ratio is the ratio between the area occupied by the real scene object behind the 3D display in the captured image and the area of ​​the captured image; step 1025 includes: adjusting the scaling factor by continuously adjusting the scaling factor coefficient correction value, and adjusting the scale of the second virtual reality model according to the adjusted scaling factor until it fits the real scene object behind the 3D display, thereby obtaining the world coordinates of the virtual reality model in the virtual space.

[0073] For example, the scaling factor can be calculated using the expression Q = ΔQ * (S1 / S2), where Q represents the scaling factor, ΔQ represents the scaling factor coefficient correction value, the initial value of ΔQ is 1, S1 represents the area occupied by the real scene objects behind the 3D display in the captured image, and S2 represents the area of ​​the captured image. By continuously adjusting the scaling factor coefficient correction value ΔQ, the scaling factor Q is adjusted, and the scale of the second virtual reality model is adjusted according to the adjusted scaling factor Q until it fits perfectly with the real scene objects behind the 3D display, thus obtaining the world coordinates (x+Δx, y+Δy, z+Δz) of the virtual reality model in virtual space.

[0074] In one embodiment, the scaling factor is adjusted by continuously adjusting the scaling factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scaling factor until it fits the real scene objects behind the 3D display, including:

[0075] Repeat the following steps until the second virtual reality model in the 3D display is aligned with the real-world objects behind the 3D display: by means of... Figure 2 The binocular monitoring camera shown simultaneously captures images of a second virtual reality model in a 3D display and real-world objects behind the 3D display; it detects the coordinates of the first checkerboard corner points of the second virtual reality model in the 3D display and the coordinates of the second checkerboard corner points of the real-world objects behind the 3D display; it determines whether the coordinates of the first checkerboard corner points and the coordinates of the second checkerboard corner points form a continuous straight line; if so, ... Figure 3 As shown, this confirms that the second virtual reality model in the 3D display fits the real-world scene objects behind the 3D display; otherwise, as... Figure 4 As shown, the scaling factor coefficient correction value is adjusted to adjust the scaling factor, and the scaling factor of the second virtual reality model is adjusted according to the adjusted scaling factor.

[0076] Step 103: Based on the parameters of the 3D display, process the real scene model and virtual reality model behind the 3D display to obtain a dense viewpoint image sequence.

[0077] In one embodiment, the parameters of the 3D display include the actual size of the 3D display, viewing distance, and display angle, etc.; step 103 includes steps 1031-1034.

[0078] Step 1031: Determine the center of the dense viewpoint shooting location as the intermediate coordinate position of the 3D display in the world coordinate system.

[0079] Step 1032: Determine the mapping relationship between the actual size of the 3D display, viewing distance, display angle and dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in the virtual space.

[0080] Specifically, the mapping relationship between parameters such as the actual size of the 3D display, viewing distance, and display angle and the dense viewpoint generation location is derived.

[0081] Step 1033: Determine the number of viewpoints N, field of view fov, spacing d, and pose based on the mapping relationship; the spacing is equal to the ratio between the field of view and the number of viewpoints generated, i.e., d = fov / N.

[0082] Step 1034: Based on the dense viewpoint imaging center, process the real scene model and virtual reality model behind the 3D display to generate a dense viewpoint image sequence that matches the number of viewpoints generated, field of view, spacing, and pose. The dense viewpoint image sequence consists of multiple discrete viewpoint images.

[0083] Step 104: Perform multi-viewpoint coding on the dense viewpoint image sequence to obtain the coded map.

[0084] Specifically, multi-viewpoint coding of dense viewpoint image sequences can encode multiple discrete viewpoint maps into a single interleaved map, i.e., a coded map, that matches the parameters of a 3D display.

[0085] Step 105: Load the coded image onto the 3D display for 3D display.

[0086] Specifically, loading the encoded image onto a 3D display for 3D display can simultaneously achieve 3D effect display of the real scene model behind the transparent background and the virtual reality model, increasing immersion and making the transparency of the background more realistic.

[0087] The 3D display method provided in this invention first acquires information about the real-world scene behind the 3D display; based on this information, it performs 3D reconstruction of the real-world scene behind the 3D display to obtain a real-world scene model and a virtual reality model; then, based on the parameters of the 3D display, it processes the real-world scene model and the virtual reality model to obtain a dense viewpoint image sequence; it performs multi-viewpoint encoding on the dense viewpoint image sequence to obtain an encoded image; finally, it loads the encoded image onto the 3D display for 3D display, which can simultaneously display stereoscopic images of the real-world scene model and the virtual reality model with a transparent background. Therefore, this invention can simultaneously display stereoscopic images of the real-world scene model and the virtual reality model and achieve background transparency, thereby improving the sense of immersion.

[0088] The three-dimensional display device provided by the present invention will be described below. The three-dimensional display device described below and the three-dimensional display method described above can be referred to in correspondence.

[0089] Please refer to Figure 5 , Figure 5This is a structural schematic diagram of the three-dimensional display device provided in an embodiment of the present invention. Figure 5 As shown, the device may include:

[0090] Acquisition module 10 is used to acquire information about the real scene behind the 3D display;

[0091] The reconstruction module 20 is used to perform three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind it, so as to obtain the real scene model behind the three-dimensional display and the virtual reality model.

[0092] The processing module 30 is used to process the real scene model and virtual reality model behind the 3D display based on the parameters of the 3D display to obtain a dense viewpoint image sequence.

[0093] Encoding module 40 is used to perform multi-viewpoint encoding on dense viewpoint image sequences to obtain an encoded map;

[0094] Display module 50 is used to load the coded image onto a 3D display for 3D display.

[0095] In one embodiment, the background real scene information includes the image coordinates of objects in the background real scene and the depth distribution information of the background real scene;

[0096] Reconstruction module 20 includes:

[0097] The reconstruction unit is used to reconstruct the underlying real-scene model based on the underlying real-scene information.

[0098] The building unit is used to construct the first virtual reality model based on the depth distribution information of the underlying real scene;

[0099] The transformation unit is used to convert the image coordinates of objects in the real scene behind the object into the world coordinates of the objects in the real scene behind the object.

[0100] The first adjustment unit is used to adjust the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the real scene object behind it, thereby obtaining the second virtual reality model.

[0101] The second adjustment unit is used to adjust the scale of the second virtual reality model according to the scaling factor so that it fits the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space.

[0102] In one embodiment, the scaling factor is the product of the scaling factor coefficient correction value and the area ratio, where the area ratio is the ratio between the area occupied by the real scene object behind the 3D display in the captured image and the area of ​​the captured image; the second adjustment unit is specifically used for:

[0103] The scaling factor is adjusted by continuously adjusting the scaling factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scaling factor until it fits the real scene objects behind the 3D display, thus obtaining the world coordinates of the virtual reality model in the virtual space.

[0104] In one embodiment, the second adjustment unit is specifically used for:

[0105] Repeat the following steps until the second virtual reality model in the 3D display is aligned with the real-world objects behind the 3D display:

[0106] Simultaneously capture images of a second virtual reality model in a 3D display and real-world objects behind the 3D display using a binocular monitoring camera.

[0107] Detect the coordinates of the first checkerboard corner points of the second virtual reality model in the 3D display and the coordinates of the second checkerboard corner points of the real scene objects behind the 3D display;

[0108] Determine whether the coordinates of the first and second chessboard corner points form a continuous straight line. If so, determine that the second virtual reality model in the 3D display is aligned with the real scene objects behind the 3D display. Otherwise, adjust the scaling factor coefficient correction value to adjust the scaling factor, and adjust the scale of the second virtual reality model according to the adjusted scaling factor.

[0109] In one embodiment, the parameters of the 3D display include the actual size of the 3D display, the viewing distance, and the display angle;

[0110] Processing module 30 is specifically used for:

[0111] The central coordinate position of the 3D display in the world coordinate system is determined as the dense viewpoint shooting center;

[0112] Determine the mapping relationship between the actual size of the 3D display, viewing distance, display angle, and dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in virtual space;

[0113] The number of viewpoints generated, the field of view, the spacing, and the pose are determined based on the mapping relationship; the spacing is equal to the ratio between the field of view and the number of viewpoints generated.

[0114] Based on the dense viewpoint shooting center, the real scene model and virtual reality model behind the 3D display are processed to generate a dense viewpoint image sequence that matches the number of viewpoints generated, field of view, spacing and pose.

[0115] In one embodiment, the acquisition module 10 is specifically used to: acquire information about the real scene behind the 3D display through a camera array;

[0116] The transformation unit is specifically used to: convert the image coordinates of the real scene objects behind the camera array into the world coordinates of the real scene objects behind the camera array, with the geometric center of the camera array as the origin of the world coordinate system.

[0117] In one embodiment, the acquisition module 10 is specifically used to: acquire information about the real scene behind the 3D display using a single camera;

[0118] The conversion unit is specifically used to convert the image coordinates of the real scene objects behind the camera into the world coordinates of the real scene objects behind the camera, with the position of a single camera as the origin of the world coordinate system.

[0119] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. For example... Figure 6 As shown, the electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a 3D display method. This method includes: acquiring information about the real scene behind the 3D display; performing 3D reconstruction of the real scene behind the 3D display based on the real scene information to obtain a 3D real scene model and a virtual reality model; processing the 3D real scene model and the virtual reality model based on the parameters of the 3D display to obtain a dense viewpoint image sequence; performing multi-viewpoint encoding on the dense viewpoint image sequence to obtain an encoded image; and loading the encoded image onto the 3D display for 3D display.

[0120] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional display method provided in the above-described method embodiments. The method includes: acquiring information about the real scene behind the three-dimensional display; performing three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information to obtain a model of the real scene behind the three-dimensional display and a virtual reality model; processing the model of the real scene behind the three-dimensional display and the virtual reality model based on the parameters of the three-dimensional display to obtain a dense viewpoint image sequence; performing multi-viewpoint encoding on the dense viewpoint image sequence to obtain an encoded map; and loading the encoded map onto the three-dimensional display for three-dimensional display.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the three-dimensional display method provided in the above-described method embodiments. The method includes: acquiring information about the real scene behind the three-dimensional display; performing three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information to obtain a model of the real scene behind the three-dimensional display and a virtual reality model; processing the model of the real scene behind the three-dimensional display and the virtual reality model based on parameters of the three-dimensional display to obtain a dense viewpoint image sequence; performing multi-viewpoint encoding on the dense viewpoint image sequence to obtain an encoded map; and loading the encoded map onto the three-dimensional display for three-dimensional display.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional display method, characterized in that, include: Acquire information about the real scene behind the 3D display; Based on the real scene information behind the 3D display, a 3D reconstruction of the real scene behind the 3D display is performed to obtain a real scene model and a virtual reality model behind the 3D display. Based on the parameters of the 3D display, the real scene model and virtual reality model behind the 3D display are processed to obtain a dense viewpoint image sequence. The dense viewpoint image sequence is multi-viewpoint encoded to obtain an encoded map; The encoded image is loaded onto the 3D display for 3D display. The real-world background information includes the image coordinates of objects in the real-world background and the depth distribution information of the real-world background. The step of performing 3D reconstruction of the real-world scene behind the 3D display based on the real-world scene information to obtain a real-world scene model and a virtual reality model behind the 3D display includes: Based on the aforementioned real-world scene information, the model of the underlying real-world scene is reconstructed; Based on the depth distribution information of the real scene behind it, a first virtual reality model is constructed; Convert the image coordinates of the real-world objects in the background to the world coordinates of the real-world objects in the background. Adjust the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the real scene object behind it to obtain the second virtual reality model; The scale of the second virtual reality model is adjusted according to the scaling factor so that it fits the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space. The parameters of the 3D display include the actual size of the 3D display, viewing distance, and display angle; The process, based on the parameters of the 3D display, involves processing the real-world scene model and virtual reality model behind the 3D display to obtain a dense viewpoint image sequence, including: The central coordinate position of the three-dimensional display in the world coordinate system is determined as the dense viewpoint shooting center; Determine the mapping relationship between the actual size, viewing distance, and display angle of the 3D display and the dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in virtual space. The number of viewpoints generated, the field of view, the spacing, and the pose are determined based on the mapping relationship; the spacing is equal to the ratio between the field of view and the number of viewpoints generated. Based on the dense viewpoint shooting center, the real scene model and virtual reality model behind the 3D display are processed to generate the dense viewpoint image sequence that matches the number of viewpoints generated, the field of view, the spacing, and the pose.

2. The three-dimensional display method according to claim 1, characterized in that, The scaling factor is the product of the scaling factor coefficient correction value and the area ratio value, and the area ratio value is the ratio between the area occupied by the real scene object behind the three-dimensional display in the captured image and the area of ​​the captured image. The step of adjusting the scale of the second virtual reality model according to a scaling factor to make it fit the real scene objects behind the 3D display, and obtaining the world coordinates of the virtual reality model in virtual space, includes: The scaling factor is adjusted by continuously adjusting the scaling factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scaling factor until it fits the real scene objects behind the three-dimensional display, so as to obtain the world coordinates of the virtual reality model in the virtual space.

3. The three-dimensional display method according to claim 2, characterized in that, The step of adjusting the scaling factor by continuously adjusting the scaling factor coefficient correction value, and adjusting the scale of the second virtual reality model according to the adjusted scaling factor until it fits the real scene objects behind the 3D display, includes: The following steps are repeated until the second virtual reality model in the 3D display is aligned with the real-world objects behind the 3D display: The second virtual reality model in the 3D display and the real scene objects behind the 3D display are captured simultaneously by a binocular monitoring camera. The coordinates of the first checkerboard corner points of the second virtual reality model in the 3D display and the coordinates of the second checkerboard corner points of the real scene objects behind the 3D display are detected. Determine whether the coordinates of the first chessboard corner point and the coordinates of the second chessboard corner point form a continuous straight line. If so, determine that the second virtual reality model in the 3D display is aligned with the real scene object behind the 3D display. Otherwise, adjust the scale factor coefficient correction value to adjust the scale factor, and adjust the scale of the second virtual reality model according to the adjusted scale factor.

4. The three-dimensional display method according to claim 1, characterized in that, The information about the real scene behind the 3D display that is being acquired includes: The camera array captures information about the real scene behind the 3D display. The step of converting the image coordinates of the real-world objects in the background to the world coordinates of the real-world objects in the background includes: Using the geometric center of the camera array as the origin of the world coordinate system, the image coordinates of the real scene objects behind the camera are converted into the world coordinates of the real scene objects behind the camera.

5. The three-dimensional display method according to claim 1, characterized in that, The information about the real scene behind the 3D display that is being acquired includes: The real scene information behind the 3D display is captured by a single camera; The step of converting the image coordinates of the real-world objects in the background to the world coordinates of the real-world objects in the background includes: Using the position of the single camera as the origin of the world coordinate system, the image coordinates of the real scene objects behind the camera are converted into the world coordinates of the real scene objects behind the camera.

6. A three-dimensional display device, characterized in that, include: The acquisition module is used to acquire information about the real scene behind the 3D display. The reconstruction module is used to perform three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind it, so as to obtain the real scene model behind the three-dimensional display and the virtual reality model. The processing module is used to process the real scene model and virtual reality model behind the 3D display based on the parameters of the 3D display to obtain a dense viewpoint image sequence. The encoding module is used to perform multi-viewpoint encoding on the dense viewpoint image sequence to obtain an encoded map; The display module is used to load the coded image onto the 3D display for 3D display. The real-world background information includes the image coordinates of objects in the real-world background and the depth distribution information of the real-world background. The reconstruction module includes: The reconstruction unit is used to reconstruct the underlying real scene model based on the underlying real scene information; A construction unit is used to construct a first virtual reality model based on the depth distribution information of the underlying real scene; A conversion unit is used to convert the image coordinates of the real scene objects behind the object into the world coordinates of the real scene objects behind the object. The first adjustment unit is used to adjust the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the real scene object behind it, thereby obtaining the second virtual reality model. The second adjustment unit is used to adjust the scale of the second virtual reality model according to the scaling factor so that it fits the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space. The parameters of the 3D display include the actual size of the 3D display, viewing distance, and display angle; The processing module is specifically used for: The central coordinate position of the three-dimensional display in the world coordinate system is determined as the dense viewpoint shooting center; Determine the mapping relationship between the actual size, viewing distance, and display angle of the 3D display and the dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in virtual space. The number of viewpoints generated, the field of view, the spacing, and the pose are determined based on the mapping relationship; the spacing is equal to the ratio between the field of view and the number of viewpoints generated. Based on the dense viewpoint shooting center, the real scene model and virtual reality model behind the 3D display are processed to generate the dense viewpoint image sequence that matches the number of viewpoints generated, the field of view, the spacing, and the pose.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional display method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional display method as described in any one of claims 1 to 5.

Images