Three-dimensional display method and device, electronic equipment and storage medium

By collecting and processing the real scene information behind the three-dimensional display, generating dense viewpoint image sequences and performing multi-viewpoint encoding, the problem that virtual reality display technology cannot display the scene behind it is solved, and the background transparent three-dimensional display is realized, which improves the immersion of the audience feel.

CN118945308AActive Publication Date: 2024-11-12BEIJING UNIV OF POSTS & TELECOMM
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410849843.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-12
Estimated Expiration
2044-06-27

AI Technical Summary

Technical Problem

Existing virtual reality display technology cannot display the scene information behind the three-dimensional display, affecting the audience's sense of immersion.

Method used

Collect the real scene information behind the three-dimensional display, perform three-dimensional reconstruction and processing, generate dense viewpoint image sequences, and realize background transparent three-dimensional display through multi-viewpoint encoding.

Benefits of technology

It realizes the three-dimensional image that displays the real scene model and the virtual reality model behind it simultaneously, improving the audience's immersion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945308A_ABST
    Figure CN118945308A_ABST
Patent Text Reader

Abstract

The invention provides a three-dimensional display method and device, electronic equipment and a storage medium, and relates to the technical field of virtual reality display, and the method comprises the steps: collecting back real scene information of a three-dimensional display; based on the back real scene information, performing three-dimensional reconstruction on the back real scene of the three-dimensional display to obtain a back real scene model and a virtual reality model of the three-dimensional display; based on the parameters of the three-dimensional display, processing a back real scene model and a virtual reality model of the three-dimensional display to obtain a dense viewpoint image sequence; performing multi-viewpoint coding on the dense viewpoint image sequence to obtain a coded image; and loading the coded graph to a three-dimensional display for three-dimensional display. According to the invention, the three-dimensional images of the back real scene model and the virtual reality model can be displayed at the same time, and the background transparency is realized, so that the immersion is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of virtual reality display technology, and in particular to a three-dimensional display method, device, electronic equipment and storage medium. Background Art

[0002] Three-dimensional display is a frontier technology field that attracts a lot of attention today. Three-dimensional display systems with full parallax, multiple angles, and simultaneous viewing by multiple people can be applied in military, advertising, medical and other display fields, allowing viewers to get the feeling of watching real scenes.

[0003] At present, virtual reality display technology usually renders directly in space and cannot display the scene information behind the three-dimensional display, nor can it achieve fused display, which affects the audience's sense of immersion. Summary of the invention

[0004] The present invention provides a three-dimensional display method, device, electronic device and storage medium, which can simultaneously display the stereoscopic images of the real scene model and the virtual reality model behind, and realize the transparency of the background, thereby improving the immersion.

[0005] The present invention provides a three-dimensional display method, comprising: Collect the real scene information behind the 3D display; Based on the real scene information behind the three-dimensional display, three-dimensionally reconstruct the real scene behind the three-dimensional display to obtain a real scene model and a virtual reality model behind the three-dimensional display; Based on the parameters of the three-dimensional display, a real scene model and a virtual reality model behind the three-dimensional display are processed to obtain a dense viewpoint image sequence; Performing multi-view encoding on the dense viewpoint image sequence to obtain a coding image; The coding map is loaded onto the three-dimensional display for three-dimensional display.

[0006] According to a three-dimensional display method provided by the present invention, the real scene information behind includes image coordinates of objects in the real scene behind and depth distribution information of the real scene behind; The method of performing three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind the three-dimensional display to obtain a real scene model and a virtual reality model behind the three-dimensional display includes: Reconstructing the real scene model behind the scene based on the real scene information behind the scene; Based on the depth distribution information of the real scene behind, construct a first virtual reality model; Convert the image coordinates of the object in the real scene behind to the world coordinates of the object in the real scene behind; Adjusting the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the object in the real scene behind it, thereby obtaining a second virtual reality model; The scale of the second virtual reality model is adjusted according to the scale factor so as to make it fit with the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space.

[0007] According to a three-dimensional display method provided by the present invention, the scale factor is the product of the scale factor coefficient correction value and the area ratio, and the area ratio is the ratio between the area occupied by the real scene object behind the three-dimensional display in the shooting picture and the area of ​​the shooting picture; The step of adjusting the scale of the second virtual reality model according to the scale factor so as to make it fit with the real scene object behind the three-dimensional display to obtain the world coordinates of the virtual reality model in the virtual space includes: The scale factor is adjusted by continuously adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor until it fits with the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space.

[0008] According to a three-dimensional display method provided by the present invention, the scale factor is adjusted by continuously adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor until it fits with the real scene object behind the three-dimensional display, including: The following steps are executed in a loop until the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display: Simultaneously photographing the second virtual reality model in the three-dimensional display and the real scene objects behind the three-dimensional display through a binocular surveillance camera; Detecting the coordinates of first checkerboard corner points of the second virtual reality model in the three-dimensional display and the coordinates of second checkerboard corner points of the real scene object behind the three-dimensional display; It is determined whether the coordinates of the first checkerboard corner point and the coordinates of the second checkerboard corner point form a continuous straight line; if so, it is determined that the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display; otherwise, the scale factor is adjusted by adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor.

[0009] According to a three-dimensional display method provided by the present invention, the parameters of the three-dimensional display include the actual size, viewing distance and display viewing angle of the three-dimensional display; The method of processing the real scene model and the virtual reality model behind the three-dimensional display based on the parameters of the three-dimensional display to obtain a dense viewpoint image sequence includes: Determine the middle coordinate position of the three-dimensional display in the world coordinate system as the dense viewpoint shooting center; Determine a mapping relationship between the actual size, viewing distance and display viewing angle of the three-dimensional display and a dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in the virtual space; The number of viewpoints generated, the field of view angle, the spacing and the posture are determined based on the mapping relationship; the spacing is equal to the ratio between the field of view angle and the number of viewpoints generated

[0010] Based on the dense viewpoint shooting center, the real scene model and the virtual reality model behind the three-dimensional display are processed to generate the dense viewpoint image sequence matching the viewpoint generation number, the field of view angle, the spacing and the posture.

[0011] According to a three-dimensional display method provided by the present invention, the collecting of real scene information behind the three-dimensional display includes: Collecting real scene information behind the three-dimensional display through a camera array; The converting the image coordinates of the real scene object behind into the world coordinates of the real scene object behind includes: taking the geometric center position of the camera array as the origin of the world coordinate system, and converting the image coordinates of the real scene object behind into the world coordinates of the real scene object behind.

[0012] According to a three-dimensional display method provided by the present invention, the collecting of real scene information behind the three-dimensional display includes: Collecting real scene information behind the three-dimensional display through a single camera; The converting the image coordinates of the object in the real scene behind into the world coordinates of the object in the real scene behind includes: taking the position of the single camera as the origin of the world coordinate system, and converting the image coordinates of the object in the real scene behind into the world coordinates of the object in the real scene behind.

[0013] The present invention also provides a three-dimensional display device, comprising: A collection module, used to collect real scene information behind the three-dimensional display; A reconstruction module, configured to perform three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind the three-dimensional display, and obtain a real scene model and a virtual reality model behind the three-dimensional display; A processing module, configured to process a real scene model and a virtual reality model behind the three-dimensional display based on parameters of the three-dimensional display to obtain a dense viewpoint image sequence; A coding module, used for performing multi-view coding on the dense viewpoint image sequence to obtain a coding image; The display module is used to load the coding map onto the three-dimensional display for three-dimensional display.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described three-dimensional display methods when executing the program.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional display method as described above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the three-dimensional display method as described above is implemented.

[0017] The three-dimensional display method, device, electronic device and storage medium provided by the present invention first collect the real scene information behind the three-dimensional display; based on the real scene information behind the three-dimensional display, the real scene behind the three-dimensional display is reconstructed in three dimensions to obtain the real scene model behind the three-dimensional display and the virtual reality model; then, based on the parameters of the three-dimensional display, the real scene model behind the three-dimensional display and the virtual reality model are processed to obtain a dense viewpoint image sequence; the dense viewpoint image sequence is multi-viewpoint encoded to obtain a coding map; finally, the coding map is loaded onto the three-dimensional display for three-dimensional display, and the stereoscopic images of the real scene model behind the transparent background and the virtual reality model can be displayed at the same time. Therefore, the present invention can simultaneously display the stereoscopic images of the real scene model behind the virtual reality model and the virtual reality model, and realize the background transparency, thereby improving the sense of immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of a three-dimensional display method provided by an embodiment of the present invention.

[0020] Figure 2 Schematic diagram of a binocular surveillance camera provided by an embodiment of the present invention.

[0021] Figure 3 It is a schematic diagram of a first chessboard corner point coordinate and a second chessboard corner point coordinate forming a continuous straight line provided by an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of a first chessboard corner point coordinate and a second chessboard corner point coordinate provided by an embodiment of the present invention not forming a continuous straight line.

[0023] Figure 5 It is a schematic diagram of the structure of a three-dimensional display device provided by an embodiment of the present invention.

[0024] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Combine the following Figure 1 The three-dimensional display method of the present invention is described.

[0027] Please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a three-dimensional display method provided by an embodiment of the present invention. Figure 1 As shown, the method may include the following steps 101-105.

[0028] Step 101: Collect real scene information behind the three-dimensional display.

[0029] Specifically, the real scene information behind the three-dimensional display can be collected by a camera array or a single camera. The camera array can be a color camera array, the single camera can be a depth camera, and the collection method can be a time series collection method, but the present embodiment is not limited thereto.

[0030] Step 102: Based on the real scene information behind the 3D display, the real scene behind the 3D display is 3D reconstructed to obtain a real scene model and a virtual reality model behind the 3D display.

[0031] In one embodiment, the real scene information behind includes image coordinates of objects in the real scene behind and depth distribution information of the real scene behind; step 102 includes steps 1021 to 1025.

[0032] Step 1021: Reconstruct a model of the real scene behind the scene based on the real scene information behind the scene.

[0033] Specifically, methods such as truncated signed distance function (TSDF), multi-view stereo (MVS), neural radiance fields (NERF) and the like may be used to reconstruct the real scene model behind the scene based on the real scene information behind the scene, but this embodiment is not limited thereto.

[0034] Step 1022: construct a first virtual reality model based on the depth distribution information of the real scene behind.

[0035] Specifically, a first virtual reality model is constructed according to the depth distribution information of the real scene behind, and the occlusion and perspective relationships between objects in the scene behind are restored.

[0036] Step 1023: Convert the image coordinates of the object in the real scene behind into the world coordinates of the object in the real scene behind.

[0037] In one embodiment, step 101 includes: collecting real scene information behind the three-dimensional display through a camera array; step 1023 includes: taking the geometric center position of the camera array as the origin (0, 0, 0) of the world coordinate system, and converting the image coordinates of the real scene objects behind into the world coordinates (x, y, z) of the real scene objects behind.

[0038] In another embodiment, step 101 includes: collecting real scene information behind the three-dimensional display through a single camera; step 1023 includes: taking the position of the single camera as the origin (0, 0, 0) of the world coordinate system, and converting the image coordinates of the real scene objects behind into world coordinates (x, y, z) of the real scene objects behind.

[0039] Step 1024: Adjust the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the object in the real scene behind it, thereby obtaining a second virtual reality model.

[0040] Specifically, the rotation angle of the first virtual reality model is continuously adjusted until the geometric center of the rotated virtual reality model coincides with the geometric center of the real scene object behind it, thereby obtaining the second virtual reality model.

[0041] Step 1025: adjust the scale of the second virtual reality model according to the scale factor so that it fits with the real scene objects behind the three-dimensional display, and obtain the world coordinates of the virtual reality model in the virtual space.

[0042] Optionally, the scale factor is the product of the scale factor coefficient correction value and the area ratio, and the area ratio is the ratio between the area occupied by the real scene object behind the three-dimensional display in the shooting picture and the area of ​​the shooting picture; step 1025 includes: adjusting the scale factor by continuously adjusting the scale factor coefficient correction value, and adjusting the scale of the second virtual reality model according to the adjusted scale factor until it fits with the real scene object behind the three-dimensional display, so as to obtain the world coordinates of the virtual reality model in the virtual space.

[0043] For example, the scale factor can be calculated by the expression Q=ΔQ*(S1 / S2), where Q represents the scale factor, ΔQ represents the scale factor coefficient correction value, the initial value of ΔQ is 1, S1 represents the area occupied by the real scene object behind the three-dimensional display in the shooting picture, and S2 represents the area of ​​the shooting picture. The scale factor Q is adjusted by continuously adjusting the scale factor coefficient correction value ΔQ, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor Q until it fits the real scene object behind the three-dimensional display, thereby obtaining the world coordinates (x+Δx, y+Δy, z+Δz) of the virtual reality model in the virtual space.

[0044] In one embodiment, the scale factor is adjusted by continuously adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor until it fits the real scene object behind the three-dimensional display, including: The following steps are executed repeatedly until the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display: Figure 2 The binocular surveillance camera shown in the figure simultaneously shoots the second virtual reality model in the three-dimensional display and the real scene object behind the three-dimensional display; detects the coordinates of the first checkerboard corner point of the second virtual reality model in the three-dimensional display and the coordinates of the second checkerboard corner point of the real scene object behind the three-dimensional display; determines whether the coordinates of the first checkerboard corner point and the second checkerboard corner point form a continuous straight line; if so, Figure 3 As shown, it is determined that the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display; if not, Figure 4 As shown, the scale factor coefficient correction value is adjusted to adjust the scale factor, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor.

[0045] Step 103: Based on the parameters of the three-dimensional display, the real scene model and the virtual reality model behind the three-dimensional display are processed to obtain a dense viewpoint image sequence.

[0046] In one embodiment, the parameters of the 3D display include the actual size, viewing distance, and display viewing angle of the 3D display; step 103 includes steps 1031 to 1034.

[0047] Step 1031: determine the middle coordinate position of the three-dimensional display in the world coordinate system as the dense viewpoint shooting center.

[0048] Step 1032: Determine the mapping relationship between the actual size, viewing distance and display viewing angle of the three-dimensional display and the dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in the virtual space.

[0049] Specifically, a mapping relationship between parameters such as the actual size of the 3D display, the viewing distance, and the display viewing angle and the dense viewpoint generation positions is derived.

[0050] Step 1033: Determine the number of viewpoints N, the field of view angle fov, the spacing d and the posture based on the mapping relationship; the spacing is equal to the ratio between the field of view angle and the number of viewpoints generated, that is, d=fov / N.

[0051] Step 1034: Based on the dense viewpoint shooting center, the real scene model and the virtual reality model behind the three-dimensional display are processed to generate a dense viewpoint image sequence that matches the viewpoint generation number, field of view angle, spacing and posture. The dense viewpoint image sequence is a plurality of discrete viewpoint images.

[0052] Step 104: Perform multi-view encoding on the dense viewpoint image sequence to obtain a coded image.

[0053] Specifically, by performing multi-view encoding on a dense viewpoint image sequence, a plurality of discrete viewpoint images can be encoded into a single interlaced image, ie, an encoded image, which matches the parameters of a three-dimensional display.

[0054] Step 105: Load the coding diagram onto a three-dimensional display for three-dimensional display.

[0055] Specifically, loading the coding map onto a three-dimensional display for three-dimensional display can simultaneously realize the 3D effect display of the real scene model behind the transparent background and the virtual reality model, increase the sense of immersion, and make the background transparency more realistic.

[0056] The three-dimensional display method provided by the embodiment of the present invention first collects the real scene information behind the three-dimensional display; based on the real scene information behind the three-dimensional display, the real scene behind the three-dimensional display is reconstructed in three dimensions to obtain the real scene model behind the three-dimensional display and the virtual reality model; then, based on the parameters of the three-dimensional display, the real scene model behind the three-dimensional display and the virtual reality model are processed to obtain a dense viewpoint image sequence; the dense viewpoint image sequence is multi-viewpoint encoded to obtain a coding map; finally, the coding map is loaded onto the three-dimensional display for three-dimensional display, and the stereoscopic images of the real scene model behind the three-dimensional display and the virtual reality model with transparent background can be displayed at the same time. Therefore, the embodiment of the present invention can simultaneously display the stereoscopic images of the real scene model behind the three-dimensional display and the virtual reality model, and realize the background transparency, thereby improving the sense of immersion.

[0057] The three-dimensional display device provided by the present invention is described below. The three-dimensional display device described below and the three-dimensional display method described above can be referred to each other.

[0058] Please refer to Figure 5 , Figure 5 is a schematic diagram of the structure of a three-dimensional display device provided by an embodiment of the present invention. Figure 5 As shown, the device may include: A collection module 10, used to collect real scene information behind the three-dimensional display; A reconstruction module 20 is used to perform three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind, so as to obtain a real scene model and a virtual reality model behind the three-dimensional display; A processing module 30 is used to process the real scene model and the virtual reality model behind the three-dimensional display based on the parameters of the three-dimensional display to obtain a dense viewpoint image sequence; The encoding module 40 is used to perform multi-view encoding on the dense viewpoint image sequence to obtain a coding image; The display module 50 is used to load the coding map onto a three-dimensional display for three-dimensional display.

[0059] In one embodiment, the real scene information behind includes image coordinates of objects in the real scene behind and depth distribution information of the real scene behind; The reconstruction module 20 comprises: A reconstruction unit, used for reconstructing a model of the real scene behind based on the real scene information behind; A construction unit, configured to construct a first virtual reality model based on the depth distribution information of the real scene behind; A conversion unit, used for converting the image coordinates of the object in the real scene behind into the world coordinates of the object in the real scene behind; A first adjustment unit is used to adjust the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the object in the real scene behind it, thereby obtaining a second virtual reality model; The second adjustment unit is used to adjust the scale of the second virtual reality model according to the scale factor so as to make it fit with the real scene objects behind the three-dimensional display, and obtain the world coordinates of the virtual reality model in the virtual space.

[0060] In one embodiment, the scale factor is the product of the scale factor coefficient correction value and the area ratio, and the area ratio is the ratio between the area occupied by the real scene object behind the three-dimensional display in the shooting picture and the area of ​​the shooting picture; the second adjustment unit is specifically used for: The scale factor is adjusted by continuously adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor until it fits with the real scene object behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space.

[0061] In one embodiment, the second adjustment unit is specifically configured to: The following steps are executed repeatedly until the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display: The second virtual reality model in the three-dimensional display and the real scene objects behind the three-dimensional display are simultaneously photographed by a binocular monitoring camera; Detecting the coordinates of first checkerboard corner points of the second virtual reality model in the three-dimensional display and the coordinates of second checkerboard corner points of the real scene object behind the three-dimensional display; It is determined whether the coordinates of the first chessboard corner point and the coordinates of the second chessboard corner point form a continuous straight line. If so, it is determined that the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display. Otherwise, the scale factor is adjusted by adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor.

[0062] In one embodiment, the parameters of the 3D display include the actual size, viewing distance and display viewing angle of the 3D display; The processing module 30 is specifically used for: Determine the middle coordinate position of the three-dimensional display in the world coordinate system as the dense viewpoint shooting center; Determine the mapping relationship between the actual size, viewing distance and display viewing angle of the three-dimensional display and the dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in the virtual space; Based on the mapping relationship, the number of viewpoints generated, the field of view angle, the spacing and the posture are determined; the spacing is equal to the ratio between the field of view angle and the number of viewpoints generated; Based on the dense viewpoint shooting center, the real scene model and virtual reality model behind the three-dimensional display are processed to generate a dense viewpoint image sequence that matches the viewpoint generation number, field of view angle, spacing and posture.

[0063] In one embodiment, the acquisition module 10 is specifically used to: acquire real scene information behind the three-dimensional display through a camera array; The conversion unit is specifically used to: use the geometric center position of the camera array as the origin of the world coordinate system, and convert the image coordinates of the real scene object behind into the world coordinates of the real scene object behind.

[0064] In one embodiment, the acquisition module 10 is specifically used to: acquire real scene information behind the three-dimensional display through a single camera; The conversion unit is specifically used to: take the position of a single camera as the origin of the world coordinate system, and convert the image coordinates of the object in the real scene behind into the world coordinates of the object in the real scene behind.

[0065] Please refer to Figure 6 , Figure 6 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the three-dimensional display method, which includes: collecting the real scene information behind the three-dimensional display; based on the real scene information behind the three-dimensional display, performing three-dimensional reconstruction on the real scene behind the three-dimensional display to obtain the real scene model and virtual reality model behind the three-dimensional display; based on the parameters of the three-dimensional display, processing the real scene model and virtual reality model behind the three-dimensional display to obtain a dense viewpoint image sequence; performing multi-viewpoint encoding on the dense viewpoint image sequence to obtain a coding map; loading the coding map onto the three-dimensional display for three-dimensional display.

[0066] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0067] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional display method provided by the above-mentioned method embodiments, which method includes: collecting real scene information behind the three-dimensional display; based on the real scene information behind the three-dimensional display, performing three-dimensional reconstruction of the real scene behind the three-dimensional display to obtain a real scene model and a virtual reality model behind the three-dimensional display; based on the parameters of the three-dimensional display, processing the real scene model and the virtual reality model behind the three-dimensional display to obtain a dense viewpoint image sequence; performing multi-viewpoint encoding on the dense viewpoint image sequence to obtain a coded map; and loading the coded map onto the three-dimensional display for three-dimensional display.

[0068] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the three-dimensional display method provided by the above-mentioned method embodiments, the method comprising: collecting real scene information behind the three-dimensional display; based on the real scene information behind the three-dimensional display, performing three-dimensional reconstruction of the real scene behind the three-dimensional display to obtain a real scene model and a virtual reality model behind the three-dimensional display; based on parameters of the three-dimensional display, processing the real scene model and the virtual reality model behind the three-dimensional display to obtain a dense viewpoint image sequence; performing multi-viewpoint encoding on the dense viewpoint image sequence to obtain a coded map; and loading the coded map onto the three-dimensional display for three-dimensional display.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional display method, characterized in that: include: Collect the real scene information behind the 3D display; Based on the real scene information behind the three-dimensional display, three-dimensionally reconstruct the real scene behind the three-dimensional display to obtain a real scene model and a virtual reality model behind the three-dimensional display; Based on the parameters of the three-dimensional display, a real scene model and a virtual reality model behind the three-dimensional display are processed to obtain a dense viewpoint image sequence; Performing multi-view encoding on the dense viewpoint image sequence to obtain a coding image; The coding map is loaded onto the three-dimensional display for three-dimensional display.

2. The three-dimensional display method according to claim 1, characterized in that: The real scene information behind includes image coordinates of objects in the real scene behind and depth distribution information of the real scene behind; The method of performing three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind the three-dimensional display to obtain a real scene model and a virtual reality model behind the three-dimensional display includes: Reconstructing the real scene model behind the scene based on the real scene information behind the scene; Based on the depth distribution information of the real scene behind, construct a first virtual reality model; Convert the image coordinates of the object in the real scene behind to the world coordinates of the object in the real scene behind; Adjusting the rotation angle of the first virtual reality model so that its geometric center coincides with the geometric center of the object in the real scene behind it, thereby obtaining a second virtual reality model; The scale of the second virtual reality model is adjusted according to the scale factor so as to make it fit with the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space.

3. The three-dimensional display method according to claim 2, characterized in that: The scale factor is the product of the scale factor coefficient correction value and the area ratio, and the area ratio is the ratio between the area occupied by the real scene object behind the three-dimensional display in the shooting picture and the area of ​​the shooting picture; The step of adjusting the scale of the second virtual reality model according to the scale factor so as to make it fit with the real scene object behind the three-dimensional display to obtain the world coordinates of the virtual reality model in the virtual space includes: The scale factor is adjusted by continuously adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor until it fits with the real scene objects behind the three-dimensional display, thereby obtaining the world coordinates of the virtual reality model in the virtual space.

4. The three-dimensional display method according to claim 3, characterized in that: The step of adjusting the scale factor by continuously adjusting the scale factor coefficient correction value, and adjusting the scale of the second virtual reality model according to the adjusted scale factor until the second virtual reality model fits the real scene object behind the three-dimensional display, includes: The following steps are executed in a loop until the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display: Simultaneously photographing the second virtual reality model in the three-dimensional display and the real scene objects behind the three-dimensional display through a binocular surveillance camera; Detecting the coordinates of first checkerboard corner points of the second virtual reality model in the three-dimensional display and the coordinates of second checkerboard corner points of the real scene object behind the three-dimensional display; It is determined whether the coordinates of the first checkerboard corner point and the coordinates of the second checkerboard corner point form a continuous straight line; if so, it is determined that the second virtual reality model in the three-dimensional display is aligned with the real scene object behind the three-dimensional display; otherwise, the scale factor is adjusted by adjusting the scale factor coefficient correction value, and the scale of the second virtual reality model is adjusted according to the adjusted scale factor.

5. The three-dimensional display method according to claim 2, characterized in that: The parameters of the three-dimensional display include the actual size, viewing distance and display viewing angle of the three-dimensional display; The method of processing the real scene model and the virtual reality model behind the three-dimensional display based on the parameters of the three-dimensional display to obtain a dense viewpoint image sequence includes: Determine the middle coordinate position of the three-dimensional display in the world coordinate system as the dense viewpoint shooting center; Determine a mapping relationship between the actual size, viewing distance and display viewing angle of the three-dimensional display and a dense viewpoint generation position; the dense viewpoint generation position is the world coordinate of the virtual reality model in the virtual space; The number of viewpoints generated, the field of view angle, the spacing and the posture are determined based on the mapping relationship; the spacing is equal to the ratio between the field of view angle and the number of viewpoints generated Based on the dense viewpoint shooting center, the real scene model and the virtual reality model behind the three-dimensional display are processed to generate the dense viewpoint image sequence matching the viewpoint generation number, the field of view angle, the spacing and the posture.

6. The three-dimensional display method according to claim 2, characterized in that: The collecting of the real scene information behind the three-dimensional display includes: Collecting real scene information behind the three-dimensional display through a camera array; The step of converting the image coordinates of the object in the real scene behind into the world coordinates of the object in the real scene behind includes: The geometric center position of the camera array is used as the origin of the world coordinate system, and the image coordinates of the object in the real scene behind are converted into the world coordinates of the object in the real scene behind.

7. The three-dimensional display method according to claim 2, characterized in that: The collecting of the real scene information behind the three-dimensional display includes: Collecting real scene information behind the three-dimensional display through a single camera; The step of converting the image coordinates of the object in the real scene behind into the world coordinates of the object in the real scene behind includes: The position of the single camera is used as the origin of the world coordinate system, and the image coordinates of the object in the real scene behind are converted into the world coordinates of the object in the real scene behind.

8. A three-dimensional display device, characterized in that: include: A collection module, used to collect real scene information behind the three-dimensional display; A reconstruction module, configured to perform three-dimensional reconstruction of the real scene behind the three-dimensional display based on the real scene information behind the three-dimensional display, and obtain a real scene model and a virtual reality model behind the three-dimensional display; A processing module, configured to process a real scene model and a virtual reality model behind the three-dimensional display based on parameters of the three-dimensional display to obtain a dense viewpoint image sequence; A coding module, used for performing multi-view coding on the dense viewpoint image sequence to obtain a coding image; The display module is used to load the coding map onto the three-dimensional display for three-dimensional display.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the three-dimensional display method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional display method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Presenting object models in augmented reality images

    CN105074623A

  • Reality-mixed helmet display system and control method

    CN106055113A

  • Three-dimensional model dynamic rendering method and device, electronic equipment and storage medium

    CN112446939A

  • Light field display method and system based on three-dimensional scene stylization

    CN116418961A

  • Augmented reality-based remote guidance method and apparatus, terminal, and storage medium

    US20210120221A1