Three-dimensional scene display method and system based on digital twin

Abstract

The present invention relates to the field of three-dimensional display technology, and discloses a three-dimensional scene display method based on digital twin and its system, the method comprising: obtaining a first original disparity map; the first original disparity map is a disparity map in a first direction; according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane, arranging each viewpoint in the first original disparity map into a first coded disparity map; sending the first coded disparity map to a display plane module, instructing the display plane module to display the first coded disparity map. The present invention can significantly improve the viewpoint density of a three-dimensional light field without changing the resolution of a light field digital twin display device.

Description

Technical Field

[0001] The present invention relates to the field of equipment detection technology, and in particular to a three-dimensional scene display method and system based on digital twins. Background Art

[0002] Current display technology is undergoing a revolution from two-dimensional to three-dimensional. With the rise of various new media technologies and human-computer interaction technologies, human demand for display technologies that can provide more realistic and immersive experiences has increased dramatically. Against this background, light field display technology came into being. Its core lies in the ability to record and reproduce the direction and intensity information of light, thereby presenting users with a real three-dimensional visual effect without the need to wear special glasses. Three-dimensional display adds depth information to traditional two-dimensional display, and can truly restore three-dimensional objects and scenes. It has broad application prospects in medical imaging, military command, intelligent manufacturing, and distance education. Three-dimensional display usually uses display panels and light control devices to construct different parallax images at different angles, providing viewers with a close-to-real three-dimensional visual experience.

[0003] In 3D display technology, viewpoint density is an important indicator that affects the accuracy of 3D display. The more complex the 3D scene is, the denser the viewpoints are needed to ensure the accuracy of the display. However, due to the limitations of existing 3D light field digital twin display devices, the viewpoint density is not the greater the better. Excessively increasing the viewpoint density will lead to a decrease in spatial resolution, resulting in a decrease in the clarity of the displayed image, which seriously affects the viewing experience. However, if the viewpoint density is too small, the details of the 3D display image will be lost, and the highly complex content in the 3D scene cannot be accurately displayed. Summary of the invention

[0004] In response to the above-mentioned problems, the present invention provides a three-dimensional scene display method and system based on digital twins, which are used to significantly improve the viewpoint density of the three-dimensional light field without changing the resolution of the light field digital twin display device.

[0005] In a first aspect, the present invention provides a three-dimensional scene display method based on digital twins, which is applied to a digital twin display device, wherein the digital twin display device includes a dense array light control module and a display plane module arranged in parallel in sequence; the display plane module includes a display panel with point-to-point light control; the dense array light control module includes an opaque optical waveguide material and a light control array attached to the optical waveguide material in a preset distribution manner; the light emitted by the backlight light source of the digital twin display device, after passing through the bonding surface of the light control array and the optical waveguide material, is refracted by the light control array to form a plurality of backlight areas, and the three-dimensional scene display method based on digital twins includes:

[0006] Acquire a first original disparity map; the first original disparity map is a disparity map in a first direction;

[0007] Arranging each viewpoint in the first original disparity map into a first coded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; wherein the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material;

[0008] The first encoded disparity map is sent to a display plane module, and the display plane module is instructed to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units whose display viewpoints are consecutively numbered in the first original disparity map to the same projection unit in the first direction under the viewing plane.

[0009] According to the three-dimensional scene display method based on digital twins provided by the present invention, the viewpoints in the first original disparity map are arranged into the first encoded disparity map according to the light field viewpoint density in the first direction and the projection positions of the display units on the display panel in the first direction under the viewing plane, including:

[0010] The number of the viewpoint in the first original disparity map at which each viewpoint position on the first encoded disparity map is filled is obtained in the following manner:

[0011] Using the light field viewpoint density in the first direction, a rounding operation is performed on the projection position of the display unit on the display panel corresponding to the viewpoint position in the first direction under the viewing plane; wherein the projection position is represented by a ratio between a distance between a projection point of the display unit in the first direction under the viewing plane and a starting projection point of a backlight area covering the display unit in the first direction under the viewing plane and a width of the projection unit;

[0012] According to the result of the rounding operation, obtain the number of the viewpoint in the first original disparity map in which the viewpoint position is filled;

[0013] Fill each viewpoint in the first original disparity map into the first encoded disparity map according to the number of the viewpoint in the first original disparity map in which each viewpoint position is filled.

[0014] According to the three-dimensional scene display method based on digital twins provided by the present invention, the following formula is used to perform a rounding operation on the projection position of the display unit on the display panel corresponding to the viewpoint position in the first direction under the viewing plane using the light field viewpoint density in the first direction, and obtain the number of the viewpoint in the first original disparity map filled with the viewpoint position according to the result of the rounding operation:

[0015]

[0016] Wherein, d is the inverse of the light field viewpoint density in the first direction, D is the projection position, and n is the number of the viewpoint in the first original disparity map where the viewpoint position is filled.

[0017] According to the three-dimensional scene display method based on digital twins provided by the present invention, the distribution of the light control array on the surface of the optical waveguide material is as follows:

[0018] Each group of the light control arrays is tilted and distributed on the surface of the optical waveguide material at a preset angle other than 90 degrees;

[0019] The projection position of each display unit on the display panel in the first direction below the viewing plane is obtained in the following manner:

[0020] The projection position of the display unit in the first direction below the viewing plane is obtained according to the horizontal distance between the upper left corner of any display unit and the left edge of the light control array.

[0021] According to the three-dimensional scene display method based on digital twins provided by the present invention, the three-dimensional scene display method based on digital twins also includes:

[0022] Acquire a second original disparity map; the second original disparity map is a disparity map having disparities in multiple directions;

[0023] For each of the multiple directions, perform the following operations:

[0024] Arrange each viewpoint corresponding to the direction in the second original disparity map into a second coded disparity map according to the light field viewpoint density in the direction and the projection position of each display unit on the display panel in the direction under the viewing plane; wherein the projection position of each display unit on the display panel in the direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material;

[0025] The second encoded disparity map is sent to a display plane module, and the display plane module is instructed to display the second encoded disparity map so that the backlight area projects multiple display units on the display panel covered by it, which are located in different rows and whose displayed viewpoints are in any direction with consecutive numbers in the second original disparity map, into the same projection unit in the direction under the viewing plane.

[0026] According to the three-dimensional scene display method based on digital twins provided by the present invention, the viewpoints corresponding to the direction in the second original disparity map are arranged into the second encoded disparity map according to the light field viewpoint density in the direction and the projection positions of the display units on the display panel in the direction under the viewing plane, including:

[0027] The number of the viewpoint in the second original disparity map at which each viewpoint position on the second encoded disparity map is filled is obtained in the following manner:

[0028] Using the light field viewpoint density in the direction, a rounding operation is performed on the projection position of the display unit on the display panel corresponding to the viewpoint position in the direction under the viewing plane; wherein the projection position is represented by a ratio between a distance between a projection point of the display unit in the direction under the viewing plane and a starting projection point of a backlight area covering the display unit in the direction under the viewing plane and a width of the projection unit in the direction;

[0029] According to the result of the rounding operation, obtain the number of the viewpoint in the second original disparity map in which the viewpoint position is filled;

[0030] Fill each viewpoint in the second original disparity map into the second encoded disparity map according to the number of the viewpoint in the second original disparity map in which each viewpoint position is filled.

[0031] In the second aspect, the digital twin display device is applied, the digital twin display device includes a dense array light control module and a display plane module arranged in parallel in sequence; the display plane module includes a display panel with point-to-point light control; the dense array light control module includes an opaque optical waveguide material and a light control array attached to the optical waveguide material in a preset distribution manner; the light emitted by the backlight light source of the digital twin display device passes through the bonding surface of the light control array and the optical waveguide material, and is refracted by the light control array to form multiple backlight areas. The three-dimensional scene display system based on digital twins includes:

[0032] An acquisition module, configured to acquire a first original disparity map; the first original disparity map is a disparity map in a first direction;

[0033] an encoding module, configured to arrange each viewpoint in the first original disparity map into a first encoded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; wherein the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material;

[0034] A sending module is used to send the first encoded disparity map to a display plane module, instructing the display plane module to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units whose display viewpoints are consecutively numbered in the first original disparity map into the same projection unit in a first direction under the viewing plane.

[0035] In a third aspect, the present invention also provides an electronic device, comprising: a memory for storing a computer software program; and a processor for reading and executing the computer software program, thereby implementing the three-dimensional scene display method based on digital twins as described in the first aspect above.

[0036] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, in which a computer software program is stored. When the computer software program is executed by a processor, the three-dimensional scene display method based on digital twins as described in the first aspect above is implemented.

[0037] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the three-dimensional scene display method based on digital twins as described in the first aspect above.

[0038] The three-dimensional scene display method based on digital twin provided by the present invention arranges each viewpoint in the first original disparity map into the first coded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; the first coded disparity map is sent to the display plane module, and the display plane module is instructed to display the first coded disparity map. Since the positions of each viewpoint in the first coded disparity map are determined according to the projection position of the display unit on the display panel corresponding to it in the first direction under the viewing plane, after the first coded disparity map is displayed by the display panel, under the projection of the backlight area of ​​the dense array light control module, multiple different rows on the display panel and the display units whose displayed viewpoints are numbered continuously in the first original disparity map can be projected into the same projection unit in the first direction under the viewing plane, thereby significantly improving the viewpoint density of the three-dimensional light field without changing the resolution of the light field digital twin display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is a structural schematic diagram of the digital twin display device provided by the present invention;

[0041] Figure 2 It is a schematic diagram of a display plane module, a light control array and an optical waveguide material provided by the present invention;

[0042] Figure 3 is a schematic diagram of a light-controlling grating structure provided by the present invention;

[0043] Figure 4 is a schematic diagram of another light-controlling grating structure provided by the present invention;

[0044] Figure 5 It is a schematic diagram of a dense point array light control module provided by the present invention;

[0045] Figure 6 It is a schematic diagram of lens array specifications of the dense array light control module provided by the present invention;

[0046] Figure 7 It is one of the flow charts of the three-dimensional scene display method based on digital twin provided by the present invention;

[0047] Figure 8 is a schematic diagram of a flow chart of a method for obtaining a first encoded disparity map provided by the present invention;

[0048] Fig. 9 This is the second flow chart of the three-dimensional scene display method based on digital twins provided by the present invention;

[0049] Fig.10 is a schematic diagram of a flow chart of a method for obtaining a second encoded disparity map provided by the present invention;

[0050] Fig.11 It is a schematic diagram of a dense line array light control module and corresponding viewpoint coding provided by the present invention;

[0051] Fig.12 It is a schematic diagram of local viewpoint coding of a dense array light control module provided by the present invention;

[0052] Fig.13 It is a schematic diagram of the arrangement of viewpoint coding information provided by the present invention;

[0053] Fig.14 Schematic diagram of the arrangement of viewpoints of L1 row pixels provided by the present invention;

[0054] Fig.15 Schematic diagram of the arrangement of viewpoints of L2 rows of pixels provided by the present invention;

[0055] Fig.16 Schematic diagram of the arrangement of viewpoints of L3 rows of pixels provided by the present invention;

[0056] Fig.17 It is a structural schematic diagram of a three-dimensional scene display system based on digital twins provided by the present invention;

[0057] Fig.18 is a schematic diagram of the structure of an electronic device provided by the present invention;

[0058] Fig.19 A schematic diagram of an embodiment of a computer-readable storage medium provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0061] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0062] Figure 1 Schematic diagram of the structure of the digital twin display device provided by the present invention. Figure 1 As shown, the digital twin display device includes a dense array light control module and a display plane module arranged in parallel in sequence.

[0063] The display plane module includes a display panel with point-to-point light control. A single point or single line light control unit from the dense array light control module can control all display units in a corresponding area of ​​the display panel. In the specific implementation process, for different types of display panels, the display unit can be a pixel unit or a sub-pixel unit.

[0064] like Figure 2 As shown, the dense array light control module is composed of a panel-shaped component, including an opaque optical waveguide material and a light control array attached to the optical waveguide material in a preset distribution manner. After passing through the bonding surface of the light control array and the optical waveguide material, the light emitted by the backlight source of the digital twin display device is refracted by the light control array to form multiple backlight areas.

[0065] The dense array light control module part can control the luminous effect of the backlight source, and transform the complete continuous surface backlight source of the general digital twin display device into two different light control areas: the luminous area and the non-luminous area. In the luminous area, light can escape normally from the backlight, forming a light source with uniform brightness including but not limited to a point array light source, a line array light source, etc., which can light up the corresponding area in the display panel; in the non-luminous area, light is restricted from escaping from the backlight, forming a dark area without light.

[0066] The light control array is made of periodically arranged strips (such as Figure 3 The light control grating structure shown) or a point array light source (such as Figure 4 The light-controlling grating structure shown in the figure can control the light to form the required line array or point array backlight source luminous effect.

[0067] In some embodiments, the grating has a directional light control effect, and a light control grating is selected as the light control array. A light control grating module that conforms to the light control effect of the three-dimensional light field display is attached to the upper surface of the light waveguide material panel near the digital twin display device module, so that light can propagate inside the light waveguide material panel, and can only escape the light waveguide material from the area where the light control grating module is attached, forming a luminous area, and the remaining area where the light control grating module is not attached forms a non-luminous area. The light control grating is a cylinder with a semi-elliptical cross-section, and one side of its plane is tightly attached to the light waveguide material panel.

[0068] Just as an example, Figure 3The light-controlling grating structure shown is composed of a plurality of cylindrical lenses of the same specifications, which can make the light emitted by the dense array light-control module part arranged in a linear array. For the preferred cylindrical lens grating, its size should be consistent with the size of the light source in the light-emitting area of ​​the dense array light-control module part in the design, and the refractive index of the lens grating should be similar to that of the optical waveguide material. For example, the width of the bonding area between a single grating and the optical waveguide material panel should be less than or equal to the width of the smallest display unit (for example, pixel or pixel) of the display panel, the grating height should be less than or equal to half the width of the bonding area, and the length should be consistent with the height and width of the waveguide material panel.

[0069] For the optical waveguide material, a light-transmitting material with a refractive index greater than 1.4 is preferably used. In the specific implementation process, a glue layer with a refractive index close to that of the optical waveguide material and the light control array can be used to stick the two together. For a preferred dense array light control module part, a manufacturing process combining the optical waveguide material and the light control array is used to directly form a grating lens-shaped protrusion on the smooth surface of the optical waveguide material, while maintaining the light control effect, reducing the light loss caused by the bonding of the two.

[0070] like Figure 5 and Figure 6 In the display panel and light control array shown, the vertical length of the smallest controllable display unit (pixel or sub-pixel) is a, and the horizontal width is b. On the dense array light control module part, the vertical length of a single light-emitting area (for example, a single light control grating) is h, and the horizontal width is l. The horizontal period spacing between the single light-emitting areas of the light control array is P, and the vertical period spacing is Q.

[0071] When the digital twin display device performs a single-direction parallax three-dimensional light field display task, the size of a single light-emitting area on the dense array light control module part must satisfy l≤b, and there is no restriction on h.

[0072] The viewing angle θ of the three-dimensional image obtained by the digital twin display device is determined by the distance L between the dense array light control module and the display panel, and the horizontal period spacing P of the light-emitting area on the module, and satisfies the following formula:

[0073]

[0074] When the digital twin display device performs a three-dimensional light field display task with parallax in both the horizontal and vertical directions, the size of each luminous area must satisfy h≤a, l≤b.

[0075] Horizontal viewing angle θ of the three-dimensional image obtained using the digital twin display device w Vertical viewing angle θ h, which is determined by the distance L between the dense array light control module and the display panel, and the horizontal period spacing P and vertical period spacing Q of the light-emitting area on the module, and satisfies the following formula:

[0076]

[0077] In the specific implementation process, in order to obtain a clearer display effect and avoid two pixels of the same color being imaged adjacently, the horizontal period spacing P of the light-emitting area, the vertical period spacing Q, and the horizontal displacement R of the two rows of light-emitting areas should satisfy the following formula:

[0078] P=i×b+x (4)

[0079] Q=j×a+y (5)

[0080] P>R>0 (6)

[0081] Among them, i is a positive integer that is not an integer multiple of 3, j is an arbitrary positive integer, x is an arbitrary number between 0 and b, y is an arbitrary number between 0 and a, and R is an arbitrary number between 0 and P.

[0082] As an example only, a digital twin display device can be used as follows Figure 5 and Figure 6 The 4k resolution LCD panel shown is used as a display panel. The smallest controllable unit of the LCD panel is a single pixel. The vertical length of the pixel is a=36.75μm and the horizontal width is b=12.25μm. The pixels are arranged in a cyclical pattern of RGB (red, green and blue) in the horizontal direction; in the vertical direction, each column of pixels is the same color. When the design only has a light field display system with N=21 viewpoints in the horizontal direction under the viewing plane. According to the design parameters, the horizontal period spacing of the line array should be P=85.75μm, the horizontal width of a single line light source n=l=12.25μm, and the vertical length of the line light source is the same as the vertical length of the waveguide material. There is a glass interlayer with a thickness of 600μm between the three-dimensional light guide plate and the LCD panel. This parameter achieves a viewing range of 28cm continuous visible three-dimensional scenes at a viewing distance of 40cm, and the viewing angle of the three-dimensional scene reaches about 40 degrees.

[0083] The digital twin display device provided by the present invention utilizes a dense array light control module including an opaque optical waveguide material and a light control array attached to the optical waveguide material in a preset distribution manner to generate a backlight area that can light up the corresponding area of ​​the display panel with point-to-point light control, and then can be distributed in accordance with the light control array on the optical waveguide material, in conjunction with the disparity map encoding algorithm in the processor (for detailed description, see Figures 7 to 10 The relevant content in the description will not be repeated here), thereby improving the viewpoint density of the three-dimensional image obtained by using the digital twin display device.

[0084] Figure 7 is one of the flow charts of the three-dimensional scene display method based on digital twin provided by the present invention, such as Figure 7 As shown, the method includes the following:

[0085] Step 701: Obtain a first original disparity map; the first original disparity map is a disparity map in a first direction.

[0086] The original disparity map is an image generated when estimating the depth information of each point in the scene by comparing two or more images captured from different angles (for example, images captured by left and right cameras) in a stereoscopic vision or binocular vision system. The first original disparity map is a disparity map in a single direction. For example, the first original disparity map can be an original disparity map with disparity in the horizontal direction or the vertical direction.

[0087] In the specific implementation process, the module executing the disparity map encoding algorithm in the processor obtains the first original disparity map from other processing modules through common methods (for example, interface call or memory sharing, etc.), which is not limited by the description of this specification.

[0088] Step 702: Arrange each viewpoint in the first original disparity map into a first encoded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; wherein the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution of the light control array on the surface of the optical waveguide material.

[0089] The viewing plane is the plane parallel to the display panel where the observer's line of sight is located when the observation distance between the observer and the digital twin display device is given. The angle of the lateral observation distance at which the observer can move freely without seeing an inverted or distorted image.

[0090] The light field viewpoint density is the number of viewpoints of the same projection unit in the first direction under the viewing plane. The projection unit is a display unit in the display panel that is projected by the backlight area generated by the dense array light control module to the area in the first direction under the viewing plane. In the related art, the number of viewpoints projected to each projection unit is 1. In the embodiment provided by the present invention, multiple viewpoints can be projected into the same projection unit by using the distribution of the light control array on the optical waveguide material.

[0091] The higher the light field viewpoint density, that is, the more viewpoints per unit area on the viewpoint plane, the greater the amount of spatial information it contains, thus being able to present more natural and realistic depth clue information.

[0092] In a specific implementation process, the light field viewpoint density can be set by adjusting the distribution of the light control array on the surface of the optical waveguide material. Fig.12 The light control array shown can project three display units onto the same projection unit in the first direction under the viewing plane. Therefore, the light field viewpoint density corresponding to this distribution mode of the light control array is 3. For another example, by increasing Fig.12 The angle between the light control array and the vertical line on the surface of the optical waveguide material shown can project 5 display units into the same projection unit in the first direction under the viewing plane, and the light field viewpoint density corresponding to the adjusted distribution mode is 5. In a specific implementation process, the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material.

[0093] In some embodiments, the light control array is distributed on the surface of the optical waveguide material in a manner such as: Fig.11 As shown in FIG. 1 , each group of light control arrays is tilted and distributed on the surface of the optical waveguide material at a preset angle other than 90 degrees. Fig.12 As shown, according to the lens projection principle, the projection position of each display unit on the display panel in the first direction under the viewing plane can be obtained in the following manner: according to the horizontal distance from the upper left corner of any display unit to the left edge of the light control array, the projection position of the display unit in the first direction under the viewing plane is obtained.

[0094] In the specific implementation process, the viewpoints in the first original disparity map can be arranged into the first coded disparity map in a variety of ways according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane. For example, a preset number of viewpoints (corresponding to the light field viewpoint density) arranged continuously in the original disparity map can be arranged into different rows of the first coded disparity map according to a certain algorithm, and after being displayed by the display panel, they will be projected to the position in the same projection unit by the projection light source emitted by the dense array light control module. For an embodiment of the method for obtaining the first coded disparity map, see Figure 8 The relevant content in will not be repeated here.

[0095] Step 703: send the first encoded disparity map to the display plane module, instructing the display plane module to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units whose viewpoints are displayed in the first original disparity map with consecutive numbers to the same projection unit in the first direction under the viewing plane.

[0096] In the specific implementation process, the module executing the disparity map encoding algorithm can send the first encoded disparity map to the relevant driving module in the display plane module in a common manner (for example, interface call or memory sharing, etc.), and the driving module displays the first encoded disparity map on the display panel. When the display plane module displays the first encoded disparity map, each backlight area formed by the dense array light control module projects the multiple different rows on the display panel covered by it and the display units with consecutive numbers in the first original disparity map whose viewpoints are displayed to the same projection unit in the first direction under the viewing plane.

[0097] In the embodiment provided by the present invention, the viewpoints in the first original disparity map are arranged into the first coded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; after the first coded disparity map is sent to the display plane module for display, each backlight area formed by the dense array light control module projects the display units with consecutive numbers in the first original disparity map and multiple different rows on the display panel covered by it to the same projection unit in the first direction under the viewing plane. Due to the use of light field reconstruction technology, multiple pixels on the display panel are moved to the same projection unit in the same horizontal direction under the viewing plane. Compared with the related art, the number of viewpoints projected to each projection unit is 1, which significantly improves the pixel density in the horizontal direction of the viewing plane, forms a dense and continuous viewpoint map arrangement in the viewing plane, and achieves a three-dimensional light field display effect with high viewpoint density without changing the resolution of the existing display panel.

[0098] Figure 8 is a flow chart of a method for obtaining a first coded disparity map provided by the present invention, such as Figure 8 As shown, the method uses the following steps to obtain the number of the viewpoint in the first original disparity map where each viewpoint position on the first encoded disparity map is filled.

[0099] Step 801: using the light field viewpoint density, perform a rounding operation on the projection position of the display unit on the display panel corresponding to the viewpoint position in the first direction under the viewing plane.

[0100] For a detailed description of the light field viewpoint density, see Figure 7 The relevant content in will not be repeated here.

[0101] In a specific implementation process, the projection position of the display unit on the display panel corresponding to any viewpoint position in the first encoded disparity map in the first direction under the viewing plane can be expressed by the ratio of the distance between the projection point of the display unit in the first direction under the viewing plane and the starting projection point of the backlight area covering the display unit in the first direction under the viewing plane to the width of the projection unit.

[0102] As an example only, if you want to get a 3D image display effect with a light field viewpoint density of 3, from a bird's-eye view, the backlight area of ​​the dense line array light control module should be Fig.12 The gray area diagonally extending from the upper left to the lower right in the display panel (displaying the first encoded disparity map). Fig.13 The three display units in the L1th row, L2th row and L3th row in the display panel shown are covered by the backlight area: the first column of the L1th row, the second column of the L2th row, and the third column of the L3th row, which are projected onto the same projection unit in the first direction under the viewing plane.

[0103] like Fig.14 As shown, the projection point of the display unit in the first column of row L1 in the first direction under the viewing plane coincides with the starting projection point of the backlight area covering the display unit in the first direction under the viewing plane, so its projection position is 0. Similarly, the projection position of the display unit in the seventh column of row L1 in the first direction under the viewing plane is 6d. w , d w is the width of the projection unit.

[0104] like Fig.15 As shown, the projection position of the display unit in the second column of row L2 in the first direction under the viewing plane is 2 / 3d w ;like Fig.16 As shown, the projection position of the display unit in the second column of row L3 in the first direction under the viewing plane is 1 / 3d w .

[0105] The projection position is rounded to an integer, that is, the projection position represented by the fraction is converted into an integer, so that the number of the viewpoint in the first original disparity map where the viewpoint position is filled is obtained according to the integer in subsequent calculation.

[0106] Step 802: According to the result of the rounding operation, obtain the number of the viewpoint in the first original disparity map where the viewpoint position is filled.

[0107] In a specific implementation process, the following formula may be used to perform a rounding operation on the projection position of the display unit on the display panel corresponding to the viewpoint position in the first direction under the viewing plane using the light field viewpoint density in the first direction, and obtain the number of the viewpoint in the first original disparity map in which the viewpoint position is filled according to the result of the rounding operation:

[0108]

[0109] Wherein, d is the inverse of the light field viewpoint density in the first direction, D is the projection position, and n is the number of the viewpoint in the first original disparity map where the viewpoint position is filled.

[0110] Just as an example, Fig.12 In the first encoded disparity map shown in FIG. 1 , the light field viewpoint density and the projection position are substituted into formula (7) to calculate the number of the viewpoints in the first original disparity map filled in the first column of row L1: The number of viewpoints in the first original disparity map filled with the second column of row L2 is: The number of the viewpoints in the first original disparity map filled in the second column of row L3 is: The number of viewpoints in the first original disparity map filled in the 7th column of row L3 is:

[0111]

[0112] Step 803: fill each viewpoint in the first original disparity map into the first encoded disparity map according to the number of the viewpoint in the first original disparity map where each viewpoint position is filled.

[0113] In a specific implementation process, after the number of the viewpoints in the first original disparity map in which each viewpoint position is filled in the first encoded disparity map is obtained by using formula (7), each viewpoint in the first original disparity map can be filled into the first encoded disparity map according to the number of the viewpoint in the first original disparity map in which each viewpoint position is filled in the first encoded disparity map. After the first encoded disparity map is loaded onto the display panel for display, under the projection of the background light source formed by the dense array light control module, the following can be obtained: Fig.13 The shown 3D display effect shows a high density of 21 viewpoints in 7 projection units.

[0114] In the embodiment provided by the present invention, the number of viewpoints that can be displayed in the first direction under the viewing plane is as follows: If R (the horizontal displacement of two rows of backlight areas) is an integer, then: x =ρn; if R is not an integer, then: Wherein [R / b] represents the integer of R / b, n is the number of display units in the first direction included in the display panel, and ρ is the light field viewpoint density (see the relevant content in step 702, which will not be repeated here).

[0115] Fig. 9 This is the second flow chart of the three-dimensional scene display method based on digital twins provided by the present invention, such as Fig. 9 As shown, the method includes the following:

[0116] Step 901: Obtain a second original disparity map.

[0117] The second original disparity map is an original disparity map having disparity in multiple directions (eg, horizontal direction and vertical direction, etc.). For example, the second original disparity map may be an original disparity map having disparity in both the horizontal direction and the vertical direction.

[0118] In a specific implementation process, the module executing the disparity map encoding algorithm in the processor may obtain the second original disparity map from other processing modules through common methods (eg, interface call or memory sharing), which is not limited by the description of this specification.

[0119] For each of the multiple directions, perform the following operations:

[0120] Step 902: Arrange the viewpoints corresponding to the directions in the second original disparity map into the second encoded disparity map according to the light field viewpoint density in the direction and the projection positions of the display units on the display panel in the direction under the viewing plane; wherein the projection positions of the display units on the display panel in the direction under the viewing plane are determined by the distribution of the light control array on the surface of the optical waveguide material.

[0121] For a detailed description of the light field viewpoint density and projection position, see Figure 7 Related content in ,.

[0122] As an example, for a preferred digital twin display device with parallax in both horizontal and vertical directions, the dense array light control module part is pasted on the optical waveguide material panel as shown in FIG. Figure 4 The lens array shown is composed of multiple dot lenses of the same specifications, so that the light emitted by the dense array light control module is arranged in a dot array. The size of the point light source generated by the dot lens should be consistent with or smaller than the screen pixel size. The projection position of each display unit on the display panel of the digital twin display device in a certain direction under the viewing plane is determined by the distribution of the dot lenses in that direction.

[0123] In the specific implementation process, the viewpoints in the second original disparity map can be arranged into the second encoded disparity map in a variety of ways according to the light field viewpoint density in a certain direction and the projection position of each display unit on the display panel in this direction under the viewing plane. For example, a preset number of viewpoints (corresponding to the light field viewpoint density) arranged continuously in the original disparity map can be arranged into different rows of the second encoded disparity map according to a certain algorithm, and after being displayed by the display panel, they will be projected by the projection light source emitted by the dense array light control module to the position in the same projection unit in this direction. For an embodiment of the method for obtaining the second encoded disparity map, see Fig.10 The relevant content in will not be repeated here.

[0124] Step 903: Send the second encoded disparity map to the display plane module, and instruct the display plane module to display the second encoded disparity map so that the backlight area projects multiple display units located in different rows on the display panel covered by it and whose displayed viewpoints are in any direction with consecutive numbers in the second original disparity map to the same projection unit in the corresponding direction under the viewing plane.

[0125] In a specific implementation, the module executing the disparity map encoding algorithm can send the second encoded disparity map to the relevant driving module in the display plane module in a common manner (such as interface call or memory sharing, etc.), and the driving module displays the second encoded disparity map on the display panel. When the display plane module displays the second encoded disparity map, the backlight area formed by the dense array light control module projects the display units in any direction whose viewpoints are located in different rows on the display panel covered by it and whose display units are numbered consecutively in the second original disparity map to the same projection unit in the corresponding direction under the viewing plane.

[0126] Fig.10 is a flow chart of a method for obtaining a second coded disparity map provided by the present invention, such as Fig.10 As shown, the method uses the following method to obtain the number of the viewpoint in the second original disparity map where each viewpoint position on the second encoded disparity map is filled.

[0127] Step 1001, using the light field viewpoint density, rounding the projection position of the display unit on the display panel corresponding to the viewpoint position in the direction below the viewing plane; wherein the projection position is represented by the ratio of the distance between the projection point of the display unit in the direction below the viewing plane and the starting projection point of the backlight area covering the display unit in the direction below the viewing plane to the width of the display unit.

[0128] In a specific implementation, the projection position of the display unit on the display panel corresponding to any viewpoint position in the second encoded disparity map in a certain direction under the viewing plane can be represented by the ratio between the distance between the projection point of the display unit in the direction under the viewing plane and the starting projection point of the backlight area covering the display unit in the direction under the viewing plane and the width of the projection unit. For a specific example of obtaining the projection position of the display unit on the display panel corresponding to any viewpoint position in a certain direction under the viewing plane, see step 801. Fig.14 , Fig.15 as well as Fig.16 The examples are not repeated here.

[0129] The projection position is rounded to an integer, that is, the projection position represented by the fraction is converted into an integer, so that the number of the viewpoint in the second original disparity map in which the viewpoint position is filled is obtained according to the integer in subsequent calculation.

[0130] Step 1002: According to the result of the rounding operation, obtain the number of the viewpoint in the second original disparity map where the viewpoint position is filled.

[0131] In a specific implementation process, for the second original disparity map having disparity in both the horizontal and vertical directions, the following formula can be used to perform a rounding operation on the projection position of the display unit on the display panel corresponding to the viewpoint position in any direction under the viewing plane using the light field viewpoint density, and a step of obtaining the number of the viewpoint in the second original disparity map in which the viewpoint position is filled based on the result of the rounding operation.

[0132]

[0133] Among them, d1 is the inverse of the light field viewpoint density in the horizontal direction, D1 is the projection position in the horizontal direction, and x is the number of the viewpoint in the second original disparity map in which the viewpoint position in the horizontal direction is filled; d2 is the inverse of the light field viewpoint density in the vertical direction, D2 is the projection position in the vertical direction, and y is the number of the viewpoint in the second original disparity map in which the viewpoint position in the vertical direction is filled.

[0134] Step 1003: fill each viewpoint in the second original disparity map into the second encoded disparity map according to the number of the viewpoint in the second original disparity map in which each viewpoint position is filled.

[0135] In a specific implementation process, after obtaining the number of viewpoints in the second original disparity map in which each viewpoint position in the second encoded disparity map is filled using formulas (8) and (9), each viewpoint in the second original disparity map can be filled into the second encoded disparity map according to the number of viewpoints in the second original disparity map in which each viewpoint position in the second encoded disparity map is filled.

[0136] The three-dimensional scene display system based on digital twin provided by the present invention is described below. The three-dimensional scene display system based on digital twin described below and the three-dimensional scene display method based on digital twin described above can be referenced to each other.

[0137] Fig.17It is a structural schematic diagram of the three-dimensional scene display system based on digital twins provided by the present invention. The three-dimensional scene display system based on digital twins is applied to the digital twin display device, which includes a dense array light control module and a display plane module arranged in parallel in sequence; wherein the display plane module includes a display panel with point-to-point light control; the dense array light control module includes an opaque optical waveguide material and a light control array bonded to the optical waveguide material in a preset distribution manner; the light emitted by the backlight source of the digital twin display device, after passing through the bonding surface of the light control array and the optical waveguide material, is refracted by the light control array to form multiple backlight areas. Fig.17 As shown, the system includes the following modules:

[0138] The acquisition module 1710 is used to acquire a first original disparity map; the first original disparity map is a disparity map in a first direction.

[0139] The encoding module 1720 is used to arrange the viewpoints in the first original disparity map into the first encoded disparity map according to the light field viewpoint density in the first direction and the projection positions of the display units on the display panel in the first direction under the viewing plane; wherein the projection positions of the display units on the display panel in the first direction under the viewing plane are determined by the distribution of the light control array on the surface of the optical waveguide material.

[0140] The sending module 1730 is used to send the first encoded disparity map to the display plane module, instructing the display plane module to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units with consecutive numbers of the viewpoints displayed in the first original disparity map to the same projection unit in the first direction under the viewing plane.

[0141] The three-dimensional scene display system based on digital twin provided by the present invention arranges each viewpoint in the first original disparity map into the first coded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; sends the first coded disparity map to the display plane module, instructing the display plane module to display the first coded disparity map. Since the positions of each viewpoint in the first coded disparity map are determined according to the projection position of the display unit on the display panel corresponding to it in the first direction under the viewing plane, after the first coded disparity map is displayed by the display panel, under the projection of the backlight area of ​​the dense array light control module, multiple different rows on the display panel and the display units whose displayed viewpoints are numbered continuously in the first original disparity map can be projected into the same projection unit in the first direction under the viewing plane, thereby significantly improving the viewpoint density of the three-dimensional light field without changing the resolution of the light field digital twin display device.

[0142] See also Fig.18 , Fig.18 Schematic diagram of an electronic device provided by an embodiment of the present invention. Fig.18 As shown, an embodiment of the present invention provides an electronic device 1800, including a memory 1810, a processor 1820, and a computer program 1811 stored in the memory 1810 and executable on the processor 1820. When the processor 1820 executes the computer program 1811, the following steps are implemented:

[0143] Obtaining a first original disparity map; the first original disparity map is a disparity map in a first direction;

[0144] Arranging each viewpoint in the first original disparity map into the first coded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; wherein the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material;

[0145] The first encoded disparity map is sent to the display plane module, instructing the display plane module to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units whose viewpoints are displayed in the first original disparity map with consecutive numbers into the same projection unit in the first direction under the viewing plane.

[0146] See also Fig.19 , Fig.19 Schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Fig.19 As shown, this embodiment provides a computer-readable storage medium 1900, on which a computer program 1911 is stored. When the computer program 1911 is executed by a processor, the following steps are implemented:

[0147] Obtaining a first original disparity map; the first original disparity map is a disparity map in a first direction;

[0148] Arranging each viewpoint in the first original disparity map into the first coded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; wherein the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material;

[0149] The first encoded disparity map is sent to the display plane module, instructing the display plane module to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units whose viewpoints are displayed in the first original disparity map with consecutive numbers into the same projection unit in the first direction under the viewing plane.

[0150] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0152] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A three-dimensional scene display method based on digital twins, characterized in that: Applied to a digital twin display device, the digital twin display device includes a dense array light control module and a display plane module arranged in parallel in sequence; the display plane module includes a display panel with point-to-point light control; the dense array light control module includes an opaque optical waveguide material and a light control array attached to the optical waveguide material in a preset distribution manner; the light emitted by the backlight light source of the digital twin display device, after passing through the bonding surface of the light control array and the optical waveguide material, is refracted by the light control array to form multiple backlight areas, and the three-dimensional scene display method based on digital twins includes: Acquire a first original disparity map; the first original disparity map is a disparity map in a first direction; Arranging each viewpoint in the first original disparity map into a first coded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; wherein the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material; The first encoded disparity map is sent to a display plane module, and the display plane module is instructed to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units whose display viewpoints are consecutively numbered in the first original disparity map to the same projection unit in the first direction under the viewing plane.

2. The three-dimensional scene display method based on digital twin according to claim 1, characterized in that: The arranging each viewpoint in the first original disparity map into the first encoded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane comprises: The number of the viewpoint in the first original disparity map at which each viewpoint position on the first encoded disparity map is filled is obtained in the following manner: Using the light field viewpoint density in the first direction, a rounding operation is performed on the projection position of the display unit on the display panel corresponding to the viewpoint position in the first direction under the viewing plane; wherein the projection position is represented by a ratio between a distance between a projection point of the display unit in the first direction under the viewing plane and a starting projection point of a backlight area covering the display unit in the first direction under the viewing plane and a width of the projection unit; According to the result of the rounding operation, obtain the number of the viewpoint in the first original disparity map in which the viewpoint position is filled; Fill each viewpoint in the first original disparity map into the first encoded disparity map according to the number of the viewpoint in the first original disparity map in which each viewpoint position is filled.

3. The three-dimensional scene display method based on digital twin according to claim 2, characterized in that: The following formula is used to perform a rounding operation on the projection position of the display unit on the display panel corresponding to the viewpoint position in the first direction under the viewing plane by using the light field viewpoint density in the first direction, and obtain the number of the viewpoint in the first original disparity map in which the viewpoint position is filled according to the result of the rounding operation: Wherein, d is the inverse of the light field viewpoint density in the first direction, D is the projection position, and n is the number of the viewpoint in the first original disparity map where the viewpoint position is filled.

4. The three-dimensional scene display method based on digital twin according to claim 3 is characterized in that: The distribution of the light control array on the surface of the optical waveguide material is as follows: Each group of the light control arrays is tilted and distributed on the surface of the optical waveguide material at a preset angle other than 90 degrees; The projection position of each display unit on the display panel in the first direction below the viewing plane is obtained in the following manner: The projection position of the display unit in the first direction below the viewing plane is obtained according to the horizontal distance between the upper left corner of any display unit and the left edge of the light control array.

5. The three-dimensional scene display method based on digital twin according to claim 1, characterized in that: The three-dimensional scene display method based on digital twins also includes: Acquire a second original disparity map; the second original disparity map is a disparity map having disparities in multiple directions; For each of the multiple directions, perform the following operations: Arrange each viewpoint corresponding to the direction in the second original disparity map into a second coded disparity map according to the light field viewpoint density in the direction and the projection position of each display unit on the display panel in the direction under the viewing plane; wherein the projection position of each display unit on the display panel in the direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material; The second encoded disparity map is sent to a display plane module, and the display plane module is instructed to display the second encoded disparity map so that the backlight area projects multiple display units on the display panel covered by it, which are located in different rows and whose displayed viewpoints are in any direction with consecutive numbers in the second original disparity map, into the same projection unit in the direction under the viewing plane.

6. The three-dimensional scene display method based on digital twin according to claim 5 is characterized in that: Arranging each viewpoint corresponding to the direction in the second original disparity map into the second encoded disparity map according to the light field viewpoint density in the direction and the projection position of each display unit on the display panel in the direction under the viewing plane includes: The number of the viewpoint in the second original disparity map at which each viewpoint position on the second encoded disparity map is filled is obtained in the following manner: Using the light field viewpoint density in the direction, a rounding operation is performed on the projection position of the display unit on the display panel corresponding to the viewpoint position in the direction under the viewing plane; wherein the projection position is represented by a ratio between a distance between a projection point of the display unit in the direction under the viewing plane and a starting projection point of a backlight area covering the display unit in the direction under the viewing plane and a width of the projection unit in the direction; According to the result of the rounding operation, obtain the number of the viewpoint in the second original disparity map in which the viewpoint position is filled; Fill each viewpoint in the second original disparity map into the second encoded disparity map according to the number of the viewpoint in the second original disparity map in which each viewpoint position is filled.

7. A three-dimensional scene display system based on digital twins, characterized in that: Applied to a digital twin display device, the digital twin display device includes a dense array light control module and a display plane module arranged in parallel in sequence; the display plane module includes a display panel with point-to-point light control; the dense array light control module includes an opaque optical waveguide material and a light control array attached to the optical waveguide material in a preset distribution manner; the light emitted by the backlight light source of the digital twin display device, after passing through the bonding surface of the light control array and the optical waveguide material, is refracted by the light control array to form multiple backlight areas, and the three-dimensional scene display system based on digital twin includes: An acquisition module, configured to acquire a first original disparity map; the first original disparity map is a disparity map in a first direction; an encoding module, configured to arrange each viewpoint in the first original disparity map into a first encoded disparity map according to the light field viewpoint density in the first direction and the projection position of each display unit on the display panel in the first direction under the viewing plane; wherein the projection position of each display unit on the display panel in the first direction under the viewing plane is determined by the distribution mode of the light control array on the surface of the optical waveguide material; A sending module is used to send the first encoded disparity map to a display plane module, instructing the display plane module to display the first encoded disparity map so that the backlight area projects multiple different rows on the display panel covered by it and the display units whose display viewpoints are consecutively numbered in the first original disparity map into the same projection unit in a first direction under the viewing plane.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the three-dimensional scene display method based on digital twins is implemented as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional scene display method based on digital twins as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional scene display method based on digital twins as described in any one of claims 1 to 6 is implemented.

Images