A freely expandable stereo digital sandbox system and light field rendering method

Abstract

The present invention proposes a freely expandable stereoscopic digital sandbox system and light field rendering method, wherein the system includes a data encoding module, a data decoding module, a human eye tracking module, a human-computer interaction module, a light field image rendering module, a driving module and a display module. On the one hand, the seamless splicing of high-density display screens is achieved through collimated backlight control technology and an optical microstructure array, so that the display size of the digital sandbox can be freely expanded. On the other hand, in conjunction with human eye tracking technology, on the basis of neural radiation field three-dimensional static scene rendering, in conjunction with traditional graphics rendering pipeline technology, the fusion of static and dynamic scenes is further realized, so that the 3D scenes of implicit expression and explicit expression can be rendered and displayed in real time in a unified framework, and at the same time, the amount of image rendering data is effectively reduced by rendering data only at the human eye viewing position. Combining the above two technologies, observers can obtain a better naked-eye stereoscopic immersion at any viewing angle.

Description

Technical Field

[0001] The present application relates to the field of stereoscopic display and artificial intelligence, and specifically to a freely expandable stereoscopic digital sandbox system and a light field rendering method. Background Art

[0002] Human eyes can perceive the three-dimensional world and see the stereoscopic effect because when humans observe the three-dimensional world, the left and right eyes see slightly different images. The brain can merge the two slightly different images into a three-dimensional effect, so that people can get a three-dimensional sense. 3D display technology uses this principle to allow the observer's left and right eyes to observe two slightly different images, thereby obtaining a three-dimensional sense. At present, 3D display technology can be divided into two categories: naked eye type and glasses type. Glasses type can be divided into shutter glasses, polarized glasses, VR helmets and chromatic aberration glasses. Naked eye type can be divided into three types: parallax barrier (Barrier) technology, lenticular lens (Lenticular Lens) technology and directional light source (Directional Backlight) technology.

[0003] Parallax barrier technology controls the generation of opaque stripes. When the image that should be seen by the left eye is displayed on the LCD screen, the opaque stripes will block the right eye; similarly, when the image that should be seen by the right eye is displayed on the LCD screen, the opaque stripes will block the left eye. By separating the visible images of the left and right eyes, the viewer can see 3D images. The main advantage of this technology is that it is compatible with LCD liquid crystal technology, so it has advantages in mass production and cost. The main disadvantage is that the picture brightness is low, and the resolution will decrease inversely as the number of images broadcast by the display at the same time increases.

[0004] The principle of cylindrical lens technology is to add a layer of cylindrical lens in front of the LCD screen. The image plane of the LCD screen is located on the focal plane of the lens, so that the pixel of the image under each cylindrical lens is divided into several sub-pixels, so that the lens can project each sub-pixel in different directions. So when the eyes watch the display from different angles, they see different sub-pixels. The main advantage of this technology is that the 3D display effect is better and the brightness is not affected; the main disadvantage is that the related manufacturing technology is incompatible with the existing LCD liquid crystal process, and new equipment and production lines need to be invested, and the resolution of the image produced by this technology is not high.

[0005] The directional light source technology is equipped with two sets of LEDs, and with a fast-response LCD panel and driving method, the 3D content is entered into the viewer's left and right eyes in a sequenced manner to exchange images to produce parallax, thereby allowing the human eye to feel the three-dimensional effect. The main advantages of this technology are that the resolution and light transmittance can be guaranteed, it will not affect the existing design framework, and the 3D display effect is excellent; the main disadvantage is that the product is immature.

[0006] The existing sand table system mainly uses sand for piling or 3D printing physical sand table, which leads to low efficiency of military simulation, inability to refresh data in real time, and requires a lot of manpower, material and financial resources, and cannot meet the needs of real-time changes in the ever-changing battlefield environment. It has poor adaptability to battlefield information data updates. In addition, there are holographic card-type stereo solutions, which have the advantages of high resolution, light weight and portability, but the disadvantages are low intelligence and interactive experience, limited projection distance and size, poor equipment adaptability, inability to refresh in real time, high requirements for working environment, and unsuitable for outdoor application environments. Therefore, it cannot adapt to the field combat needs of equipment moving with tanks at any time in a war environment, and cannot support equipment to adapt to field combat and dynamic battlefield environments. At present, the more mature technology is the glasses-type stereo sand table, which has the advantages of good stereo immersion and low cost. The disadvantages are poor convenience of use, uncomfortable viewing comfort, and inconvenience for face-to-face command and communication of users. The digital stereo sand table can dynamically interact and refresh, and has good stereo immersion. The level of integrated technology is high, and the degree of domestic and foreign technology development is low. At present, domestic and foreign research is still in a blank state.

[0007] Neural Radiance Fields (NeRF) was first proposed by Ben Mildenhall et al. in 2020. NeRF is a computer graphics technology that can generate highly realistic 3D scenes from 2D images through the learning of neural networks. The working principle of NeRF is self-supervised, and the input is the 3D coordinate position (x, y, z) and direction (θ, ), using color as a supervision term and using a fully connected network for model training. By training data on limited input views, high-quality rendering can be generated with a smaller data set. Compared with traditional methods that use discretized grids or voxels to represent scenes, NeRF's continuous function representation has advantages and can be rendered from any angle, producing amazing high-quality rendering effects. However, NeRF currently achieves good rendering effects for static scenes, but has certain limitations for dynamic scenes.

[0008] The existing three-dimensional model expression methods are divided into explicit expression and implicit expression. The explicit expression method includes voxel-based, grid-based, and point cloud-based methods. Its advantage is that it can intuitively display three-dimensional geometric objects, but the objects expressed explicitly are discrete and discontinuous, and the amount of three-dimensional scene expression information data it stores is extremely large; the implicit expression method is to describe a 3D scene with a function, such as the neural radiation field method. Its advantage is that the scene is continuous, the details can be finely modeled, the resolution can theoretically reach infinity, and it occupies a small space, but its disadvantage is that there are certain limitations in the description of dynamic scenes, and it is not easy to render dynamic objects.

[0009] The prior art has the following disadvantages:

[0010] 1. The traditional stacked sand table is expensive, cannot be refreshed, and has poor timeliness, which cannot meet the application requirements of the modern command system; the glasses-type three-dimensional sand table is not convenient to use, the viewing comfort is uncomfortable, and it is not conducive to face-to-face command and communication for users; the holographic card-type three-dimensional solution has limited projection distance and size, poor equipment adaptability, and high requirements for the working environment, and is not suitable for outdoor application environments.

[0011] 2. Glasses-type 3D display technology requires additional auxiliary equipment and is prone to visual fatigue; light barrier technology has low screen brightness, and the resolution decreases inversely as the number of images broadcast by the display at the same time increases.

[0012] 3. Most of the existing screen splicing technologies cannot eliminate the black borders of the spliced ​​screen, affecting the viewing effect.

[0013] 4. The neural radiation field has certain limitations for dynamic scenes and it is not easy to render dynamic objects.

[0014] 5. The display of the three-dimensional model takes up a large space and the scene is discontinuous.

[0015] 6. The generation of light field images faces difficulties such as large data volume and complex calculations. Summary of the invention

[0016] To solve the above problems, the present application provides a freely scalable stereo digital sandbox system and a light field rendering method.

[0017] According to the first aspect of the present application, the present application provides a freely expandable three-dimensional digital sandbox system, comprising:

[0018] Data encoding module, data decoding module, human eye tracking module, human-computer interaction module, light field image rendering module, driving module and display module. The data encoding module of the system is connected to the data decoding module and the human-computer interaction module through the HDMI / DP interface, and is used to transmit the three-dimensional model data and interaction instructions to and from the server. The light field image rendering module performs light field image rendering according to the received three-dimensional model data, human-computer interaction instructions and the real-time viewing position coordinates of the human eye. The light field image rendering module and the display module are connected through the driving module, and the driving module is used to display the light field image in three dimensions on the display module. The freely expandable display screen is seamlessly spliced ​​with multiple sub-screens, which can make the display size of the three-dimensional digital sandbox infinitely expandable.

[0019] The freely expandable three-dimensional digital sandbox system is characterized in that the user can not only interact with the digital sandbox through the 2D display screen on the PC, but also set the working mode of the digital sandbox, such as switching between 2D and 3D display models. The data encoding module is used to encode the implicitly or explicitly expressed three-dimensional model and transmit it losslessly through the standard HDMI / DP.

[0020] The freely expandable three-dimensional digital sandbox system is characterized in that the light field image rendering module adopts a modular light field image rendering hardware platform design, which is used for parallel rendering of light field images to improve the rendering speed.

[0021] The freely expandable three-dimensional digital sandbox system is characterized in that the driving module adopts an array design for partitioning and controlling the high-density liquid crystal display panel. The entire display screen can be driven by multiple sub-driving modules, and each sub-driving module can drive multiple sub-display screens to refresh.

[0022] The freely expandable 3D digital sand table system is characterized in that the display module is connected to the driving module and is used to map the display content to the liquid crystal panel and display 3D information, so that viewers can see the 3D scene with naked eyes. The module is composed of multiple sub-screens. The collimated backlight technology and optical microstructure array are used to achieve seamless splicing of high-density display screens, so that the display size of the 3D digital sand table can be freely expanded.

[0023] According to a second aspect of the present application, a freely scalable stereoscopic digital sandbox light field rendering method includes:

[0024] It includes two stages: implicit conversion of the model and hybrid rendering of viewpoints. The implicit conversion of the model includes the data set acquisition step and the neural radiation field model training step. The specific steps are described as follows:

[0025] Dataset acquisition step: Provide the original training data set for the neural rendering pipeline according to the target scene expressed by the display. The data set includes multi-view images and corresponding camera pose information;

[0026] Neural radiation field model training step: preprocessing the data set as the input of the neural network, using a fully connected network to train the model, obtaining an implicit expression of the model, and completing the model training;

[0027] Hybrid rendering steps of viewpoints: Based on the viewing position of the human eye, a hybrid rendering pipeline is used to combine the neural rendering pipeline with the traditional graphics rendering pipeline to merge static and dynamic scenes, so that 3D scenes with implicit and explicit expressions can be rendered and displayed in real time in a unified framework. The online editing function of the 3D scene is added to mark and modify the 3D scene in real time, render the image to be displayed, and increase the readability and usability of the target scene.

[0028] The freely scalable three-dimensional digital sandbox light field rendering method is characterized in that, in the data set acquisition step, the data can be a real-life map to be displayed or a virtual map scene.

[0029] The freely scalable stereoscopic digital sandbox light field rendering method is characterized in that in the perspective rendering step, a hybrid rendering pipeline is used for image generation, wherein the neural radiation field is used to render static map scenes, and the traditional rendering pipeline is used to render editable dynamic scenes, such as map identifiers.

[0030] The freely expandable three-dimensional digital sandbox light field rendering method is characterized in that the light field rendering method is a lightweight rendering algorithm based on human eye tracking data, the human eye tracking algorithm uses infrared guidance to track, the infrared camera is used to quickly locate the area of ​​interest of the face, the visible light camera accurately and quickly locates the human eye according to the area of ​​interest, and sends the human eye position coordinates to the light field image rendering module in real time, and the image rendering module only performs image rendering at the human eye viewing position, thereby reducing the data volume of the light field rendering algorithm.

[0031] The freely scalable three-dimensional digital sandbox light field rendering method is characterized in that the light field rendering method is a real-time rendering method.

[0032] The beneficial effects of this application are:

[0033] According to the above-mentioned embodiment, a freely expandable stereoscopic digital sandbox system and light field rendering method are adopted. On the one hand, the seamless splicing of high-density display screens is achieved through collimated backlight control technology and optical microstructure arrays, so that the display size of the digital sandbox can be freely expanded. On the other hand, in conjunction with human eye tracking technology, on the basis of neural radiation field three-dimensional static scene rendering, in conjunction with traditional graphics rendering pipeline technology, the fusion of static and dynamic scenes is further realized, so that the 3D scenes of implicit expression and explicit expression can be rendered and displayed in real time in a unified framework, and at the same time, the amount of image rendering data is effectively reduced by rendering data only at the human eye viewing position. Combining the above two technologies, observers can obtain a better naked-eye stereoscopic immersion at any viewing angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a freely expandable three-dimensional digital sandbox system is provided for this application;

[0035] Figure 2 A prototype conceptual diagram of a freely expandable three-dimensional digital sandbox system for this application;

[0036] Figure 3 A schematic diagram of a liquid crystal panel partition drive for a freely expandable three-dimensional digital sandbox system for this application;

[0037] Figure 4 A schematic diagram of seamless screen splicing of a freely expandable three-dimensional digital sandbox system for this application;

[0038] Figure 5 A flow chart of a freely scalable three-dimensional digital sand table light field rendering method for this application;

[0039] Figure 6 A schematic diagram of eye tracking acceleration based on a freely scalable stereo digital sandbox light field rendering method for this application. DETAILED DESCRIPTION

[0040] The present application is further described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0041] Please refer to Figure 1 The present application discloses a freely expandable three-dimensional digital sandbox system (hereinafter referred to as digital sandbox), which includes a data encoding module 11, a data decoding module 12, a human eye tracking module 13, a human-computer interaction module 14, a light field image rendering module 15, a driving module 16 and a display module 17, which are described below respectively.

[0042] The data encoding module 11 is used to encode the implicitly or explicitly expressed three-dimensional model and transmit it losslessly through the standard HDMI / DP, so as to provide the digital sandbox with relevant information such as stereoscopic display data and human-computer interaction instructions. In a specific embodiment, the data encoding module 11 is a computer with high stability and high computing power. For example, the data encoding module 11 can be a server. Of course, in some other embodiments, the data encoding module 11 can also be a conventional computer or a computer with an HDMI / DP video interface. The sent stereo data can be a traditional three-dimensional point cloud or patch model, or an implicitly expressed three-dimensional model. The data encoding module encodes the implicitly or explicitly expressed three-dimensional model and transmits it losslessly through the standard HDMI / DP.

[0043] The data decoding module 12 is connected to the data encoding module 12, and is used to receive and decode the stereoscopic display data, and transmit the three-dimensional model data to the light field image rendering module 15. In a specific example, the data decoding module 12 decodes the received encoded data to restore the implicitly or explicitly expressed three-dimensional model, and transmits the three-dimensional model data to the light field image rendering module 15.

[0044] The eye tracking module 13 includes: an infrared camera and two visible light cameras. The eye tracking algorithm uses infrared guidance to track. The infrared camera is used to quickly locate the area of ​​interest of the face. The visible light camera accurately and quickly locates the eye according to the area of ​​interest, and sends the eye position coordinates to the light field image rendering module in real time.

[0045] The human-computer interaction module 14 is connected to the data encoding module 12, receives the human-computer interaction instruction sent by the data encoding module and acts on the light field image rendering module 15, so that it performs corresponding rendering according to the instruction.

[0046] The light field image rendering module 15 is used to generate the light field image required to be displayed by the display module 17. In a specific embodiment, the light field image rendering module 15 performs light field image rendering according to the received three-dimensional model data, the real-time human eye coordinate position and the corresponding human-computer interaction instruction. During the rendering process, multiple rendering units perform light field rendering simultaneously to accelerate the generation of light field images.

[0047] The driving module 16 is connected to the light field image rendering module 15 and the display module 17, and adopts an array design to perform partition refresh and control on the high-density liquid crystal display panel. The entire display screen can be driven by multiple sub-driving modules, and each sub-driving module can drive multiple sub-display screens. In a specific embodiment, the rendering unit sends different images to different driving units through a network cable, and each driving unit controls the display of four sub-display screens respectively. The resolution of each driving unit is 3840×2160, and one driving unit can drive four sub-display screens.

[0048] The display module 17 is connected to the driving module 16 and is used to display stereoscopic information, so that viewers can see the stereoscopic effect with their naked eyes. The module uses collimated backlight technology and optical microstructure modules to achieve seamless splicing of high-density display screens, allowing the display size of the digital sand table to be freely expanded. In a specific embodiment, the display module 17 is a composite display carrier composed of a high-resolution liquid crystal display panel and an optical microstructure. The digital sand table is composed of multiple sub-screens, and the resolution of each sub-screen is 1920×1080. Figure 4As shown, the backlight unit 47 lights up the sub-screen 41 through the collimated backlight module 46. Due to the influence of the edge of each sub-screen 41, direct screen splicing will have obvious black edges, which seriously affects the overall viewing effect. The present application adds an optical microstructure 42 above the screen to regulate the light beam, expands the light beam to cover the edge of the sub-screen, and then splices the sub-screens to achieve a seamless splicing effect. Finally, a lens array 45 is added above the spliced ​​screen to enable it to display three-dimensional information.

[0049] Accordingly, the present application also proposes a freely scalable three-dimensional digital sandbox light field rendering method, such as Figure 5 As shown, it includes the following steps.

[0050] S51 Dataset acquisition step: The target scene to be displayed can provide the original training data set for the neural rendering pipeline. The data set includes images and corresponding camera pose information, such as Figure 5 In a specific embodiment, the target scene can be a virtual three-dimensional map scene in computer software or a real scene in a real environment, and the corresponding acquisition camera can be a virtual camera or a real camera. According to the target scene expressed by the display, an original training data set is provided for the neural rendering pipeline, and the data set includes multi-view images and corresponding camera pose information.

[0051] S52 neural radiation field model training step: pre-process the data set as the input of the network, use the fully connected network to train the model, obtain the implicit expression of the model, and complete the model training. In a specific example, according to the position and direction of the viewing angle, color is used as a supervision item, and the fully connected network is used to train the model to obtain the model parameters of the fully connected network.

[0052] S53 Hybrid rendering step of viewpoint: Based on the viewing position of human eyes, a hybrid rendering pipeline is used to combine the neural rendering pipeline with the traditional graphics rendering pipeline to merge static and dynamic scenes, so that the 3D scenes of implicit expression and explicit expression can be rendered and displayed in real time in a unified framework. The online editing function of the 3D scene is added to mark and modify the 3D scene in real time, render the image to be displayed, and increase the readability and usability of the target scene.

[0053] The present application proposes a freely expandable stereoscopic digital sandbox system and light field rendering method. On the one hand, the seamless splicing of high-density display screens is achieved through collimated backlight control technology and optical microstructure arrays, so that the display size of the digital sandbox can be freely expanded. On the other hand, in conjunction with human eye tracking technology, on the basis of neural radiation field three-dimensional static scene rendering, in conjunction with traditional graphics rendering pipeline technology, the fusion of static and dynamic scenes is further realized, so that the 3D scenes of implicit expression and explicit expression can be rendered and displayed in real time in a unified framework, and at the same time, the amount of image rendering data is effectively reduced by rendering data only at the viewing position of the human eye. Combining the above two technologies, observers can obtain a better naked-eye stereoscopic immersion at any viewing angle. The following is a detailed analysis.

[0054] like Figure 1 As shown, it is a schematic diagram of a freely expandable stereoscopic digital sandbox system of the present application, in which the data encoding module 11 is used to provide the three-dimensional model required for display and the human-computer interaction operation instructions, wherein the three-dimensional model can be a traditional three-dimensional point cloud or patch model, or an implicitly expressed three-dimensional model. The implicitly or explicitly expressed three-dimensional model is encoded and transmitted losslessly through the standard HDMI / DP, and the data decoding module 12 decodes the received encoded data to restore the three-dimensional model. According to the received three-dimensional model data, combined with the eye position coordinates captured in real time by the eye tracking module 13 and the interaction instructions received by the human-computer interaction module 14, the light field image rendering module 15 is composed of multiple rendering units and performs light field rendering to accelerate the generation of light field images. The generated light field image is then displayed in partitions on a high-density liquid crystal display panel by an array-designed driving module 16. The driving unit has a routing function. The rendering unit sends different images to different driving units through network cables. Each driving unit controls four sub-display screens respectively. The resolution of each driving unit is 3840×2160, and the resolution of each sub-display screen is 1920×1080. One driving unit can drive four sub-display screens, and all sub-display screens are spliced ​​to form a display module 17.

[0055] like Figure 2 As shown, this is a prototype conceptual diagram of a freely expandable three-dimensional digital sandbox system of the present application. The prototype as a whole can be divided into three layers. The bottom layer 21 includes a data encoding module, a data decoding module, and a human-computer interaction module; the middle layer 22 includes a light field image rendering module and a driving module; the top layer 23 includes a display module 24 and a human eye tracking module 25.

[0056] like Figure 3As shown, it is a data flow diagram of a freely expandable three-dimensional digital sandbox system of the present application. The three-dimensional model data and human-computer interaction instructions are transmitted by the data encoding module 31 to the data decoding module 32 and the human-computer interaction module 34 through the standard HDMI / DP, and combined with the eye position coordinates captured by the eye tracking module 33, they act on the light field image rendering module 35 at the same time. The light field image rendering module 35 is composed of multiple rendering units that perform light field rendering in parallel. The generated light field image is then displayed in partitions on the high-density liquid crystal display panel by the array-designed driving module 36. The driving unit has a routing function. The rendering unit sends different images to different driving units through network cables. Each driving unit controls four sub-display screens respectively. All sub-display screens are spliced ​​to form a display module 37.

[0057] like Figure 4 As shown, it is a schematic diagram of seamless screen splicing of a freely expandable three-dimensional digital sandbox system of the present application. The backlight unit 47 lights up the sub-screen 41 through the collimated backlight module 46. Each sub-screen unit 41 includes an LCD panel 43 in the middle of the screen and a screen frame 44. Due to the influence of the frame of each sub-screen 41, direct screen splicing will have obvious black edges, which seriously affects the overall viewing effect. The present application adds an optical microstructure 42 above the screen to regulate the light beam, expands the light beam to cover the edge of the sub-screen, and then splices the sub-screens to achieve a seamless splicing effect. Finally, a lens array 45 is added above the spliced ​​screen to enable it to display three-dimensional information.

[0058] like Figure 5 As shown, this is a flow chart of a freely expandable stereoscopic digital sandbox light field rendering method of the present application, and the algorithm includes a target scene acquisition step S51, a neural radiation field training step S52, and a perspective rendering step S53. The entire hybrid rendering pipeline includes a neural rendering pipeline and a traditional graphics rendering pipeline, wherein the neural rendering pipeline is used to render static map scenes, and the traditional rendering pipeline is used to render map identifiers. In the neural rendering pipeline, the scene is first collected for the target scene to obtain the image and the corresponding camera pose information, and then the image is preprocessed to remove dynamic information in the scene, such as pedestrians, moving vehicles, etc. The processed information is sent to the NeRF (Neural Radiance Fields) network for learning. The input of the NeRF network is the 3D coordinate position (x, y, z) and direction (θ, ), and the output is the color c = (r, g, b) of the 3D point related to the view angle and the volume density σ of the position. Through the learning of the NeRF network, the three-dimensional model can be converted into the parameters and weight coefficients of the network to obtain the implicit expression of the three-dimensional model. When rendering a new view angle, only the pose of the new view angle needs to be input to obtain the corresponding image. The traditional graphics rendering pipeline can obtain a simple map identifier image after application layer processing, geometry processing and rasterization processing. Combining the neural rendering pipeline with the traditional graphics rendering pipeline can turn the static map scene into a dynamic editable scene, increasing the readability and usability of the target scene.

[0059] like Figure 6 As shown, it is an accelerated schematic diagram of a freely expandable stereoscopic digital sandbox light field rendering method of the present application, in which S61 is the viewer's viewing position, S62 is the left view imaging plane, S63 is the right view imaging plane, S64 is the equivalent imaging target, S65 is the lens array, S66 is the sub-image unit, S67 is the sub-image array, and S68 is the invalid light. Unlike the traditional parallax-type free-stereoscopic display technology, each small window on the imaging modulation plane of the light field display technology can provide the viewer with a viewpoint image of a part of the three-dimensional scene. As the number of small holes increases, the viewpoint resolution will also increase accordingly. When using hybrid rendering technology to realize light field content rendering based on human eye positioning, the light direction and starting point of each pixel in the sub-image are different. It is necessary to reverse the temporary starting point coordinates based on the pixel coordinates of the display screen to determine the starting point of the light to solve the depth inversion phenomenon, and then determine whether the starting point of the light is located in the small range viewpoint area where the human eye is located. For ease of understanding, we regard the lens array as an ideal array of pinholes. Starting from the LCD panel, we divide the LCD panel into several display areas according to the size of the pinholes. If the size of the pinholes is K×K and the resolution of the LCD panel is M×N, then the size of each sub-image unit S66 is M / K×N / K. When rendering the light field, we start from the first pixel of the first sub-image unit in the upper left corner of the LCD panel. Each pixel in the sub-image unit and the pinhole form a beam of light. According to the extension line of the connection between the meta-image and the pinhole, it passes through the model. The point above the model position can be used as the starting point for rendering the light, so as to solve the problem of depth inversion. It should be noted that the dimensions need to be unified during the rendering process, that is, the units between the LCD panel display coordinate system, the optical microlens array coordinate system, and the world coordinate system need to be unified. Assuming that the index of the pixel on the display panel is (k, m), its coordinate (X p , Y p , Z p ) is expressed as:

[0060] X p =k*pp-pp*W / 2 (1)

[0061] Yp =pp*H / 2-m*pp (2)

[0062] Z p =0 (3)

[0063] Where pp represents the pixel size, W and H represent the horizontal and vertical resolutions of the display, respectively. Assuming that the sub-image unit resolution is N×N, the lens center coordinates (X len , Y len , Z len )for:

[0064]

[0065]

[0066] Z len =g (6)

[0067] Where ceil represents the rounding function, and g represents the distance between the lens array and the real panel. In the case of human eye positioning, a single pixel actually corresponds to the light formed by the surrounding W / N×H / N lenses. This application defines this situation as pixel reuse. From the formula, it can be deduced that the light determined by the pixel (k, m) is:

[0068]

[0069] Among them, origin() represents the starting point of the light, direction() represents the direction of the light, i and j represent the row and column indexes in the W / N×H / N lens array, the lens directly above the pixel is numbered (0, 0), and the other lenses are recursively numbered in this way. In order to solve the depth flip problem, the farthest collision object needs to be obtained during ray tracing, and this can be achieved by moving the light forward in the direction to the human eye plane to obtain a new starting point. The principle is as follows Figure 6 After the light corresponding to each pixel is determined, the pixel can be partially rendered according to the human eye coordinates. For the pixels on the display, the starting point and direction of the light are calculated according to Formula 7, and then it is determined whether the light can reach the vicinity of the human eye. If it can, it is rendered using ray tracing technology. Otherwise, it means that this light cannot be observed, which is called invalid light S68 in this application, and directly enters the next pixel until the rendering is completed.

[0070] In summary, on the one hand, this application realizes seamless splicing of high-density display screens through collimated backlight control technology and optical microstructure arrays, so that the display size of the digital sandbox can be freely expanded. On the other hand, in conjunction with human eye tracking technology, on the basis of three-dimensional static scene rendering of neural radiation field, in conjunction with traditional graphics rendering pipeline technology, the fusion of static and dynamic scenes is further realized, so that the 3D scenes of implicit expression and explicit expression can be rendered and displayed in real time in a unified framework, and at the same time, the amount of image rendering data is effectively reduced by rendering data only at the viewing position of the human eye. Combining the above two technologies, observers can obtain a better naked-eye stereoscopic immersion at any viewing angle.

[0071] The above contents are further detailed descriptions of the present application in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the inventive concept of the present application.

Claims

1. A freely expandable three-dimensional digital sandbox system, characterized in that: The three-dimensional digital sandbox system includes: a data encoding module, a data decoding module, a human eye tracking module, a human-computer interaction module, a light field image rendering module, a driving module and a display module; The data encoding module of the stereo digital sand table system is connected to the data decoding module and the human-computer interaction module through the HDMI / DP interface, and is used to transmit the three-dimensional model data and the interaction instructions to the server; the light field image rendering module performs light field image rendering according to the received three-dimensional model data, the human-computer interaction instructions and the real-time viewing position coordinates of the human eye; the light field image rendering module and the display module are connected through the driving module, and the driving module is used to display the light field image in a three-dimensional form on the display module. The free expansion display screen is seamlessly spliced ​​by multiple sub-screens, so that the display size of the stereo digital sand table can be infinitely expanded; The user can interact and control the 3D digital sandbox system through the 2D display screen on the PC, and can also set the working mode of the 3D digital sandbox system to achieve mode switching between 2D and 3D display models; The data encoding module is used to encode the implicitly or explicitly expressed three-dimensional model and transmit it losslessly through standard HDMI / DP.

2. A freely expandable three-dimensional digital sandbox system as claimed in claim 1, characterized in that: The light field image rendering module adopts a modular light field image rendering hardware platform design, which is used for parallel rendering of light field images to improve the rendering speed.

3. A freely expandable three-dimensional digital sandbox system as claimed in claim 1, characterized in that: The driving module adopts an array design and is used to refresh and control the high-density liquid crystal display panel in different areas. The entire display screen can be driven by multiple sub-driving modules, and each sub-driving module can drive multiple sub-display screens to refresh.

4. A freely expandable three-dimensional digital sandbox system as claimed in claim 1, characterized in that: The display module is connected to the driving module and is used to map the display content to the liquid crystal panel and display stereoscopic information so that viewers can watch the stereoscopic scene with naked eyes. The display module is composed of multiple sub-screens, and uses collimated backlight technology and optical microstructure array to achieve seamless splicing of high-density display screens, so that the display size of the three-dimensional digital sandbox can be freely expanded.

Images