Digital sand table system adopting image rendering and light field image rendering method

Abstract

The invention discloses a digital sand table system adopting image rendering and a light field image rendering method. According to the system, a neural radiation field is used for reconstructing a three-dimensional scene model, light field rendering is carried out on scene images, the orientation of a viewer is tracked and positioned through human eyes, directional light rendering is carried out, and compared with image rendering at all angles, the processing efficiency can be obviously improved. Moreover, the system reproduces the rendered directional rendering image through a light field display technology, and the stereoscopic image can change along with the viewer, thereby enhancing the interaction effect. Besides, the system can change the content of the stereoscopic image by changing the image of the three-dimensional scene, provides the operation of interactive processing, and has the application advantages that the content of the stereoscopic image can be changed at will and the interactive effect is good compared with a traditional sand table.

Description

Technical Field

[0001] The present application relates to the field of computer vision and optical imaging, and in particular to a digital sandbox system using image rendering and a light field image rendering method. Background Art

[0002] The existing sand table system is mainly implemented by a physical sand table model made of sand or 3D printing. However, this type of sand table system has defects such as fixed content and inability to update data in real time. For example, in the field of national defense, it will lead to a decrease in the timeliness of battlefield information data and cannot meet the needs of real-time changes in the ever-changing battlefield environment.

[0003] There are several mainstream architectures for the current three-dimensional image stereo display technology: one is to use optical instruments such as projectors as media for three-dimensional reconstruction; the other is to use wearable interactive devices (such as VR glasses) based on virtual reality technology for visual simulation. However, at the hardware level, the existence of projectors and wearable interactive devices makes it impossible for digital sand tables to integrate a more complete and flexible intelligent system. They are also deeply dependent on physical objects and cannot fully meet the needs of electronic sand table systems in specific fields. Among the currently popular three-dimensional sand tables, most of them use stereo projection and combine physical models to jointly realize the three-dimensional sand table model device.

[0004] In general, the existing technology has the following shortcomings: first, the technology of combining virtual and real is based on the physical model, which makes it difficult to change the sand table model once the structure is determined; second, the existing sand table has poor interactivity and cannot be changed in real time according to user needs; third, the stereoscopic projection and volume display sand tables occupy a large area and are not easy to disassemble, transport and move; fourth, the grating stereoscopic display can only be viewed from a fixed direction and is prone to visual fatigue, which does not meet the actual use conditions of the sand table. Summary of the invention

[0005] The technical problem solved by this application is that the existing digital sand table display is affected by the fixed model, and has problems such as poor interactivity and difficulty in changing the display scene. To solve the above problems, this application provides a digital sand table system using image rendering and a light field image rendering method.

[0006] According to the first aspect of the present application, the present application provides a digital sandbox system using image rendering, including: an image acquisition unit, used to perform image acquisition of a three-dimensional scene, obtain a scene image and transmit it to a host computer; an image rendering unit, running on the host computer, used to obtain the scene image from the image acquisition unit, and use the scene image to reconstruct and render the three-dimensional scene to obtain a light field model; a data transmission unit, running on the host computer, used to convert the light field model into a neural network framework type, obtain a conversion model and deploy it to a lower computer; a directional rendering unit, running on the lower computer, used to track and obtain the orientation information of the viewer, and use the orientation information to perform directional light rendering on the conversion model to obtain a directional rendering image; an image display unit, communicatively connected to the lower computer, used to perform three-dimensional stereoscopic projection of the directional rendering image to form a digital sandbox, and the digital sandbox includes a stereoscopic image of the three-dimensional scene facing the viewer.

[0007] Furthermore, the image rendering unit includes a model training module and a light field image generation module; the model training module is used to reconstruct the three-dimensional scene using the neural radiation field to obtain a scene model; the light field image generation module is used to perform multi-perspective light field rendering on the scene image using the scene model, generate a light field image of the three-dimensional scene and construct a corresponding light field model.

[0008] Furthermore, the data transmission unit includes a neural network framework transfer module and a distributed Ethernet port module; the neural network framework transfer module is used to convert the light field model from a first neural network framework type to a second neural network framework type to obtain a corresponding conversion model; the light field model has a first neural network framework type adapted to the upper computer, and the conversion model has a second neural network architecture type adapted to the lower computer; the distributed Ethernet port module is used to deploy the conversion model to the lower computer.

[0009] Furthermore, the directional rendering unit includes an eye tracking module, a light field image rendering module and an HDMI transmission module; the eye tracking module is used to determine the position and viewing angle of the viewer through eye tracking to form the orientation information; the light field image rendering module receives the conversion model, and is used to use the orientation information as an input parameter to perform directional light rendering on the conversion model to obtain a directional rendering image of the three-dimensional scene; the HDMI transmission module is used to transmit the directional rendering image to the image display unit via the HDMI protocol.

[0010] Furthermore, the image display unit includes a collimated backlight module, a screen display module and a compound lens array module; the collimated backlight module is used to provide collimated backlight for the screen display module; the screen display module is used to receive the directional rendering image transmitted by the HDMI module, and display the directional rendering image on the screen; the compound lens array module is used to focus the light of each pixel in the picture of the screen display module, and the three-dimensional projection generated by the projection forms the digital sandbox.

[0011] Furthermore, the collimated backlight module has a light extinction structure, and when the light is projected onto the screen display module, the stray light is eliminated by the light extinction structure, which can reduce the crosstalk of the picture display and improve the contrast and clarity of the picture.

[0012] Furthermore, the image acquisition unit is used to obtain a video of the three-dimensional scene, extract frame images from the video of the three-dimensional scene using a video frame extraction technique, and obtain images of the three-dimensional scene from multiple perspectives as the scene images.

[0013] According to a second aspect of the present application, an image rendering method is provided, comprising: acquiring an image of a three-dimensional scene to obtain a scene image; using the scene image to reconstruct and render a light field of the three-dimensional scene to obtain a light field model; converting the light field model into a neural network framework type to obtain a conversion model, which can be deployed to a lower computer; tracking and obtaining the position information of the viewer, using the position information to perform directional light rendering on the conversion model to obtain a directional rendering image, which is used for three-dimensional projection.

[0014] Furthermore, the using of the scene image to perform reconstruction and light field rendering of the three-dimensional scene to obtain a light field model includes: using a neural radiation field to reconstruct the three-dimensional scene to obtain a scene model; the light field image generation module is used to perform multi-perspective light field rendering on the scene image using the scene model, generate a light field image of the three-dimensional scene and construct a corresponding light field model; wherein, when performing multi-perspective light field rendering, the scene model can learn the emissivity of the three-dimensional scene and the propagation direction of light through the scene image.

[0015] Furthermore, the tracking obtains the position information of the viewer, and uses the position information to perform directionally ray rendering on the conversion model to obtain a directionally rendered image, including: determining the position and viewing angle of the viewer through human eye tracking to form the position information; using the position information as an input parameter, performing directionally ray rendering on the conversion model to obtain a directionally rendered image of the three-dimensional scene; wherein, when performing directionally ray rendering on the conversion model, the input parameters of the conversion model also include focal length and resolution.

[0016] The beneficial effects of this application are:

[0017] According to the digital sandbox system and image rendering method using image rendering in the above-mentioned embodiment, the system uses the neural radiation field to reconstruct the three-dimensional scene model and performs light field rendering on the scene image. The viewer's position is tracked by the human eye and directional light rendering is performed. Compared with image rendering at all angles, the processing efficiency will be significantly improved; the image acquisition unit, image rendering unit and data transmission unit in the system rely on the host computer to run, while the directional rendering unit and image display unit rely on the lower computer to run, which can realize the separate processing of establishing the light field model and generating the directional rendering image, thereby greatly saving the time of the two links and improving the data processing efficiency; the rendered directional rendering image is reproduced through the light field display technology, and the stereoscopic image can change with the viewer, enhancing the interactive effect. The digital sandbox system can change the content of the stereoscopic image by changing the image of the three-dimensional scene, and provides interactive processing operations. Compared with the traditional sandbox, it has the application advantages of arbitrary change of stereoscopic image content and good interactive effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a structural schematic diagram of a digital sandbox system using image rendering in an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of the principle of an image rendering process in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the principle of making an example model in one embodiment of the present application;

[0021] Figure 4 A schematic diagram of the principle of generating a light field image in one embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of the principle of a training model in one embodiment of the present application;

[0023] Figure 6 This is a structural schematic diagram of an image display module in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a flow chart of an image rendering method in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of image rendering and display control in one embodiment of the present application. DETAILED DESCRIPTION

[0026] In the description of the present invention, it should be understood that the described embodiments are only part of the embodiments of the present invention, not all the embodiments.

[0027] The present application is further described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0028] The present application discloses a digital sandbox system using image rendering, the structure of which is as follows: Figure 1 As shown, the digital sandbox system mainly includes an image acquisition unit 11, an image rendering unit 120, a data transmission unit 130, a directional rendering unit 140 and an image display unit 150. They are described below respectively.

[0029] The image acquisition unit 11 may be one or more cameras surrounding a three-dimensional scene, and is used to acquire images of the three-dimensional scene, obtain scene images, and transmit them to a host computer. The three-dimensional scene here refers to a space with three-dimensional objects, and the corresponding scene image refers to an image containing three-dimensional objects; the host computer here may be a data processing device such as a computer.

[0030] The image rendering unit 120 runs on the host computer and is used to obtain the scene image from the image acquisition unit 11 and use the scene image to reconstruct and render the three-dimensional scene to obtain a light field model.

[0031] The data transmission unit 130 runs on the host computer and is used to convert the light field model generated by the image rendering unit 120 into a neural network framework type, obtain the conversion model and deploy it to the slave computer. The slave computer here can be a data processing component such as a microprocessor (MCU), and can communicate data with the host computer. The early non-real-time data processing process can be performed on the host computer, and the later real-time data processing process can be performed on the slave computer. The two are reasonably matched to improve data processing efficiency.

[0032] The directional rendering unit 140 runs on the lower computer and is used to track and obtain the position information of the viewer, and use the position information to perform directional light rendering on the conversion model to obtain a directional rendering image.

[0033] The image display unit 150 is communicatively connected to the lower computer, and is used to perform three-dimensional projection of the directional rendering image to form a digital sand table, where the digital sand table includes a stereoscopic image of the three-dimensional scene facing the viewer.

[0034] It should be noted that the digital sand table is a display method that uses digital technology to combine sound, light, electricity, images, animations, etc. with physical models, or directly uses a display screen to display information in a lifelike and varied visual effect. It can provide viewers with three-dimensional images, aiming to provide viewers with an intuitive and vivid information display experience.

[0035] In a specific embodiment, the image acquisition unit 11 is used to obtain a video of a three-dimensional scene, and extracts frame images from the video of the three-dimensional scene using a video frame extraction technology, so as to obtain images of the three-dimensional scene from multiple perspectives and use them as scene images. For example, the image acquisition unit 11 can acquire multi-angle 2D images of a three-dimensional object, shoot a three-dimensional object from different angles through a camera, and transmit the images to the image rendering unit 120. Of course, in actual operation, in order to facilitate the sparse perspective acquisition of the three-dimensional scene, the video frame extraction technology is preferably used to acquire images of the three-dimensional scene, and the acquired coordinate angle information and image information are transmitted to the image rendering unit 120. It can be understood that the image data containing the three-dimensional scene information is acquired from different perspectives, and these images can be real scene images captured by a camera or other device, or synthetic images generated by computer graphics technology.

[0036] In a specific embodiment, the image rendering unit 120 includes a model training module 12 and a light field image generation module 13. The model training module 12 is used to reconstruct the three-dimensional scene using the neural radiation field to obtain a scene model; the light field image generation module 13 is used to perform multi-perspective light field rendering on the scene image using the scene model, generate a light field image of the three-dimensional scene and construct a corresponding light field model. It can be understood that the image rendering unit 120 mainly uses the neural radiation algorithm to train the three-dimensional model, and renders and generates a light field model corresponding to the multi-angle image according to the training content.

[0037] It can be understood that the model training module 12 uses the training set for deep learning, combines NeRF with the memory neural radiation field, makes the model generation efficiency faster, and improves the efficiency of the algorithm; the light field image generation module 13 uses the neural network trained by the model training module 12, and transmits the collected scene image to the neural network for generation, so that multi-perspective square rendering can be performed, thereby generating a light field image of the three-dimensional scene corresponding to the multi-perspective. In other words, the light field image generation module 13 uses the multi-angle 2D image (i.e., scene image) transmitted by the image acquisition unit 11 to perform a light field model reconstruction operation on the three-dimensional object to generate a model of the scene. The computing framework under this module is Pytroch, that is, the light field model has a first neural network framework type that adapts to the host computer.

[0038] It should be noted that Neural Radiance Field (NeRF) is a neural network that can reconstruct complex 3D scenes from partial 2D image sets. Various simulations, games, media, and Internet of Things (IoT) applications require 3D images to make digital interactions more realistic and accurate. NeRF learns the scene geometry, objects, and angles of a specific scene, and then renders realistic 3D views from new perspectives, automatically generating synthetic data to fill in the gaps.

[0039] In a specific embodiment, the data transmission unit 130 includes a neural network framework transfer module 14 and a distributed Ethernet network port module 15. The neural network framework transfer module 14 is used to convert the light field model from a first neural network framework type to a second neural network framework type to obtain a corresponding conversion model. The light field model has a first neural network framework type (such as a Pytroch framework) adapted to the upper computer, and the conversion model has a second neural network architecture type (such as a mobile terminal framework) adapted to the lower computer. The distributed Ethernet network port module 15 is used to deploy the conversion model to the lower computer.

[0040] It should be noted that the neural network framework transfer module 14 converts the training framework Pytorch into a mobile terminal framework through the ONNX structure, realizing cross-platform deployment and reasoning of the model while retaining its accuracy and performance. The distributed Ethernet network port module 15 uses network transmission technology to transmit the converted model to the lower computer for image rendering of the lower computer.

[0041] It should be noted that the data transmission unit 130 can transfer the light field model of the Pytorch framework in the light field image generation module 13 to the personal mobile device, and transmit the directionally rendered light field image to the display screen through the personal mobile device to achieve a stereoscopic image facing the viewer. Among them, the neural network framework transfer module 14 mainly converts the square model under the Pytorch framework in the light field image generation module 13 into other platform frameworks, and the distributed Ethernet port module 15 mainly connects the neural network framework transfer module 14 with the image display unit 140, and transmits the received light field model to the directional rendering unit 140.

[0042] In a specific embodiment, the directional rendering unit 140 includes a human eye tracking module 17 , a light field image rendering module 16 and an HDMI transmission module 18 .

[0043] The human eye tracking module 17 may be a camera or a video camera, which can capture and identify images of human eyes, and is used to determine the position and viewing angle of the viewer through human eye tracking to form position information. It can be understood that the human eye tracking module 17 is used to locate the position and viewing angle of the viewer, and input the viewer's position information (position and viewing angle) to the light field image rendering module 16. The human eye tracking module 17 can track the viewer's position in real time and transmit information, so that the stereoscopic effect can still be refreshed in real time when the viewer moves.

[0044] The light field image rendering module 16 runs on the lower computer and can receive the conversion model from the upper computer where the data transmission unit 130 is located, and is used to use the orientation information formed by the human eye tracking module 17 as an input parameter to perform directional light rendering on the conversion model to obtain a directional rendering image of the three-dimensional scene. It can be understood that the light field image rendering module 16 determines the viewing angle (orientation) of the viewer through the human eye tracking module 17, performs directional light rendering of the image according to the viewing angle of the viewer, and performs directional rendering through the received conversion model.

[0045] The HDMI transmission module 18 runs on the host computer and is used to transmit the directional rendering image to the image display unit 150 using the HDMI protocol. It can be understood that the HDMI transmission module 18 connects the light field image rendering module 16 and the image display unit 150 through the HDMI interface, and transmits the directional rendering image (which can be understood as the light field image to be displayed) generated by the light field image rendering module 16 to the image display unit 150 for stereoscopic display.

[0046] In a specific embodiment, the image display unit 150 includes a collimating backlight module 110 , a screen display module 19 and a compound lens array module 111 .

[0047] The collimated backlight module 110 is a light source, and is used to provide collimated backlight for the screen display module 19 .

[0048] The screen display module 19 may be a liquid crystal display screen, which is used to receive the directional rendering image sent by the HDMI transmission module and display the directional rendering image; the displayed image is illuminated by a collimated backlight.

[0049] The compound lens array module 111 is used to focus the light of each pixel in the picture of the screen display module, and the three-dimensional projection generated by the projection forms a digital sand table.

[0050] It can be understood that the image display unit 150 converts the directional rendering image sent by the HDMI transmission module into a three-dimensional stereo image through the basic principle of light field display, wherein the collimated backlight module 110 can provide the screen display module 19 with lighting of set direction and intensity, and the composite lens array module 111 can perform stereoscopic display of the light field, thereby presenting a stereoscopic image.

[0051] It can be understood that the light field image rendering module 16 uses the lower computer to perform directional rendering of the image with the help of the light field model after the frame type is converted, and the human eye tracking module 17 determines the position of the viewer through human eye tracking, so that the light field image rendering module 16 performs directional rendering, saving rendering time and improving efficiency. The light field image after directional rendering is transmitted to the screen display module 19 through the HDMI transmission module 18. The screen display module 19 can be composed of a borderless 2K / 4K screen. The borderless screen is easy to splice and combine, and it is convenient to build the minimum system into the sandbox structure we need. The collimated backlight module 110 uses a material that can absorb light to make an extinction structure. The extinction structure absorbs scattered light so that most of the light can be propagated within a certain range. The screen display module 19 receives the directional rendering image generated by the light field image rendering module 16 through the HDMI transmission module 18, and displays the directional rendering image on the screen. The composite lens array module 111 is connected to the screen display module 19, wherein the composite lens array module 111 projects the displayed image, so that the viewer can see different images at different positions, and the stereoscopic projection effect is achieved through the movement of the viewer.

[0052] In a specific embodiment, the collimated backlight module 110 has a matte structure, and when the light is projected onto the screen display module 19, the stray light is eliminated by the matte structure, which can reduce the crosstalk of the picture display and improve the contrast and clarity of the picture. For example, the collimated backlight module 110 projects the light onto the screen display module 19 through the designed matte structure to illuminate the screen image, and the matte structure can be a terraced structure to absorb part of the reflected light, reduce the interference of ambient light, and thus improve the visual effect.

[0053] In addition, the screen display module 19 can be a borderless 2K / 4K display screen, which displays the directional rendering image transmitted by the HDMI transmission module 18. Multiple display screens can be combined to achieve the splicing of the minimum system. The composite lens array module 111 uses a composite lens array to guide light through the light field display principle, thereby presenting the image displayed by the screen display module 19 in three dimensions to obtain a three-dimensional image 112.

[0054] It should be noted that the digital sandbox system displays the processed image in three dimensions through a composite lens array, and can refresh the displayed content in real time. The system performs neural radiation field rendering on the multi-view images of the pre-input model or the real physical model to obtain the meta-image and perform sub-pixel position mapping to improve the macro-pixel resolution of the light field image. The system uses a collimated light source as the backlight to eliminate the gaps between the spliced ​​screens. At the same time, it cooperates with the eye tracking technology to locate the viewer's perspective and viewing position for directional rendering to reduce rendering time. Because the digital sandbox system adopts a unit / modular design, it is convenient to expand the minimum system infinitely. The image generation method uses a neural radiation algorithm to generate light field images through end-to-end convolutional neural network training.

[0055] In one embodiment, Figure 2 A schematic diagram of the principle of the image rendering process is given, and details are given in each processing link from step 21 to step 26.

[0056] In step 21, the image acquisition unit 11 acquires data of the three-dimensional scene, that is, acquires image data containing scene information from different perspectives, that is, obtains scene images. These images can be frame-sampling processing of real scene videos captured by cameras or other devices, or they can be synthetic images generated by computer graphics technology.

[0057] In step 22, the model training module 12 in the image rendering unit 120 performs neural network training on the NeRF model (neural radiation field), uses the training set to train the NeRF neural network, and uses the light field image generation module 13 in the image rendering unit 120 to learn the radiance and propagation direction of light in the scene image, obtain a multi-perspective light field image and construct a light field model.

[0058] In step 23, the neural network framework conversion module 14 in the data transmission unit 130 first executes the conversion of the Pytorch framework to ONNX, and realizes seamless migration of models between different frameworks by converting the Pytorch framework to the ONNX format.

[0059] In step 24, the neural network framework conversion module 14 in the data transmission unit 130 executes ONNX to mobile terminal framework to realize cross-platform deployment and reasoning of the model. It can be understood that ONNX is an open deep learning model exchange format, which can convert and migrate models between different deep learning frameworks, thereby realizing cross-platform deployment.

[0060] In step 25, the distributed Ethernet port module 14 in the data transmission unit 130 deploys the conversion model to the lower computer, and the light field image rendering module 16 in the directional rendering unit 140 uses the trained NeRF model, sets the corresponding focal length and resolution, inputs the viewer's position and direction (i.e., the human eye tracking module 17 generates the orientation information) for reordering, and outputs the corresponding directional rendering image through the neural network.

[0061] Step 26, the collimated backlight module 110 in the image display unit 150 generates a collimated backlight, the screen display module 19 receives the directional rendering image generated by the light field image rendering module 16 through the HDMI transmission module 18, and displays the directional rendering image on the screen. The composite lens array module 111 projects the displayed image, so that the viewer can see different images at different positions, and a stereoscopic projection effect is achieved through the movement of the viewer, thereby generating a digital sand table.

[0062] In order to avoid the influence of device processing errors, redundant light and other factors when the camera collects multi-angle 2D images of objects, a virtual camera can be used for rendering in a computer simulation environment. This method can quickly adjust the experimental parameters. The software collects virtual 3D scenes or 3D reconstructions of real scenes, establishes a camera acquisition array, and transmits image information and camera coordinate information to the visual drive module.

[0063] In one embodiment, the specific operation is as follows Figure 3 As shown, the principle schematic diagram of making an example model is disclosed, making a simple 3D model, adjusting virtual camera parameters, adjusting the position and posture of the virtual camera, adjusting the size and orientation of the 3D model, and taking multi-angle photos. This process can be performed with the help of an image acquisition unit. The following are explained separately.

[0064] Step 31 : the image acquisition unit 11 establishes a 3D model (ie, an instance model corresponding to a three-dimensional scene) according to user requirements.

[0065] In step 32, the image acquisition unit 11 adjusts the virtual camera parameters, specifically, the virtual camera resolution needs to be set, the camera type needs to be set to perspective, the camera focal length, the lens unit, and the horizontal width of the camera sensor element need to be adjusted, etc.

[0066] Step 33: the image acquisition unit 11 adjusts the position of the virtual camera and sets virtual cameras at different positions.

[0067] Step 34: the image acquisition unit 11 adjusts the virtual camera posture and sets the virtual camera at different angles.

[0068] Step 35 : the image acquisition unit 11 adjusts the size and position of the 3D model.

[0069] Step 36: the image acquisition unit 11 adjusts the viewing angle of the 3D model.

[0070] In step 37 , the image acquisition unit 11 uses a virtual camera to capture multi-angle images of the three-dimensional model to form a scene image.

[0071] This application uses the NeRF algorithm. The basic principle is as follows Figure 4 The above can also be regarded as a schematic diagram of the principle of generating a light field image, which is divided into a data reading step 41, a 5D function expression step 42, a deep fully connected neural network optimization step 43, a volume rendering step 44, an image reordering step 45 and an output step 46. They are described below respectively.

[0072] Step 41, the image rendering unit 120 traverses the scene through camera rays to generate a set of 3D sampling points, and inputs these points and their corresponding 2D viewing angles as inputs into the neural network.

[0073] In step 42, the image rendering unit 120 represents a static scene as a continuous 5D function that outputs the radiance in each direction (θ, φ) emitted at each point (x, y, z) in space, and the density of each point, which is expressed as differential opacity and controls the amount of radiation accumulated when light passes through (x, y, z).

[0074] In step 43, the neural network does not have convolutional layers (commonly referred to as multi-layer perceptrons or MLPs), so the image rendering unit 120 needs to represent the function by regressing from a single 5D coordinate (x, y, z, θ, φ) to a single volume density and view-dependent RGB color, thereby achieving deep fully connected neural network optimization.

[0075] The image rendering unit 120 accumulates these colors and densities into a 2D image using classical volume rendering techniques, step 44. This model can be optimized using gradient descent to achieve this goal by minimizing the error between each viewed image and the corresponding rendered view from the radiation field representation.

[0076] In step 45 , the image rendering unit 120 reorders the collected 2D images using the predicted model to form a required light field image and construct a corresponding light field model.

[0077] In one embodiment, Figure 5 The principle diagram of the training model is disclosed, which can be divided into a data input step 51, a NeRF creation step 52, a rendering process step 54, and a loss calculation step 55. Each step is described below.

[0078] Step 51, the image rendering unit 120 reads all transmitted images and camera positions into the imgs and pos arrays of the algorithm to implement data input.

[0079] Step 52 is divided into step 56 of adjusting the focus, etc., step 57 of converting pixels into rays, and step 58 of obtaining position codes. The purpose of step 56 of adjusting the focus, etc. is to determine the far and near position of the image, and input the distance of the image from the center of the camera. The purpose of step 57 of converting pixels into rays is to input the width, height, focal length and external parameters (generally a 3*4 matrix, used from the camera to the world coordinates) of a certain image, and output the rays where all pixels of the image are located. The purpose of step 58 of obtaining position codes is to obtain the 5D information of the rays.

[0080] In step 53, the image rendering unit 120 performs deep neural learning training on the NeRF model through the training set, so that the NeRF algorithm generation efficiency is faster and the accuracy is higher.

[0081] Step 54 is divided into a coordinate conversion step 59 and a rendering step 510. The coordinate conversion step 59 is used to convert the image coordinates of the image into real world coordinates, and transmit the information to the rendering step 510. The rendering step 510 can convert pixels into points on the rays generated in the ray step 57 for sampling, obtain their real coordinates, and render through the coordinates to convert the 5D information into real model information.

[0082] In step 55 , the image rendering unit 120 calculates the difference between the converted model and the original model, that is, calculates the loss for training.

[0083] In one embodiment, the image display unit 160 needs to collimate the backlight module 110, see Figure 6 The collimated backlight design scheme will be described in detail below.

[0084] The light source L1 is used to provide light for the collimating backlight module. The extinction structure L2 uses a triangular extinction module to eliminate surface reflection. The display screen L3 is used to display a directional rendering image. The imaging plane L4 can form an image on this plane. The composite lens array L5 is used to display a three-dimensional effect.

[0085] In one embodiment, the present application discloses an image rendering method, such as Figure 7 As shown. Using the technology of image rendering and algorithm transplantation on the development side, neural radiation field rendering is used for image generation. The following are the respective descriptions.

[0086] Step 610: collect an image of a three-dimensional scene to obtain a scene image.

[0087] Step 620: Reconstruct the 3D scene and render the light field using the scene image to obtain a light field model.

[0088] Step 630, convert the light field model into a neural network framework type to obtain a conversion model, which can be deployed to a lower computer.

[0089] Step 640 , tracking and obtaining the position information of the viewer, using the position information to perform directional light rendering on the conversion model to obtain a directional rendering image, and the directional rendering image is used for three-dimensional projection.

[0090] In a specific embodiment, in the above step 610, image data containing scene information may be collected from different perspectives. These scene images may be real scene images captured by a camera or other device, or may be synthetic images generated by computer graphics technology.

[0091] In a specific embodiment, in the above step 620, the three-dimensional scene is reconstructed and light field rendered using the scene image to obtain a light field model, including:

[0092] (1) Use the neural radiation field to reconstruct the three-dimensional scene and obtain a scene model.

[0093] (2) Using the scene model to perform multi-perspective light field rendering on the scene image, generate a light field image of the three-dimensional scene and construct a corresponding light field model; wherein, when performing multi-perspective light field rendering, the scene model can learn the emissivity and light propagation direction of the three-dimensional scene through the scene image.

[0094] It can be understood that in step 620, the collected data can be used to train the NeRF neural network, which will learn the radiance and propagation direction of light in the scene.

[0095] In a specific embodiment, in the above step 630, the light field model is converted into a neural network framework type to obtain a conversion model, including:

[0096] (1) Obtaining a first neural network framework type based on the light field model, where the first neural network framework type is configured as a Pytorch framework and is used to enable the light field model to run on a host computer.

[0097] (2) Converting the light field model from the first neural network framework type to the second neural network framework type to obtain a corresponding conversion model, wherein the second neural network framework type is configured as an ONNX framework and is used to enable the conversion model to run on a lower computer.

[0098] It can be understood that in step 630, it can be regarded as a Pytorch framework to ONNX step and an ONNX to mobile terminal framework step to achieve cross-platform deployment and reasoning of the model. ONNX is an open deep learning model exchange format that can perform model conversion and migration between different deep learning frameworks. By converting the PyTorch model to the ONNX format, the model can be seamlessly migrated between different frameworks, thereby achieving cross-platform deployment.

[0099] In a specific embodiment, in the above step 640, tracking and obtaining the position information of the viewer, and using the position information to perform directional light rendering on the conversion model to obtain a directional rendering image include:

[0100] (1) The viewer’s position and viewing angle are determined through eye tracking to form orientation information.

[0101] (2) Using the orientation information as an input parameter, the conversion model is subjected to directional light rendering to obtain a directional rendering image of the three-dimensional scene; wherein, when the conversion model is subjected to directional light rendering, the input parameters of the conversion model also include focal length and resolution.

[0102] It can be understood that in step 640, a trained NeRF model can be used to input the position and direction of the viewer, and the neural network will output a corresponding directional rendering image.

[0103] In one embodiment, the digital sand table system and the image generation method disclosed above are combined to control the digital sand table system to display, such as Figure 7 As shown, the control process will be specifically described below. In actual use, the specific operations are as follows.

[0104] Step 71, the digital sandbox system starts running.

[0105] Step 72, constructing a three-dimensional model to be displayed in stereoscopic form.

[0106] Step 73, collect multi-angle images of the three-dimensional model to generate a scene image.

[0107] Step 74, training the NeRF model so that it can render light field images with better quality.

[0108] Step 75: Use the NeRF model to perform light field rendering on the scene image.

[0109] Step 76, obtaining a light field image and constructing a corresponding light field model according to the rendering result.

[0110] Step 77, convert the light field model and deploy it to the lower computer through data transmission.

[0111] Step 78, determining the viewer's position information by eye tracking.

[0112] Step 79, on the lower computer, with the help of the conversion model, input the orientation information, perform directional rendering according to the viewer's orientation, and obtain a directional rendering image.

[0113] Step 710: Display the directional rendering image on the display screen, and display the image as a three-dimensional image through a composite lens array, thereby forming a digital sand table.

[0114] It should be noted that the digital sand table system is based on cutting-edge technologies such as neural radiation field and light field stereoscopic display, focusing on the shortcomings of current digital sand table technology, which is equivalent to providing a feasible minimum system of intelligent digital sand table, which can promote the progress of third-generation display technology. Moreover, the digital sand table system can dynamically interact and refresh, has a good three-dimensional immersive feeling, and supports multiple users to use and communicate at the same time. For example, commanders can use mouse, keyboard, gestures, baton or other input devices to deduce battlefield environment and situation information in real time.

[0115] The above contents are further detailed descriptions of the present application in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the inventive concept of the present application.

Claims

1. A digital sandbox system using image rendering, characterized in that: include: An image acquisition unit is used to acquire images of three-dimensional scenes, obtain scene images and transmit them to a host computer; An image rendering unit, running on the host computer, is used to obtain the scene image from the image acquisition unit, and to reconstruct and render the three-dimensional scene using the scene image to obtain a light field model; A data transmission unit, running on the host computer, for converting the light field model into a neural network framework type, obtaining a conversion model and deploying it to the slave computer; A directional rendering unit, running on the lower computer, is used to track and obtain the position information of the viewer, and use the position information to perform directional light rendering on the conversion model to obtain a directional rendering image; The image display unit is communicatively connected to the lower computer and is used for performing three-dimensional projection on the directional rendering image to form a digital sand table, wherein the digital sand table includes a stereoscopic image of the three-dimensional scene facing the viewer.

2. The digital sandbox system according to claim 1, characterized in that: The image rendering unit includes a model training module and a light field image generation module; The model training module is used to reconstruct the three-dimensional scene using the neural radiation field to obtain a scene model; The light field image generation module is used to perform multi-perspective light field rendering on the scene image using the scene model, generate a light field image of the three-dimensional scene and construct a corresponding light field model.

3. The digital sandbox system as claimed in claim 2, characterized in that: The data transmission unit includes a neural network framework transfer module and a distributed Ethernet network port module; The neural network framework transfer module is used to convert the light field model from the first neural network framework type to the second neural network framework type to obtain a corresponding conversion model; the light field model has a first neural network framework type adapted to the upper computer, and the conversion model has a second neural network framework type adapted to the lower computer; The distributed Ethernet network port module is used to deploy the conversion model to the lower computer.

4. The digital sandbox system according to claim 1, characterized in that: The directional rendering unit includes a human eye tracking module, a light field image rendering module and an HDMI transmission module; The human eye tracking module is used to determine the position and viewing angle of the viewer through human eye tracking to form the orientation information; The light field image rendering module receives the conversion model and is used to perform directional light rendering on the conversion model using the orientation information as an input parameter to obtain a directional rendering image of the three-dimensional scene; The HDMI transmission module is used to transmit the directional rendering image to the image display unit using the HDMI protocol.

5. The digital sandbox system as claimed in claim 4, characterized in that: The image display unit includes a collimating backlight module, a screen display module and a compound lens array module; The collimated backlight module is used to provide collimated backlight for the screen display module; The screen display module is used to receive the directional rendering image transmitted by the HDMI module, and display the directional rendering image; The compound lens array module is used to focus the light of each pixel in the picture of the screen display module, and the three-dimensional projection generated by the projection forms the digital sand table.

6. The digital sandbox system as claimed in claim 5, characterized in that: The collimated backlight module has a light extinction structure, and when the light is projected onto the screen display module, the stray light is eliminated by the light extinction structure, thereby reducing the crosstalk of the picture display and improving the contrast and clarity of the picture.

7. The digital sandbox system according to claim 1, characterized in that: The image acquisition unit is used to obtain the video of the three-dimensional scene, extract frame images from the video of the three-dimensional scene using video frame extraction technology, and obtain images of the three-dimensional scene from multiple viewing angles as the scene images.

8. A light field image rendering method, characterized in that: include: Collecting an image of a three-dimensional scene to obtain a scene image; Reconstructing and rendering the three-dimensional scene using the scene image to obtain a light field model; Convert the light field model into a neural network framework type to obtain a conversion model, and the conversion model can be deployed to a lower computer; The position information of the viewer is tracked and obtained, and the conversion model is directional-ray rendered using the position information to obtain a directional-rendered image, which is used for three-dimensional projection.

9. The light field image rendering method according to claim 8, characterized in that: The reconstructing and light field rendering the three-dimensional scene using the scene image to obtain a light field model includes: Reconstructing the three-dimensional scene using the neural radiation field to obtain a scene model; Performing multi-perspective light field rendering on the scene image using the scene model to generate a light field image of the three-dimensional scene and construct a corresponding light field model; Wherein, when performing multi-view light field rendering, the scene model can learn the radiance of the three-dimensional scene and the propagation direction of light through the scene image.

10. The light field image rendering method according to claim 8, characterized in that: The tracking obtains the position information of the viewer, and uses the position information to perform directional light rendering on the conversion model to obtain a directional rendering image, including: Determining the position and viewing angle of the viewer by eye tracking to form the orientation information; Using the orientation information as an input parameter, performing directional light rendering on the conversion model to obtain a directional rendering image of the three-dimensional scene; Wherein, when directional light rendering is performed on the conversion model, the input parameters of the conversion model also include focal length and resolution.

Images