Digital sand table-oriented minimum system and end-to-end light field rendering method

Abstract

The invention discloses a digital sand table-oriented minimum system and an end-to-end light field rendering method, and compared with a traditional light field rendering method, the digital sand table-oriented minimum system and the end-to-end light field rendering method adopt a light-level end-to-end rendering method based on a neural network, so that viewpoint synthesis and pixel coding steps in a current light field rendering process are saved. According to the invention, only the light under the viewing angle is rendered based on the human eye position, so that the light field rendering process is simplified, and the light field rendering speed is remarkably improved. Besides, a gray NeRF model with small depth and width is adopted to render most of gray light under the viewing angle according to a proportion, and a color NeRF model with large depth and width is adopted to render a small part of color light under the viewing angle according to a proportion, so that compared with a method for rendering the light only by using the color NeRF model, the method has the advantages that the light rendering efficiency is greatly improved; according to the method, the rendering speed is increased, and the method is more in line with human visual watching characteristics.

Description

Technical Field

[0001] The present application relates to the field of computer image processing, and in particular to a minimum system and end-to-end light field rendering method for a digital sandbox. Background Art

[0002] Light field 3D display, an emerging 3D display technology, reconstructs information from each field of view of a 3D scene by simulating the luminous properties of 3D object surfaces. It not only faithfully reproduces the spatial characteristics of a 3D scene but also accurately depicts the inter-occlusion relationships between different objects, making it a 3D display technology that better aligns with the human eye's viewing characteristics.

[0003] Light field 3D display technology dates back to 1903, when Ives invented the binocular parallax display system, which utilized pinhole imaging technology. However, due to the limitations of computing power and technological advancements at the time, light field theory was difficult to effectively verify and implement. However, after Levoy and Gortler formalized the light field concept in 1996, the field of light field 3D display rapidly developed, yielding a series of classic theories, such as integrated imaging light field display, projection array light field display, and stacked light field display.

[0004] Existing light field 3D display technology has the following disadvantages:

[0005] 1. Due to the large amount of light field information and the complexity of the network structure, it is difficult to achieve end-to-end light field three-dimensional stereoscopic display under a specified human viewing angle.

[0006] 2. Affected by the viewpoint synthesis and pixel encoding operations in the light field rendering step, the rendering calculation amount increases significantly, reducing the rendering speed.

[0007] 3. Currently, most of the display screens used for 3D light field rendering are difficult to freely splice, which greatly reduces scalability and flexibility. Summary of the Invention

[0008] To solve the above problems, the present application provides a minimum system and end-to-end light field rendering method for a digital sandbox.

[0009] According to the first aspect of the present application, the present application provides a minimum system for a digital sandbox, comprising: an image acquisition module for acquiring original RGB data of a target object at different angles and orientations to form a sparse data set; a grayscale image training unit, connected to the image acquisition module signal, for training the acquired sparse data set in a preset grayscale network to obtain corresponding weight parameters, and obtaining a grayscale NeRF model; an RGB image training unit, connected to the image acquisition module signal, for training the acquired sparse data set in a preset RGB network to obtain corresponding weight parameters, and obtaining a color NeRF model; an eye tracking module, for acquiring the viewing angle information of the target user's eyes, and estimating the light emission position information of the target object. , obtaining light information under a specified viewing angle; a light field rendering module, signal-connected to the human eye tracking module, the grayscale image training unit, and the RGB image training unit, feeding the light information under a specified viewing angle into the grayscale NeRF model and the color NeRF model according to a preset ratio for non-uniform rendering, to obtain light rendering information; a driving module, signal-connected to the light field rendering module, for generating display driving information according to the light rendering information, wherein the display driving information can generate driving amounts for different display contents according to different light properties of the light rendering information; a display module, signal-connected to the driving module, for outputting the light field information of a specified viewing angle on a display screen based on the display driving information, thereby achieving the purpose of three-dimensional stereoscopic display.

[0010] According to the second aspect of the present application, the present application provides an end-to-end light field rendering method, including: a three-dimensional scene sparse acquisition step: including acquiring the original RGB data of the target object at different angles and orientations to form a sparse data set; a grayscale NeRF model training step: including acquiring the sparse data set, training the sparse data set in a preset grayscale network to obtain corresponding weight parameters, and obtaining a grayscale NeRF model; a color NeRF model training step: including acquiring the sparse data set, training the sparse data set in a preset RGB network to obtain corresponding weight parameters, and obtaining a color NeRF model; human eye viewing The perspective acquisition step includes acquiring the viewing perspective information of the target user's eyes and estimating the light emission position information of the target object to obtain the light information under the specified viewing perspective; the non-uniform light rendering step includes sending the light information under the specified viewing perspective into the grayscale NeRF model and the color NeRF model according to a preset ratio for non-uniform rendering to obtain light rendering information; the light field three-dimensional stereoscopic display step includes generating display driving information according to the light rendering information, and outputting the light field information of the specified viewing perspective on the display screen based on the display driving information, thereby achieving the purpose of three-dimensional stereoscopic display.

[0011] The beneficial results of this application are:

[0012] The minimal system and end-to-end light field rendering method for a digital sandbox described in the above embodiments utilizes a neural network-based, ray-level, end-to-end rendering method, compared to traditional light field rendering methods. This approach eliminates the viewpoint synthesis and pixel encoding steps in the current light field rendering process. Based on the eye's position, only rays from the desired viewing angle are rendered, simplifying the light field rendering process and significantly improving rendering speed.

[0013] In addition, the present invention uses a grayscale NeRF model with smaller depth and width to proportionally render most of the grayscale light under the desired viewing angle, and uses a color NeRF model with larger depth and width to proportionally render a small part of the colored light under the desired viewing angle. Compared with the method of rendering light only with the color NeRF model, the present invention not only speeds up the rendering speed but also is more in line with the visual viewing characteristics of the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram of the structure of the minimum system for the digital sandbox in one embodiment of the present application;

[0015] Figure 2 A comparative schematic diagram of a minimum system for a digital sandbox in an embodiment of the present application;

[0016] Figure 3 This is a schematic diagram of the principle of non-uniform light rendering based on the human eye's perspective in one embodiment of the present application;

[0017] Figure 4 This is a flow chart of an end-to-end light field rendering method in one embodiment of the present application;

[0018] Figure 5 This is a flowchart of grayscale and color NeRF model training in one embodiment of the present application;

[0019] Figure 6 A schematic diagram of the principle of high-dimensional mapping of multi-dimensional information in one embodiment of the present application;

[0020] Figure 7 This is a schematic diagram of the network architecture of a preset neural network for a color NeRF model in one embodiment of the present application;

[0021] Figure 8 This is a schematic diagram of the network architecture of the preset neural network of the grayscale NeRF model in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The present application will be further described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0023] In one embodiment of the present application, a minimum system for digital sandbox is provided. Figure 1 The present application discloses a minimum system for a digital sandbox, which includes an image acquisition module 11, a grayscale image training unit 12, an RGB image training unit 13, an image training module 14, an eye tracking module 15, a light field rendering module 16, a driving module 17, and a display module 18, which are described below.

[0024] Image acquisition module 11 is used to collect raw RGB data of the target object at different angles and orientations, forming a sparse dataset. In one embodiment, this process includes: first determining the number of detection cameras required to capture the target object or scene; then determining the position of each camera based on the number of detection cameras, ensuring a uniform distance and angle between cameras; then adjusting the exposure, focal length, and other parameters of each detection camera to ensure consistent capture; and finally, synchronizing the shutter speeds of all cameras to ensure simultaneous exposure during capture.

[0025] The image training module 14 includes two sub-units, namely a grayscale image training unit 12 and an RGB image training unit 13, which are described below respectively.

[0026] The grayscale image training unit 12 is connected to the image acquisition module 11 by signal, and is used to train the collected sparse data set in a preset grayscale network to obtain corresponding weight parameters and obtain a grayscale NeRF model. In a specific embodiment, it includes: calculating the light direction, color and optical center coordinates based on the data of the collected sparse data set and the internal and external parameters of the detection camera and forming them into light information; using the light direction and color corresponding to each imaging light point in the light information, calculating the coordinate values ​​x, y, z of each feature point of the target object or scene in three-dimensional space, and the angle values ​​θ, θ and θ of each feature point relative to the detection camera. And form multi-dimensional information; encode the spatial position information and detection direction information, and send them to an MLP network with smaller depth and width for training, that is, an MLP network with a depth of 6 and a width of 128, to output the grayscale value of each light; calculate the error between the predicted grayscale value and the true grayscale value, and adjust the network parameters according to the error information until the error information is less than the set error threshold and the training is stopped. At this time, the weight parameters of the grayscale network can be obtained.

[0027] The RGB image training unit 13 is connected to the image acquisition module 11 by signal, and is used to train the acquired sparse data set in a preset RGB network to obtain corresponding weight parameters and obtain a color NeRF model. In a specific embodiment, it includes: calculating the light direction, color and optical center coordinates based on the data of the acquired sparse data set and the internal and external parameters of the detection camera and forming them into light information; using the light direction and color corresponding to each imaging light point in the light information, calculating the coordinate values ​​x, y, z of each feature point of the target object or scene in three-dimensional space, and the angle values ​​θ, θ, and θ of each feature point relative to the detection camera. And form multi-dimensional information; encode the spatial position information and detection direction information, and send it to an MLP network with large depth and width for training, that is, an MLP network with a depth of 8 and a width of 256, to output the RGB value of each light; calculate the error between the predicted RGB value and the true RGB value, and adjust the network parameters according to the error information until the error information is less than the set error threshold and the training is stopped. At this time, the weight parameters of the RGB network can be obtained.

[0028] The eye tracking module 15 is configured to collect the target user's viewing angle information and estimate the light emission position information of the target object to obtain the light information at a specified viewing angle. In one embodiment, the module includes: collecting the desired viewing angle information, which can be expressed as a rotation angle relative to a reference coordinate system (such as a world coordinate system or a camera coordinate system); and estimating the light emission position information of the target object or scene to obtain the light information at the specified viewing angle.

[0029] The light field rendering module 16 is signal-connected to the eye tracking module 15 and the image training module 14, and sends the light information under the specified viewing angle to the grayscale NeRF model and the color NeRF model according to a preset ratio for non-uniform rendering to obtain light rendering information. In a specific embodiment, it includes: randomly sending the light information under the specified viewing angle to two NeRF models with different depths and widths for rendering according to a set ratio (such as grayscale value accounting for 60%), wherein a grayscale NeRF model with smaller depth and width is used to render most of the light, and a color NeRF model with larger depth and width is used to render a small part of the light. That is, a NeRF model with a depth of 6 and a width of 128 is used as a grayscale network; and a NeRF model with a depth of 8 and a width of 256 is used as an RGB network. Here, the depth refers to the number of layers of the multi-layer perceptron (MLP) in the NeRF model, and the width refers to the number of neurons in a certain layer of the multi-layer perceptron (MLP) in the NeRF model.

[0030] The driver module 17 is signal-connected to the light field rendering module 16 and is configured to generate display driver information based on the light rendering information. The display driver information is capable of generating different drive levels for displaying different content based on the different light properties of the light rendering information. In one specific embodiment, the display driver information is displayed using different driver units, namely, a grayscale driver unit and an RGB driver unit, depending on whether the rendered light information is grayscale or color. The display driver information is capable of generating different drive levels for displaying different content based on the different light properties of the light rendering information. The grayscale driver unit, which outputs grayscale light, changes the display by adjusting the brightness of the light source, and the RGB driver unit, which outputs RGB light, changes the display by controlling the brightness of the red, green, and blue lights, respectively.

[0031] Display module 18 is signal-connected to driver module 17 and configured to output light field information for a specified viewing angle on the display screen based on the display driver information, thereby achieving a three-dimensional stereoscopic display. In one embodiment, this includes outputting grayscale and RGB light information corresponding to the specified viewing angle on the display screen based on drive instructions from the grayscale and RGB driver units, thereby achieving end-to-end light field three-dimensional stereoscopic display.

[0032] Accordingly, this application also proposes an end-to-end light field rendering method, such as Figure 4 As shown, it includes the following steps.

[0033] S41, 3D scene sparse acquisition step: This step can be understood as acquiring the original RGB data of the target object at different angles and orientations to form a sparse data set.

[0034] In a specific embodiment, the 3D scene sparse acquisition step includes the following process:

[0035] (1) Determine the number, model, and layout of detection cameras based on the target object or scene. For example, depending on the characteristics of the target object or scene and the shooting requirements, dozens or even hundreds of high-pixel cameras may be needed to surround the target object or scene.

[0036] (2) Determine the position of each camera based on the number of detection cameras, and evenly distribute the cameras around the target object or scene to ensure that all angles of the target object can be fully covered.

[0037] (3) Adjust the exposure, focal length and other parameters of each camera to ensure consistent shooting effects; synchronize the shutter time of all cameras to ensure simultaneous exposure during shooting.

[0038] Steps S42-S43 are to obtain the sparse data set, train the sparse data set in networks of different depths and widths to obtain corresponding weight parameters, and obtain the corresponding NeRF model, which are explained below.

[0039] S42, grayscale NeRF model training step: This step can be understood as obtaining the sparse data set, training the sparse data set in a preset grayscale network to obtain corresponding weight parameters, and obtaining a grayscale NeRF model.

[0040] In a specific embodiment, the grayscale NeRF model training step includes the following process:

[0041] (1) Acquire detection data of the target object or scene from the detection camera, and determine the imaging light points corresponding to the detection data in the camera coordinate system of the detection camera. Calculate the light direction, color, and optical center coordinates corresponding to each imaging light point based on the imaging light points and the internal and external parameters of the detection camera to form light information.

[0042] (2) Using the light direction and color corresponding to each imaging light point in the light information, the coordinate values ​​x, y, and z of each feature point of the target object or scene in the three-dimensional space are calculated as the spatial position information of the target object or scene; the angle values ​​θ, y, and z of each feature point relative to the detection camera are calculated. As the detection direction information of the target object or scene, multi-dimensional information of the target object or scene is formed based on the spatial position information and the detection direction information.

[0043] (3) The spatial position information and detection direction information are encoded in a high-dimensional manner, and the encoded information is input into an MLP preset neural network training with a small depth and width, that is, an MLP network with a depth of 8 and a width of 256 is used for training, thereby outputting the predicted grayscale value of each light ray.

[0044] (4) Calculate the error between the predicted grayscale value and the true grayscale value of each light. The error information includes the PSMR value and the mean square error value. Adjust the network parameters according to the error information until the error information is less than the set error threshold and stop training. At this time, the weight parameters of the grayscale network can be obtained.

[0045] S43, color NeRF model training step: This step can be understood as obtaining the sparse data set, training the sparse data set in a preset RGB network to obtain corresponding weight parameters, and obtaining a color NeRF model.

[0046] In a specific embodiment, the color NeRF model training step includes the following process:

[0047] (1) Acquire detection data of the target object or scene from the detection camera, and determine the imaging light points corresponding to the detection data in the camera coordinate system of the detection camera. Calculate the light direction, color, and optical center coordinates corresponding to each imaging light point based on the imaging light points and the internal and external parameters of the detection camera to form light information.

[0048] (2) Using the light direction and color corresponding to each imaging light point in the light information, the coordinate values ​​x, y, and z of each feature point of the target object or scene in the three-dimensional space are calculated as the spatial position information of the target object or scene; the angle values ​​θ, y, and z of each feature point relative to the detection camera are calculated. As the detection direction information of the target object or scene, multi-dimensional information of the target object or scene is formed based on the spatial position information and the detection direction information.

[0049] (3) The spatial position information and detection direction information are encoded in a high-dimensional manner, and the encoded information is input into an MLP preset neural network with large depth and width for training, that is, an MLP network with a depth of 8 and a width of 256 is used for training, thereby outputting the predicted RGB value of each light ray.

[0050] (4) Calculate the error between the predicted RGB value and the true RGB value of each light. The error information includes the PSMR value and the mean square error value. Adjust the network parameters according to the error information until the error information is less than the set error threshold and stop training. At this time, the weight parameters of the RGB network can be obtained.

[0051] S44, step of collecting the viewing angle of the human eye: This step can be understood as collecting the viewing angle information of the target user's human eye, and estimating the light emission position information of the target object to obtain the light information under the specified viewing angle.

[0052] In a specific embodiment, the step of collecting the viewing angle of the human eye includes the following process:

[0053] (1) Collect the human eye viewing angle information to be queried, which can be expressed as a rotation angle relative to a reference coordinate system (such as the world coordinate system or the camera coordinate system);

[0054] (2) Estimate the light emission position information of the target object or scene to obtain the light information under the viewing angle of the human eye.

[0055] S45, non-uniform light rendering step: This step can be understood as sending the light information under a specified viewing angle into the grayscale NeRF model and the color NeRF model according to a preset ratio for non-uniform rendering to obtain light rendering information.

[0056] In one embodiment, the non-uniform light rendering step includes the following process:

[0057] (1) Using the light information under the viewing angle, the coordinate values ​​x, y, and z of the feature point corresponding to each light ray in the three-dimensional space are calculated as the spatial position information of the light ray; the angle values ​​θ, y, and z of each light ray relative to the detection camera are calculated. As the detection direction information of the light, multi-dimensional information of the light is formed based on the spatial position information and the detection direction information.

[0058] (2) The spatial position information and detection direction information are encoded in a high-dimensional manner, and the encoded information is randomly sent to two MLP preset neural networks with different depths and widths for rendering according to a set ratio (such as grayscale value accounting for 60%), thereby outputting the predicted RGB value or grayscale value of each light.

[0059] (3) Among them, a grayscale NeRF model with a small depth and width is used to render most of the light, and a color NeRF model with a large depth and width is used to render a small part of the light. That is, a NeRF model with a depth of 6 and a width of 128 is used as the grayscale network, while a NeRF model with a depth of 8 and a width of 256 is used as the RGB network. Here, the depth refers to the number of layers of the multilayer perceptron (MLP) in the NeRF model, and the width refers to the number of neurons in a layer of the multilayer perceptron (MLP) in the NeRF model.

[0060] S46, light field three-dimensional stereoscopic display step: This step can be understood as generating display driving information according to the light rendering information, and outputting light field information of a specified viewing angle on a display screen based on the display driving information, thereby achieving the purpose of three-dimensional stereoscopic display.

[0061] In a specific embodiment, the light field 3D stereoscopic display step includes the following process:

[0062] Based on the driving display information, the grayscale light information and RGB light information output by the grayscale NeRF model and the color NeRF model at a specified viewing angle are displayed on the display screen, thereby realizing end-to-end light field three-dimensional stereoscopic display.

[0063] The minimum system and end-to-end light field rendering method for digital sandbox proposed in this application, compared with the traditional light field rendering method, adopts a ray-level end-to-end rendering method based on a neural network, which saves the viewpoint synthesis and pixel encoding steps in the current light field rendering process. Based on the position of the human eye, only the light under the desired viewing angle is rendered, which not only simplifies the light field rendering process, but also significantly improves the light field rendering speed. In addition, the present invention uses a grayscale NeRF model with a smaller depth and width to proportionally render most of the grayscale light under the desired viewing angle, and uses a color NeRF model with a larger depth and width to proportionally render a small part of the colored light under the desired viewing angle. Compared with the method of rendering light using only a color NeRF model, the present invention not only speeds up the rendering speed but also is more in line with the visual viewing characteristics of the human eye. The following is a detailed analysis.

[0064] The sparse data set collected by the detection camera is sent to the preset grayscale network and RGB network for training to obtain the corresponding weight parameters, and the grayscale NeRF model and color NeRF model are obtained.

[0065] In both networks, the detection camera parameters of each image in the sparse dataset are first obtained through methods such as colmap, and the collected images and corresponding parameters are input into the NeRF model to calculate the optical center coordinates and light direction of the light corresponding to each feature point of the target object or scene in the actual physical space. The blocks are divided according to the color features of the image, and the importance is assigned based on whether the block contains the target object. The blocks with low importance generate fewer lights and compress the number of lights. A fixed number of lights are randomly extracted in each round of training, and a certain number of query positions are randomly sampled on the light according to the optical center coordinates and light direction. The coordinate values ​​x, y, z and the query direction θ, θ at each query position are calculated. In order to make the preset neural network extract information more effectively, the spatial position information and the detection direction information are encoded in a high-dimensional manner. That is, an L value is set for the spatial position information and the detection direction information respectively, and the encoding method shown in formula (1) is constructed to perform a high-dimensional mapping of the Fourier characteristics of the information, so as to obtain the high-dimensional encoding information corresponding to the spatial position and detection direction of each query point.

[0066] γ(p)=(sin(2 0 πp), cos(2 0 πp),…,sin(2 L-1 πp), cos(2 L-1 πp)) (1) The encoded data is sent to the MLP preset neural network to calculate the color feature c corresponding to each query point i and the transparency feature σ i The grayscale value or RGB value at the optical center is calculated using the color feature and transparency feature of each feature point on the same light. For each feature point i on any light, its opacity feature T is calculated using formula (2) i , and then use formula (3) to calculate the predicted grayscale value (or RGB value) at the optical center of the light.

[0067] T i =(1-σ i-1 )*T i-1 (2)

[0068]

[0069] The mean squared error (MSE) between the predicted grayscale values ​​(or RGB values) and the true grayscale values ​​(or RGB values) is calculated using formula (4). The calculated error is automatically back-propagated according to the computational graph to modify each weight of the MLP network, completing a training cycle. The training process is repeated until the error information is less than the set error threshold. At this point, the weight parameters of the grayscale NeRF model and the color NeRF model are obtained.

[0070]

[0071] During the rendering process, the NeRF model with weight parameters obtained through the above training process is used to non-uniformly render the light at a specified viewing angle to obtain rendering information, including the following steps:

[0072] The human viewing angle information to be queried is collected, which can be expressed as a rotation angle relative to a reference coordinate system (such as the world coordinate system or the camera coordinate system); the light emission position information of the target object or scene is estimated to obtain the light information at the specified viewing angle. According to a set ratio (such as 60%), the light at the specified viewing angle is sent to the grayscale NeRF model and the color NeRF model for rendering. In each model, the above training steps are repeated for the input light to obtain the rendering information of each light, that is, the corresponding grayscale value or RGB value.

[0073] For Figure 2 The traditional light field 3D display method shown in (a) performs a series of complex viewpoint synthesis and pixel encoding steps in the light field rendering process after the neural network training step, resulting in slow rendering speed and low rendering efficiency. Figure 2 (b) To simplify the light field rendering process and improve its efficiency, the end-to-end light field rendering process designed in this application estimates the light emission position information of the target object or scene based on the human eye's viewing angle information, and only renders the light under the specified viewing angle. This not only greatly simplifies the light field rendering process, but also significantly improves the light field rendering speed.

[0074] like Figure 3 As shown in the figure, for all the light rays under the specified viewing angle of the human eye, according to a certain ratio, such as 60% of the light rays are sent to the trained grayscale NeRF model for rendering, and the grayscale value corresponding to the light rays is obtained (such as Figure 3 The remaining 40% of the light is sent to the trained color NeRF model for rendering to obtain the RGB value corresponding to the light (as shown in gray). Figure 3As shown in the RGB mark in the figure). Since the human eye is relatively insensitive to colored light, but more sensitive to grayscale light, this application uses a grayscale NeRF model with smaller depth and width to render most of the grayscale light under the desired viewing angle, and uses a color NeRF model with larger depth and width to render a small part of the colored light under the desired viewing angle. That is, a NeRF model with a depth of 6 and a width of 128 is used as the grayscale network; and a NeRF model with a depth of 8 and a width of 256 is used as the RGB network. The depth here refers to the number of layers of the multi-layer perceptron (MLP) in the NeRF model, and the width refers to the number of neurons in a certain layer of the multi-layer perceptron (MLP) in the NeRF model. Not only does it greatly improve the rendering rate, it is also more in line with the visual viewing characteristics of the human eye.

[0075] Please refer to Figure 5 ,The training steps of the grayscale NeRF model and the color NeRF model have similar ,training processes, which are analyzed in detail below.

[0076] Step S51, inputting the collected sparse data set. In a specific embodiment, this step includes the following process: the collected original RGB data of the target object or scene at various angles and orientations are sent as sparse data sets to the grayscale NeRF model and the color NeRF model for training.

[0077] Step S52: Generate a light bundle and corresponding light information. In one embodiment, this step includes the following process:

[0078] (1) For a detection camera, each imaging light point is a reflection of a ray (radiation or reflection) from the target object or scene on the detection camera's imaging surface. The detection data in the detection camera is the imaging result of each corresponding imaging light point on the imaging surface. Therefore, the detection data of the target object or scene can be obtained from the detection camera, and each imaging light point corresponding to the detection data can be determined in the camera coordinate system of the detection camera.

[0079] (2) The light direction, color and optical center coordinates corresponding to each imaging light point are calculated based on the intrinsic parameter K and the external parameter c2w of the imaging light point and the detection camera, forming the light information.

[0080] Step S53, calculating multi-dimensional information of the target object or scene. In a specific embodiment, this step includes the following process:

[0081] (1) Using the light direction, optical center coordinates and color corresponding to each imaging light point in the light information, the coordinate values ​​x, y, z of each feature point of the target object or scene in three-dimensional space are obtained, and the angle values ​​θ,

[0082] (2) The coordinates x, y, and z are used as the spatial position information of the target object or scene, and the angle value θ is used as the As the detection direction information of the target object or scene, θ, The horizontal and vertical deflection angles of the target point relative to the detection point are used to generate multi-dimensional information about the target object or scene based on the spatial position information and the detection direction information.

[0083] Step S54, encoding the multi-dimensional information. In a specific embodiment, this step includes the following process:

[0084] (1) In order to enable the preset neural network to extract information more effectively, the spatial position information and the detection direction information are encoded in a high-dimensional manner to obtain encoding information corresponding to the spatial position and the detection direction respectively.

[0085] (2) For details, please refer to Figure 6 , set an L value for each feature point's spatial position information (marked 61) and detection direction information (marked 62) in the multi-dimensional information, construct a coding method as shown in formula (1) to perform a high-dimensional mapping of the Fourier characteristics of the information, and obtain high-dimensional coding information corresponding to each feature point's spatial position and detection direction.

[0086] γ(p)=(sin(2 0 πp), cos(2 0 πp),…,sin(2 L-1 πp), cos(2 L-1 πp)) (1)

[0087] According to the above encoding method, a high-dimensional mapping of the Fourier characteristics of the spatial position information and the detection direction information is achieved, where p represents the data to be processed.

[0088] Step S55, MLP network training, in a specific embodiment, this step includes the following process:

[0089] (1) The encoding information corresponding to the spatial position of any light ray is input into the front network layer of the preset MLP neural network. The output result of the front network layer is the transparency feature of each feature point of the light ray, that is, the transparency value σ.

[0090] (2) The transparency value σ of each feature point of the light and the coding information corresponding to the detection direction are input into the back network layer of the preset MLP neural network, and the color feature (i.e., grayscale value or RGB value) at each feature point of the light is output after fusion processing.

[0091] (3) Calculate the corresponding opacity value according to the transparency value at each feature point of the light, and use its transparency value, color feature and opacity value to calculate the grayscale information or color information at the optical center of the light.

[0092] (4) Calculate the error between the grayscale information of each light and the true grayscale value (or calculate the error between the color information of each light and the true RGB value). The error information includes the PSMR value and the mean square error value. Adjust the MLP network parameters according to the error information until the error information is less than the set error threshold and stop training. At this time, the weight parameters of the grayscale network or RGB network can be obtained.

[0093] Step S56, obtaining the network weight parameters of the NeRF model, which includes the following process:

[0094] The training results of the MLP network, ie, the weight parameters of the grayscale NeRF model or the color NeRF model, can be obtained through the training of steps S51 to S55.

[0095] During the training and rendering of the color NeRF model, a preset MLP neural network is used. Please refer to Figure 7 , detailed description is given below.

[0096] Figure 7 The Input (marked 71) in the figure is used to input the coded information corresponding to the spatial position of any light ray. The Add-Linear (marked 72) represents the previous network layer, and the Output (marked 73) represents the output of the previous network layer, which is the transparency value σ at each feature point of any light ray. The output transparency value at each feature point of the light ray and the coded information corresponding to the detection direction are input into the subsequent network layer of the preset neural network, and the RGB value of the light ray is obtained through fusion processing. Figure 7 The Direction (marked 74) in the figure represents the post-network layer, which fuses the transparency value at each feature point of any light and the encoding information corresponding to the detection direction. The Output (marked 75) represents the output result of the post-network layer, that is, the RGB value of any light.

[0097] In the training and rendering process of the grayscale NeRF model, another preset MLP neural network is used, please refer to Figure 8 , detailed description is given below.

[0098] Figure 8The Input (marked 81) in the figure is used to input the coded information corresponding to the spatial position of any light ray. The Add-Linear (marked 82) represents the previous network layer, and the Output (marked 83) represents the output of the previous network layer, which is the transparency value σ at each feature point of any light ray. The transparency value at each feature point and the coded information corresponding to the detection direction are input into the subsequent network layer of the preset neural network, and the grayscale value of the light ray is obtained through fusion processing. Figure 8 The Direction (marked 84) in the figure represents the post-network layer, which fuses the transparency value at each feature point of any light and the encoding information corresponding to the detection direction. The Output (marked 85) represents the output result of the post-network layer, that is, the grayscale value of any light.

[0099] In the preset MLP network used by the color NeRF model, the depth of the front network layer is 8 and the width is 256, and the scale of the network and the computational complexity are relatively large. In the preset MLP network used by the grayscale NeRF model, the depth of the front network layer is 6 and the width is 128, and the scale of the network and the computational complexity are greatly reduced. In a specific embodiment, for the light under the specified viewing angle of the human eye, 60% of the light is sent to the grayscale NeRF model for rendering, and the remaining 40% of the light is sent to the color NeRF model for rendering. Compared with the method of sending all light to the color NeRF model for rendering, the three-dimensional rendering method of the present application greatly improves the rendering speed and greatly reduces the amount of calculation. At the same time, because the human eye is relatively insensitive to colored light and more sensitive to grayscale light, the three-dimensional rendering method of the present application uses the grayscale NeRF model to render most of the light under the specified viewing angle of the human eye, and the proportion of grayscale light is higher, which is more in line with the visual viewing characteristics of the human eye.

[0100] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A minimum system for digital sandbox, characterized by: include: The image acquisition module is used to collect the original RGB data of the target object at different angles and orientations to form a sparse data set; A grayscale image training unit, signal-connected to the image acquisition module, is used to train the acquired sparse data set in a preset grayscale network to obtain corresponding weight parameters and obtain a grayscale NeRF model; An RGB image training unit, signal-connected to the image acquisition module, is used to train the acquired sparse data set in a preset RGB network to obtain corresponding weight parameters and obtain a color NeRF model; The eye tracking module is used to collect the viewing angle information of the target user's eyes and estimate the light emission position information of the target object to obtain the light information under the specified viewing angle; a light field rendering module, signal-connected to the eye tracking module, the grayscale image training module, and the RGB image training module, and feeding light information at a specified viewing angle into the grayscale NeRF model and the color NeRF model according to a preset ratio for non-uniform rendering to obtain light rendering information; a driving module, signal-connected to the light field rendering module, configured to generate display driving information according to the light rendering information, wherein the display driving information can generate driving amounts of different display contents according to different light attributes of the light rendering information; The display module is signal-connected to the driving module and is used to output light field information of a specified viewing angle on a display screen based on the display driving information, thereby achieving the purpose of three-dimensional stereoscopic display.

2. The minimum system for digital sandbox according to claim 1, characterized in that: The grayscale image training module uses a NeRF network with smaller depth and width as the grayscale network, that is, a NeRF network with a depth of 6 and a width of 128.

3. The minimum system for digital sandbox according to claim 1, characterized in that: The RGB image training module uses a NeRF network with a large depth and width as the RGB network, that is, a NeRF network with a depth of 8 and a width of 256.

4. The minimum system for digital sandbox according to claim 1, characterized in that: The light field rendering module renders most of the light as grayscale light and a small part of the light as colored light, which is used to adapt to the characteristics of the human eye that is sensitive to grayscale light but insensitive to colored light, so as to conform to the viewing characteristics of the human eye.

5. The minimum system for digital sandbox according to claim 1, characterized in that: The display module uses multiple LED dot matrix screens as display screens, and enhances the scalability and flexibility of light field three-dimensional display by freely splicing multiple LED dot matrix screens.

6. An end-to-end light field rendering method, characterized in that: include: The sparse acquisition step of the three-dimensional scene includes acquiring the original RGB data of the target object at different angles and orientations to form a sparse data set; A grayscale NeRF model training step includes obtaining the sparse data set, training the sparse data set in a preset grayscale network to obtain corresponding weight parameters, and obtaining a grayscale NeRF model; The color NeRF model training step includes obtaining the sparse data set, training the sparse data set in a preset RGB network to obtain corresponding weight parameters, and obtaining a color NeRF model; The step of collecting the viewing angle of the human eye includes collecting the viewing angle information of the target user and estimating the light emission position information of the target object to obtain the light information under the specified viewing angle; The non-uniform light rendering step includes inputting the light information under a specified viewing angle into the grayscale NeRF model and the color NeRF model according to a preset ratio for non-uniform rendering to obtain light rendering information; The light field three-dimensional stereoscopic display step includes generating display driving information according to the light rendering information, and outputting light field information of a specified viewing angle on a display screen based on the display driving information, thereby achieving the purpose of three-dimensional stereoscopic display.

7. The end-to-end light field rendering method according to claim 6, wherein: In the grayscale NeRF model training step, a NeRF network with smaller depth and width is used as the grayscale network and the sparse data set is trained, that is, a NeRF network training data set with a depth of 6 and a width of 128 is used.

8. The end-to-end light field rendering method according to claim 6, wherein: In the color NeRF model training step, a NeRF network with larger depth and width is used as the RGB network and the sparse data set is trained, that is, a NeRF network training data set with a depth of 8 and a width of 256 is used.

9. The end-to-end light field rendering method according to claim 6, wherein: In the non-uniform light rendering step, grayscale light accounts for the majority of all light, and colored light accounts for a smaller portion of all light, thereby conforming to the characteristics of the human eye that is sensitive to grayscale light but insensitive to colored light, and more in line with the viewing characteristics of the human eye.

10. The end-to-end light field rendering method according to claim 6, wherein: In the light field three-dimensional stereoscopic display step, the display screen is configured as a plurality of LED dot matrix screens, and the scalability and flexibility of the light field three-dimensional stereoscopic display are enhanced by freely splicing the plurality of LED dot matrix screens.

Images