Method and device for generating three-dimensional light field content of recurrent nerve radiation field

Abstract

The embodiment of the invention relates to the technical field of three-dimensional light field content generation and three-dimensional light field imaging, and provides a method and a device for generating three-dimensional light field content of a recurrent nerve radiation field, multi-view images of a scene are obtained through a plurality of paths of RGB cameras, camera calibration is carried out, and a point density nerve radiation field is defined by internal parameters of the recurrent nerve radiation field. In the training stage, the parameters of the recurrent nerve radiation field are converged by minimizing a preset error function. In the rendering stage, virtual camera array parameters are set based on the three-dimensional light field display device, and rendering light needed for synthesizing images is emitted. A position coding module is used for processing coordinates of an initial sampling point on light, resampling is carried out, the position and direction vectors of the resampling point are input into a recurrent nerve radiation field through a corresponding coding module, colors of sub-pixels in a composite image are calculated, and finally a three-dimensional light field is generated. Therefore, the content generation quality of the three-dimensional light field can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional light field content generation, and in particular to a method and device for generating three-dimensional light field content of a circulatory neural radiation field. Background Art

[0002] In recent years, the application of computer vision in the field of 3D light field display has received widespread attention. Multi-viewpoint generation based on neural networks has gradually become a new trend in 3D light field content generation and is widely used in education, entertainment, office, medical and other fields. With the rapid development of neural network technology, these methods enable researchers to effectively extract features from sparse images and then perform 3D reconstruction and multi-viewpoint generation.

[0003] Learning-based methods learn the geometric distribution and color information of the scene by training a neural network model. However, these methods still face some challenges. In particular, it is often difficult to strike a good balance between model training time, rendering speed, and generation quality. In recent years, neural radiance fields have shown strong capabilities in multi-view generation and have become a hot topic of research. However, neural radiance fields also have some significant disadvantages. For example, the training time is usually long, especially in high-resolution and complex scenes, the training process may take hours or even days. In addition, neural radiance field models are usually deep and have a large number of parameters, which also leads to long training time. In application scenarios where the target three-dimensional light field needs to be generated quickly, it is difficult to meet user needs. For large-scale scenes, the video memory consumption of neural radiance fields is also high, which limits its application in resource-constrained environments. Summary of the invention

[0004] The present invention provides a method and device for generating three-dimensional light field content using a recurrent neural radiation field, which are used to solve the defects of the prior art three-dimensional light field content generation method based on learning, such as long training time, high video memory consumption, poor quality of generated viewpoints, and difficulty in meeting user needs, and realize the rapid convergence of the neural network in the three-dimensional light field content generation method and the high-quality generation of three-dimensional light field content.

[0005] The present invention provides a method for generating three-dimensional light field content of a recurrent neural radiation field, comprising: Using a multi-channel RGB camera to collect original images of the target scene from multiple perspectives, perform camera calibration, and complete the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function; Based on the light field coding method, a rendering camera is set according to preset parameters of the three-dimensional light field display device, and each sub-pixel emits a rendering ray; Collecting a plurality of initial sampling points on the rendering light, and obtaining a first position code corresponding to each initial sampling point through a position coding module; Inputting first position codes corresponding to the plurality of initial sampling points into a point density neural radiation field inside the circulatory neural radiation field, generating point density through the point density neural radiation field, thereby generating a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the circulatory neural radiation field; Resampling is performed according to the first cumulative transparency distribution data, and a second position code and a direction code corresponding to each resampling point obtained by the resampling are obtained; Calculate the color of each resampling point based on the second position encoding and direction encoding input recurrent neural radiation field corresponding to each resampling point; Calculate the sub-pixel color corresponding to the rendering light based on the color accumulation of each resampling point; Based on the light field encoding method, the sub-pixel color calculated corresponding to each rendering light is placed in the corresponding sub-pixel coordinates in the synthetic image, so as to calculate the synthetic image, realize the generation of three-dimensional light field content, and generate the target three-dimensional light field through the three-dimensional light field display device.

[0006] In a possible implementation, the method further includes: During the training process of the recurrent neural radiation field, each calibrated multi-channel RGB camera emits light for initial sampling point and resampling point calculation, and each pixel in the original image corresponds to a light ray emitted by the multi-channel RGB camera, and the point coordinates, pixel color and cumulative transparency distribution corresponding to each light ray are used to calculate the minimum error function.

[0007] In a possible implementation, the method further includes: During the training process of the recurrent neural radiation field, the predicted pixel color of each ray and the corresponding original image pixel color are used to optimize the recurrent neural radiation field with the goal of minimizing a preset color error function; Based on the second position-encoded and direction-encoded input recurrent neural radiation fields corresponding to each resampling point, the color and point density of each resampling point are calculated to generate a second cumulative transparency distribution; Using the initial sampling point coordinates and the first cumulative transparency distribution, and the resampling point coordinates and the second cumulative transparency distribution on each ray, the recurrent neural radiation field is optimized and trained with the goal of minimizing a preset resampling consistency error function.

[0008] In a possible implementation, the method further includes: In the rendering process of the synthetic image, based on the light field coding method, the correspondence between each sub-pixel and each rendering camera in the synthetic image is determined according to the preset three-dimensional light field display device parameters, and then the corresponding virtual rendering camera emits rendering light to calculate the corresponding sub-pixel color.

[0009] In a possible implementation, the method further includes: Mapping the multiple initial sampling points to a high-dimensional space based on a position encoding module as first position encodings of the multiple initial sampling points; The first position code is input into the point density neural radiation field, and the point density of the multiple initial sampling points is calculated to generate a first cumulative transparency distribution of the initial sampling points on a rendering ray.

[0010] In a possible implementation, the method further includes: Mapping the coordinates of the multiple resampling points into a high-dimensional space based on a position encoding module as second position encodings of the multiple resampling points; Mapping the direction vectors corresponding to the multiple resampling points to a high-dimensional space based on a direction encoding module as direction encodings of the multiple resampling points; The second position code and the direction code are feature fused, and the feature fusion result is input into the recurrent neural radiation field to calculate the color of each resampling point.

[0011] In a possible implementation, the method further includes: Based on the light field encoding method and the parameters of the preset three-dimensional light field display device, setting the parameters of the rendering camera array; Based on the rendering camera emitting rendering light, the sub-pixel color corresponding to the rendering light is calculated by the point density neural network and the recurrent neural network; Placing the sub-pixel colors into corresponding sub-pixel coordinates in the composite image based on the light field encoding method to achieve three-dimensional light field content generation; The three-dimensional light field content is input into a three-dimensional light field display device to generate a target three-dimensional light field.

[0012] The present invention also provides a device for generating three-dimensional light field content of a cyclic neural radiation field, comprising the following modules: The acquisition module is used to use multiple RGB cameras to acquire original images of the target scene from multiple perspectives and perform camera calibration; A training module, used to complete the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function; A setting module, used to set a rendering camera according to preset parameters of a three-dimensional light field display device based on a light field encoding method, so that each sub-pixel emits a rendering ray; A sampling module, used for collecting a plurality of initial sampling points on the rendering light, and obtaining a first position code corresponding to each initial sampling point through a position coding module; a calculation module, configured to input the first position codes corresponding to the plurality of initial sampling points into a point density neural radiation field inside the circulatory neural radiation field, generate point density through the point density neural radiation field, and thus generate a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the circulatory neural radiation field; The sampling module is further used to resample according to the first cumulative transparency distribution data, and obtain a second position code and a direction code corresponding to each resampling point obtained by the resampling; The calculation module is further used to calculate the color of each resampling point based on the second position code and direction code input recurrent neural radiation field corresponding to each resampling point; and calculate the sub-pixel color corresponding to the rendering light based on the color accumulation of each resampling point; A generation module is used to place the sub-pixel color calculated corresponding to each rendering light into the corresponding sub-pixel coordinates in the synthetic image based on the light field encoding method, thereby calculating the synthetic image, realizing three-dimensional light field content generation, and generating a target three-dimensional light field through a three-dimensional light field display device.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating three-dimensional light field content of a circulatory neural radiation field as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating three-dimensional light field content of a circulatory neural radiation field as described in any one of the above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for generating three-dimensional light field content of a recurrent neural radiation field as described in any one of the above.

[0016] The present invention provides a method and device for generating three-dimensional light field content of a recurrent neural radiation field. The method and device use a multi-channel RGB camera to collect original images of a target scene under multiple viewing angles, perform camera calibration, and complete the training of the recurrent neural radiation field by minimizing a preset error function. The preset error function includes a color error function and a resampling consistency error function. Based on a light field coding method, a rendering camera is set according to preset parameters of a three-dimensional light field display device, and each sub-pixel emits a rendering light. Multiple initial sampling points are collected on the rendering light, and a first position code corresponding to each initial sampling point is obtained through a position coding module. The first position codes corresponding to the multiple initial sampling points are input into a point density neural radiation field inside the recurrent neural radiation field, and the point density neural radiation field is used to generate the first position code corresponding to each initial sampling point. A radiation field generates a point density, thereby generating a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of a circulatory neural radiation field; resampling is performed according to the first cumulative transparency distribution data, and a second position code and a direction code corresponding to each resampling point obtained by the resampling are obtained; the second position code and the direction code corresponding to each resampling point are input into the circulatory neural radiation field to calculate the color of each resampling point; the sub-pixel color corresponding to the rendering light is calculated based on the color accumulation of each resampling point; the sub-pixel color corresponding to each rendering light is placed in the corresponding sub-pixel coordinates in a composite image based on the light field encoding method, thereby calculating the composite image, realizing three-dimensional light field content generation, and generating a target three-dimensional light field through a three-dimensional light field display device. Compared with the existing learning-based 3D light field content generation methods, it is often difficult to achieve a good balance between model training time and generation quality. The neural radiation field has the defects of long training time, high video memory consumption, poor generated viewpoint quality, and difficulty in meeting user needs. This method, by combining multi-view images, point density neural radiation fields, light sampling and resampling, neural network training, and 3D light field rendering technologies, achieves rapid convergence of neural networks and high-quality generation of 3D light field content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow charts of the method for generating three-dimensional light field content of the recurrent neural radiation field provided by the present invention.

[0019] Figure 2 This is the second flow chart of the method for generating three-dimensional light field content of the recurrent neural radiation field provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the data processing flow of the circulatory neural radiation field provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the device for generating three-dimensional light field content of a circulatory neural radiation field provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0025] Figure 1 is one of the flow charts of the method for generating three-dimensional light field content of a recurrent neural radiation field provided by the present invention, such as Figure 1 As shown, the method specifically includes: S11. Use a multi-channel RGB camera to collect original images of the target scene from multiple perspectives, perform camera calibration, and complete the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function.

[0026] In the scenario where the embodiment of the present invention is used to generate three-dimensional light field content, a recurrent neural radiation field technology is proposed based on the neural radiance field (NeRF), and the concept of point density neural radiation field is introduced. By combining multi-view images, point density neural radiation field, light sampling and resampling, neural network training, and three-dimensional light field rendering technologies, efficient and high-quality three-dimensional light field content generation for three-dimensional scenes is achieved.

[0027] Specifically, a multi-channel RGB camera is used to collect the original images of the target scene from multiple perspectives. After camera calibration, distortion correction and color correction, the target three-dimensional scene image to be reconstructed by the algorithm is obtained, and the three-dimensional world coordinate system is defined by the multi-channel RGB camera calibration results. The camera calibration is used to obtain accurate camera intrinsic and extrinsic parameters. After the camera calibration process, the multi-view images are subjected to distortion correction and color correction to ensure that the obtained images have good performance in terms of geometric structure and color consistency. The calibrated camera parameters and the corrected original images provide reliable basic data for subsequent three-dimensional reconstruction and multi-viewpoint generation. The camera calibration method can select the Zhang Zhengyou calibration method, or the open source calibration software Colmap. The color correction process of the multi-view image can collect and color calibrate the 24-color standard color plate in the target scene.

[0028] Furthermore, the recurrent neural radiance field is trained based on the calibrated camera parameters and the corrected multi-view images. Each calibrated multi-channel RGB camera emits light for initial sampling point and resampling point calculation. Each pixel in the original image corresponds to a ray emitted by the multi-channel RGB camera, and the point coordinates, pixel color and cumulative transparency distribution corresponding to each ray are used to minimize the calculation of the preset error function.

[0029] The preset error functions include a color error function and a resampling consistency error function. During the training process of the recurrent neural radiation field, the predicted pixel color of each ray and the corresponding original image pixel color are used to optimize the training of the recurrent neural radiation field with the goal of minimizing the preset color error function. Based on the second position encoding and direction encoding corresponding to each resampling point, the recurrent neural radiation field is input, and the color and point density of each resampling point are calculated to generate a second cumulative transparency distribution. Using the initial sampling point coordinates and the first cumulative transparency distribution on each ray, and the resampling point coordinates and the second cumulative transparency distribution, the recurrent neural radiation field is optimized and trained with the goal of minimizing the preset resampling consistency error function.

[0030] Specifically, the recurrent neural radiation field is trained by minimizing the error function. The error function includes a color error function and a resampling consistency error function. The color error function can use methods such as the 1-norm, 2-norm, or structural similarity function to evaluate the difference between the predicted color and the true color. The resampling consistency error function is used to measure the consistency between the initial sampling point and the resampling point in position and point density. The cross entropy error function and other methods can be used to weakly constrain the point density weighted position similarity.

[0031] S12. Based on the light field coding method, a rendering camera is set according to preset parameters of the three-dimensional light field display device, and each sub-pixel emits a rendering light.

[0032] After completing the recurrent neural network training, in the rendering process of the synthetic image, based on the light field coding method, the correspondence between each sub-pixel in the synthetic image and each rendering camera is determined according to the preset three-dimensional light field display device parameters, and then the corresponding virtual rendering camera emits rendering light to calculate the corresponding sub-pixel color.

[0033] S13, collecting a plurality of initial sampling points on the rendering light, and obtaining a first position code corresponding to each initial sampling point through a position coding module.

[0034] A plurality of initial sampling points are collected on the rendering light. The collection method may perform rough sampling according to the position of the target object in the three-dimensional space to obtain the plurality of initial sampling points.

[0035] Furthermore, the first position code corresponding to each initial sampling point is obtained through the position coding module. The coordinates of the sampling points on each rendering ray during the sampling process are processed by a preset position coding module. This module maps the position of each sampling point to a high-dimensional feature, so that the network can more effectively utilize the position distribution of the sampling points and improve the performance of the recurrent neural network. ‌High-dimensional features‌ refer to mapping the original data to a new feature space in the feature extraction process, and the dimension of the feature is higher than the dimension of the original data.

[0036] The position encoding module can use methods such as sine and cosine encoding or hash encoding with different frequencies to perform position encoding on the coordinates of the initial sampling points obtained by the initial sampling.

[0037] S14. Input the first position codes corresponding to the multiple initial sampling points into the point density neural radiation field inside the circulatory neural radiation field, generate point density through the point density neural radiation field, and thus generate a first cumulative transparency distribution.

[0038] The recurrent neural radiation field internally includes a point density neural radiation field defined by some of its parameters, including: in the recurrent neural radiation field, the position encoding of the input is calculated as a neural network of point density (point density neural radiation field), and then the point density is used to calculate and generate a cumulative transparency distribution according to a mathematical definition.

[0039] The first position code and the second position code are calculated according to the above process to obtain the first cumulative transparency distribution and the second cumulative transparency distribution.

[0040] S15. Resampling is performed according to the first cumulative transparency distribution data, and a second position code and a direction code corresponding to each resampling point obtained by the resampling are obtained.

[0041] The resampling process is performed based on the first cumulative transparency distribution of the initial sampling point to ensure that the resampling point can more accurately reflect the geometric features contained in each ray. The initial sampling point can be resampled according to the Cumulative Distribution Function (CDF) method.

[0042] The position encoding module can use methods such as sine and cosine encoding or hash encoding of different frequencies to perform position encoding on each resampling point obtained by resampling, and obtain a second position code corresponding to each resampling point. The direction encoding module can use methods such as sine and cosine encoding of the direction vector, spherical harmonic function encoding, etc. to perform direction encoding on the direction vector corresponding to the resampling point, and obtain a direction code corresponding to each resampling point.

[0043] S16. Calculate the color of each resampling point based on the second position encoding and direction encoding input recurrent neural radiation field corresponding to each resampling point.

[0044] By defining a lightweight neural network model and introducing cyclic sampling into the learning process of the target pixel, the spatial distribution and color characteristics of the target scene can be captured more quickly. The advantage of this method is that the cyclic sampling process can better learn the geometric structure of the target scene, and the lightweight model can reduce the training time of the network model and generate high-quality light field content of the target scene.

[0045] Specifically, the second position code and direction code are feature fused through the recurrent neural radiation field, and the feature fusion result is input into the recurrent neural radiation field to calculate the color of each resampled point. In this process, the collaborative work of multiple encoding modules ensures that the position and direction coding information of each resampled point can be smoothly transmitted to the recurrent neural radiation field network model, so as to calculate the corresponding color and point density.

[0046] S17, calculating the sub-pixel color corresponding to the rendering light based on the color accumulation of each resampling point.

[0047] In the recurrent neural radiance field, the color of the resampled point on each ray is accumulated according to the transparency of the resampled point to calculate the sub-pixel color corresponding to this ray in the composite image.

[0048] S18. Based on the light field encoding method, the sub-pixel color calculated corresponding to each rendering light is placed into the corresponding sub-pixel coordinates in the synthetic image, thereby calculating the synthetic image, realizing three-dimensional light field content generation, and generating a target three-dimensional light field through a three-dimensional light field display device.

[0049] In the rendering stage, the parameters of the rendering camera array are first set based on the light field encoding method and the parameters of the preset three-dimensional light field display device. After determining the virtual camera information corresponding to all sub-pixels in the synthetic image, the rendering camera emits rendering light, and the sub-pixel color corresponding to the rendering light is calculated through the point density neural network and the recurrent neural network. Finally, based on the light field encoding method, the sub-pixel color is placed in the corresponding sub-pixel coordinates in the synthetic image to achieve three-dimensional light field content generation, ensuring that the final generated three-dimensional light field content has high-quality visual effects.

[0050] The generated three-dimensional light field content is input into a three-dimensional light field display device to generate a three-dimensional light field.

[0051] The target three-dimensional light field content presented by the three-dimensional light field display device using the three-dimensional light field content information includes the depth and multi-viewing information of the scene, and can provide users with a more continuous and realistic three-dimensional visual experience.

[0052] The present invention provides a method for generating three-dimensional light field content of a recurrent neural radiation field. The method uses a multi-channel RGB camera to collect original images of a target scene under multiple viewing angles, performs camera calibration, and completes the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function. Based on a light field coding method, a rendering camera is set according to preset parameters of a three-dimensional light field display device, and each sub-pixel emits a rendering light ray. Multiple initial sampling points are collected on the rendering light ray, and a first position code corresponding to each initial sampling point is obtained through a position coding module. The first position codes corresponding to the multiple initial sampling points are input into a point density neural radiation field inside the recurrent neural radiation field, and the point density neural radiation The invention discloses a method for generating a point density of a neural radiation field, thereby generating a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the circulatory neural radiation field; resampling is performed according to the first cumulative transparency distribution data, and a second position code and a direction code corresponding to each resampling point obtained by resampling are obtained; the circulatory neural radiation field is input based on the second position code and the direction code corresponding to each resampling point, and the color of each resampling point is calculated; the sub-pixel color corresponding to the rendering light is calculated based on the color accumulation of each resampling point; the sub-pixel color corresponding to each rendering light is placed in the corresponding sub-pixel coordinates in the synthetic image based on the light field encoding method, thereby calculating the synthetic image, realizing the generation of three-dimensional light field content, and generating the target three-dimensional light field through the three-dimensional light field display device. Compared with the existing learning-based 3D light field content generation methods, it is often difficult to achieve a good balance between model training time and generation quality. The neural radiation field has the defects of long training time, high video memory consumption, poor generated viewpoint quality, and difficulty in meeting user needs. This method, by combining multi-view images, cyclic neural radiation fields, light sampling and resampling, neural network training, and 3D light field rendering technologies, achieves rapid convergence of neural networks and high-quality generation of 3D light field content.

[0053] Figure 2 This is the second flow chart of the method for generating three-dimensional light field content of the recurrent neural radiation field provided by the present invention, such as Figure 2 As shown, the method specifically includes: S21. Use multiple RGB cameras to collect original images of the target scene from multiple perspectives and perform camera calibration.

[0054] In the scene where the embodiment of the present invention is used to generate three-dimensional light field content, a recurrent neural radiance field technology is proposed based on the neural radiance field (NeRF). Figure 3As shown in the figure, the concept of point density neural radiation field is introduced. By combining multi-view images, point density neural radiation field, light sampling and resampling, neural network training, and three-dimensional light field rendering technologies, efficient and high-quality three-dimensional light field content generation for three-dimensional scenes is achieved.

[0055] Specifically, a multi-channel RGB camera is used to collect the original images of the target scene from multiple perspectives. After camera calibration, distortion correction and color correction, the target three-dimensional scene image to be reconstructed by the algorithm is obtained, and the three-dimensional world coordinate system is defined by the multi-channel RGB camera calibration results. The camera calibration is used to obtain accurate camera intrinsic and extrinsic parameters. After the camera calibration process, the multi-view images are subjected to distortion correction and color correction to ensure that the obtained images have good performance in terms of geometric structure and color consistency. The calibrated camera parameters and the corrected original images provide reliable basic data for subsequent three-dimensional reconstruction and multi-viewpoint generation. The camera calibration method can select the Zhang Zhengyou calibration method, or the open source calibration software Colmap. The color correction process of the multi-view image can collect and color calibrate the 24-color standard color plate in the target scene.

[0056] S22. During the training of the recurrent neural radiation field, each calibrated multi-channel RGB camera emits light for initial sampling point and resampling point calculation.

[0057] Furthermore, the recurrent neural radiance field is trained based on the calibrated camera parameters and the corrected multi-view images. Each pixel in the original image corresponds to a ray emitted by a multi-channel RGB camera, and the point coordinates, pixel color, and cumulative transparency distribution corresponding to each ray are used to minimize the calculation of the preset error function.

[0058] S23. During the training process of the cyclic neural radiation field, the cyclic neural radiation field is optimized and trained with the goal of minimizing a preset color error function and a resampling consistency error function.

[0059] Using the predicted pixel color of each ray and the corresponding original image pixel color, the recurrent neural radiation field is optimized and trained with the goal of minimizing a preset color error function.

[0060] The second position code and direction code corresponding to each resampling point are input into the recurrent neural radiation field, and the color and point density of each resampling point are calculated to generate a second cumulative transparency distribution.

[0061] Furthermore, the initial sampling point coordinates and the first cumulative transparency distribution, and the resampling point coordinates and the second cumulative transparency distribution on each ray are used to optimize the recurrent neural radiation field training with the goal of minimizing the preset resampling consistency error function.

[0062] S24. Complete the training of the cyclic neural radiation field by minimizing the preset error function.

[0063] The preset error functions include a color error function and a resampling consistency error function. During the training process of the recurrent neural radiation field, the predicted pixel color of each ray and the corresponding original image pixel color are used to optimize the training of the recurrent neural radiation field with the goal of minimizing the preset color error function. Based on the second position encoding and direction encoding corresponding to each resampling point, the recurrent neural radiation field is input, and the color and point density of each resampling point are calculated to generate a second cumulative transparency distribution. Using the initial sampling point coordinates and the first cumulative transparency distribution on each ray, and the resampling point coordinates and the second cumulative transparency distribution, the recurrent neural radiation field is optimized and trained with the goal of minimizing the preset resampling consistency error function.

[0064] Specifically, the recurrent neural radiation field is trained by minimizing the error function. The error function includes a color error function and a resampling consistency error function. The color error function can use methods such as the 1-norm, 2-norm, or structural similarity function to evaluate the difference between the predicted color and the true color. The resampling consistency error function is used to measure the consistency between the initial sampling point and the resampling point in position and point density. The cross entropy error function and other methods can be used to weakly constrain the point density weighted position similarity.

[0065] S25. Based on the light field coding method, the corresponding relationship between each sub-pixel and each rendering camera in the synthetic image is determined according to preset parameters of the three-dimensional light field display device, and each sub-pixel emits a rendering light.

[0066] After the recurrent neural network training is completed, in the rendering process of the synthetic image, based on different three-dimensional light field generation devices, the off-axis, convergent or parallel light field multi-viewpoint encoding methods are selected to set the number of cameras, camera arrangement and position, camera spacing and camera orientation of the virtual camera array. Each sub-pixel in the synthetic image corresponds to a rendering ray emitted by the rendering camera, and the position and orientation of the rendering ray to be calculated by the recurrent neural radiation field are finally determined to calculate the corresponding sub-pixel color.

[0067] S26. Collect multiple initial sampling points on the rendering light, and obtain a first position code corresponding to each initial sampling point through a position coding module.

[0068] A plurality of initial sampling points are collected on the rendering light. The collection method may perform rough sampling according to the position of the target object in the three-dimensional space to obtain the plurality of initial sampling points.

[0069] Furthermore, the first position code corresponding to each initial sampling point is obtained through the position coding module. The coordinates of the sampling points on each rendering ray during the sampling process are processed by a preset position coding module. This module maps the position of each sampling point to a high-dimensional feature, so that the network can more effectively utilize the position distribution of the sampling points and improve the performance of the recurrent neural network. ‌High-dimensional features‌ refer to mapping the original data to a new feature space in the feature extraction process, and the dimension of the feature is higher than the dimension of the original data.

[0070] The position encoding module can use methods such as sine and cosine encoding or hash encoding with different frequencies to perform position encoding on the coordinates of the initial sampling points obtained by the initial sampling.

[0071] S27. Based on the position encoding module, the multiple initial sampling points are mapped into a high-dimensional space as the first position encoding of the multiple initial sampling points, and input into the point density neural radiation field to calculate the point density and the first cumulative transparency distribution of the multiple initial sampling points.

[0072] The recurrent neural radiation field internally includes a point density neural radiation field defined by some of its parameters, including: in the recurrent neural radiation field, the position encoding of the input is calculated as a neural network of point density (point density neural radiation field), and then the point density is used to calculate and generate a cumulative transparency distribution according to a mathematical definition.

[0073] The first position code and the second position code are calculated according to the above process to obtain the first cumulative transparency distribution and the second cumulative transparency distribution.

[0074] S28. Resample according to the first cumulative transparency distribution data, and obtain a second position code and a direction code corresponding to each resampling point obtained by the resampling.

[0075] The resampling process is performed based on the first cumulative transparency distribution of the initial sampling point to ensure that the resampling point can more accurately reflect the geometric features contained in each ray. The initial sampling point can be resampled according to the Cumulative Distribution Function (CDF) method.

[0076] The position encoding module can use methods such as sine and cosine encoding or hash encoding of different frequencies to perform position encoding on each resampling point obtained by resampling, and obtain a second position code corresponding to each resampling point. The direction encoding module can use methods such as sine and cosine encoding of the direction vector, spherical harmonic function encoding, etc. to perform direction encoding on the direction vector corresponding to the resampling point, and obtain a direction code corresponding to each resampling point.

[0077] S29, perform feature fusion on the second position code and direction code, and then input the feature fusion result into the recurrent neural radiation field to calculate the color of each resampling point.

[0078] By defining a lightweight neural network model and introducing cyclic sampling into the learning process of the target pixel, the spatial distribution and color characteristics of the target scene can be captured more quickly. The advantage of this method is that the cyclic sampling process can better learn the geometric structure of the target scene, and the lightweight model can reduce the training time of the network model and generate high-quality light field content of the target scene.

[0079] During this process, the collaborative work of multiple encoding modules ensures that the position and direction encoding information of each resampling point can be smoothly transmitted to the recurrent neural radiation field network model, thereby calculating the corresponding color and point density.

[0080] S210 , calculating the sub-pixel color corresponding to the rendering light based on the color accumulation of each resampling point.

[0081] In the recurrent neural radiance field, the color of the resampled point on each ray is accumulated according to the transparency of the resampled point to calculate the sub-pixel color corresponding to this ray in the composite image.

[0082] S211 . Placing the sub-pixel colors into corresponding sub-pixel coordinates in the composite image based on the light field encoding method to realize three-dimensional light field content generation, and generating a target three-dimensional light field through a three-dimensional light field display device.

[0083] After determining the virtual camera information corresponding to all sub-pixels in the composite image, a rendering ray is emitted based on the rendering camera, and the sub-pixel color corresponding to the rendering ray is calculated through the point density neural network and the recurrent neural network.

[0084] Furthermore, the generated three-dimensional light field content is input into a three-dimensional light field display device to generate a three-dimensional light field.

[0085] The present invention provides a method for generating three-dimensional light field content of a recurrent neural radiation field. The method uses a multi-channel RGB camera to collect original images of a target scene under multiple viewing angles, performs camera calibration, and completes the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function. Based on a light field coding method, a rendering camera is set according to preset parameters of a three-dimensional light field display device, and each sub-pixel emits a rendering light ray. Multiple initial sampling points are collected on the rendering light ray, and a first position code corresponding to each initial sampling point is obtained through a position coding module. The first position codes corresponding to the multiple initial sampling points are input into a point density neural radiation field inside the recurrent neural radiation field, and the point density neural radiation The field generates point density, thereby generating a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the cyclic neural radiation field; resampling is performed according to the first cumulative transparency distribution data, and the second position code and direction code corresponding to each resampling point obtained by resampling are obtained; the color of each resampling point is calculated based on the second position code and direction code corresponding to each resampling point input into the cyclic neural radiation field; the sub-pixel color corresponding to the rendering light is calculated based on the color accumulation of each resampling point; the sub-pixel color corresponding to each rendering light is placed into the corresponding sub-pixel coordinates in the synthetic image based on the light field encoding method, thereby calculating the synthetic image, realizing the generation of three-dimensional light field content, and generating the target three-dimensional light field through the three-dimensional light field display device. By combining multi-view images, point density neural radiation fields, light sampling and resampling, neural network training, and three-dimensional light field rendering, this method realizes the rapid convergence of the neural network and the high-quality generation of three-dimensional light field content.

[0086] The following is a description of the circulatory neural radiation field three-dimensional light field content generation device provided by the present invention. The circulatory neural radiation field three-dimensional light field content generation device described below and the circulatory neural radiation field three-dimensional light field content generation method described above can be referenced to each other.

[0087] Figure 4 : is a schematic diagram of the structure of the device for generating the three-dimensional light field content of the recurrent neural radiation field provided by the present invention, which specifically includes: The acquisition module 401 is used to use multiple RGB cameras to acquire original images of the target scene under multiple viewing angles and perform camera calibration. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0088] The training module 402 is used to complete the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0089] The setting module 403 is used to set the rendering camera according to the preset parameters of the three-dimensional light field display device based on the light field encoding method, and each sub-pixel emits a rendering light. For detailed description, please refer to the relevant description of the above method embodiment, which will not be repeated here.

[0090] The sampling module 404 is used to collect multiple initial sampling points on the rendering light, and obtain the first position code corresponding to each initial sampling point through the position coding module. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0091] The calculation module 405 is used to input the first position codes corresponding to the multiple initial sampling points into the point density neural radiation field inside the cyclic neural radiation field, generate point density through the point density neural radiation field, and thus generate a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the cyclic neural radiation field. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0092] The sampling module 404 is further used to resample according to the first cumulative transparency distribution data, and obtain the second position code and direction code corresponding to each resampled point obtained by resampling. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0093] The calculation module 405 is further used to calculate the color of each resampling point based on the second position code and direction code input recurrent neural radiation field corresponding to each resampling point; and calculate the sub-pixel color corresponding to the rendering light based on the color accumulation of each resampling point. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0094] The generation module 406 places the sub-pixel color calculated corresponding to each rendering light into the corresponding sub-pixel coordinates of the composite image based on the light field encoding method, thereby calculating the composite image, realizing the generation of three-dimensional light field content, and generating the target three-dimensional light field through the three-dimensional light field display device. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0095] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the method for generating three-dimensional light field content of the recurrent neural radiation field, the method comprising: using a multi-channel RGB camera to collect original images of the target scene under multiple perspectives, performing camera calibration, and completing the training of the recurrent neural radiation field by minimizing a preset error function, the preset error function including a color error function and a resampling consistency error function; based on the light field coding method, setting the rendering camera according to the preset three-dimensional light field display device parameters, each sub-pixel emits a rendering light; collecting multiple initial sampling points on the rendering light, and obtaining the first position code corresponding to each initial sampling point through the position coding module; inputting the first position code corresponding to the multiple initial sampling points into the point density neural radiation field inside the recurrent neural radiation field. , generating a point density through the point density neural radiation field, thereby generating a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the cyclic neural radiation field; resampling is performed according to the first cumulative transparency distribution data, and a second position code and a direction code corresponding to each resampling point obtained by resampling are obtained; based on the second position code and the direction code corresponding to each resampling point, the cyclic neural radiation field is input to calculate the color of each resampling point; based on the color accumulation of each resampling point, the sub-pixel color corresponding to the rendering light is calculated; based on the light field encoding method, the sub-pixel color corresponding to each rendering light is placed in the corresponding sub-pixel coordinates in the synthetic image, thereby calculating the synthetic image, realizing the generation of three-dimensional light field content, and generating a target three-dimensional light field through a three-dimensional light field display device.

[0096] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the recurrent neural radiation field three-dimensional light field content generation method provided by the above-mentioned methods, and the method includes: using a multi-channel RGB camera to collect original images of the target scene under multiple perspectives, performing camera calibration, and completing the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function; based on the light field coding method, setting the rendering camera according to preset three-dimensional light field display device parameters, each sub-pixel emits a rendering ray; collecting multiple initial sampling points on the rendering ray, and obtaining the first position code corresponding to each initial sampling point through a position coding module; and converting the multiple initial sampling points into a first position code. The first position code corresponding to the sample point is input into the point density neural radiation field inside the circulatory neural radiation field, and the point density is generated by the point density neural radiation field, thereby generating a first cumulative transparency distribution, and the point density neural radiation field is defined by some internal parameters of the circulatory neural radiation field; resampling is performed according to the first cumulative transparency distribution data, and the second position code and direction code corresponding to each resampling point obtained by resampling are obtained; the second position code and direction code corresponding to each resampling point are input into the circulatory neural radiation field, and the color of each resampling point is calculated; the sub-pixel color corresponding to the rendering light is calculated based on the color accumulation of each resampling point; based on the light field encoding method, the calculated sub-pixel color corresponding to each rendering light is placed in the corresponding sub-pixel coordinates in the synthetic image, thereby calculating the synthetic image, realizing three-dimensional light field content generation, and generating a target three-dimensional light field through a three-dimensional light field display device.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the recurrent neural radiation field three-dimensional light field content generation method provided by the above-mentioned methods, the method comprising: using a multi-channel RGB camera to collect original images of the target scene under multiple perspectives, performing camera calibration, and completing the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function; based on the light field coding method, setting a rendering camera according to preset three-dimensional light field display device parameters, each sub-pixel emits a rendering ray; collecting multiple initial sampling points on the rendering ray, and obtaining a first position code corresponding to each initial sampling point through a position coding module; inputting the first position codes corresponding to the multiple initial sampling points into In the point density neural radiation field inside the circulatory neural radiation field, point density is generated through the point density neural radiation field, thereby generating a first cumulative transparency distribution, and the point density neural radiation field is defined by some internal parameters of the circulatory neural radiation field; resampling is performed according to the first cumulative transparency distribution data, and a second position code and a direction code corresponding to each resampling point obtained by resampling are obtained; based on the second position code and the direction code corresponding to each resampling point, the circulatory neural radiation field is input to calculate the color of each resampling point; based on the color accumulation of each resampling point, the sub-pixel color corresponding to the rendering light is calculated; based on the light field encoding method, the calculated sub-pixel color corresponding to each rendering light is placed in the corresponding sub-pixel coordinates in the synthetic image, thereby calculating the synthetic image, realizing three-dimensional light field content generation, and generating a target three-dimensional light field through a three-dimensional light field display device.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating three-dimensional light field content of a recurrent neural radiation field, characterized in that: include: Using a multi-channel RGB camera to collect original images of the target scene from multiple perspectives, perform camera calibration, and complete the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function; Based on the light field coding method, a rendering camera is set according to preset parameters of the three-dimensional light field display device, and each sub-pixel emits a rendering ray; Collecting a plurality of initial sampling points on the rendering light, and obtaining a first position code corresponding to each initial sampling point through a position coding module; Inputting first position codes corresponding to the plurality of initial sampling points into a point density neural radiation field inside the circulatory neural radiation field, generating point density through the point density neural radiation field, thereby generating a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the circulatory neural radiation field; Resampling is performed according to the first cumulative transparency distribution data, and a second position code and a direction code corresponding to each resampling point obtained by the resampling are obtained; Calculate the color of each resampling point based on the second position encoding and direction encoding input recurrent neural radiation field corresponding to each resampling point; Calculate the sub-pixel color corresponding to the rendering light based on the color accumulation of each resampling point; Based on the light field encoding method, the sub-pixel color calculated corresponding to each rendering light is placed in the corresponding sub-pixel coordinates in the synthetic image, so as to calculate the synthetic image, realize the generation of three-dimensional light field content, and generate the target three-dimensional light field through the three-dimensional light field display device.

2. The method according to claim 1, characterized in that The training of the cyclic neural radiation field is completed by minimizing a preset error function, including: During the training process of the recurrent neural radiation field, each calibrated multi-channel RGB camera emits light for initial sampling point and resampling point calculation, and each pixel in the original image corresponds to a light ray emitted by the multi-channel RGB camera, and the point coordinates, pixel color and cumulative transparency distribution corresponding to each light ray are used to calculate the minimum error function.

3. The method according to claim 2, characterized in that The method further comprises: During the training process of the recurrent neural radiation field, the predicted pixel color of each ray and the corresponding original image pixel color are used to optimize the recurrent neural radiation field with the goal of minimizing a preset color error function; Based on the second position-encoded and direction-encoded input recurrent neural radiation fields corresponding to each resampling point, the color and point density of each resampling point are calculated to generate a second cumulative transparency distribution; Using the initial sampling point coordinates and the first cumulative transparency distribution, and the resampling point coordinates and the second cumulative transparency distribution on each ray, the recurrent neural radiation field is optimized and trained with the goal of minimizing a preset resampling consistency error function.

4. The method according to claim 1, characterized in that: The light field encoding method is based on setting a rendering camera according to preset parameters of a three-dimensional light field display device, and each sub-pixel emits a rendering light, including: In the rendering process of the synthetic image, based on the light field coding method, the correspondence between each sub-pixel and each rendering camera in the synthetic image is determined according to the preset parameters of the three-dimensional light field display device, and then the corresponding virtual rendering camera emits rendering light to calculate the corresponding sub-pixel color.

5. The method according to claim 4, characterized in that The step of inputting the first position codes corresponding to the plurality of initial sampling points into the point density neural radiation field inside the cyclic neural radiation field, generating point density through the point density neural radiation field, and thus generating a first cumulative transparency distribution includes: Mapping the multiple initial sampling points to a high-dimensional space based on a position encoding module as first position encodings of the multiple initial sampling points; The first position code is input into the point density neural radiation field, and the point density of the multiple initial sampling points is calculated to generate a first cumulative transparency distribution of the initial sampling points on a rendering ray.

6. The method according to claim 5, characterized in that The step of calculating the color of each resampling point based on the second position encoding and direction encoding input recurrent neural radiation field corresponding to each resampling point comprises: Mapping the coordinates of the multiple resampling points into a high-dimensional space based on a position encoding module as second position encodings of the multiple resampling points; Mapping the direction vectors corresponding to the multiple resampling points to a high-dimensional space based on a direction encoding module as direction encodings of the multiple resampling points; The second position code and the direction code are feature fused, and the feature fusion result is input into the recurrent neural radiation field to calculate the color of each resampling point.

7. The method according to claim 6, characterized in that The method of placing the sub-pixel color calculated corresponding to each rendering light into the corresponding sub-pixel coordinates of the composite image based on the light field encoding method, thereby calculating the composite image, realizing the generation of three-dimensional light field content, and generating a target three-dimensional light field through a three-dimensional light field display device, includes: Based on the light field encoding method and the parameters of the preset three-dimensional light field display device, setting the parameters of the rendering camera array; Based on the rendering camera emitting rendering light, the sub-pixel color corresponding to the rendering light is calculated by the point density neural network and the recurrent neural network; Placing the sub-pixel colors into corresponding sub-pixel coordinates in the composite image based on the light field encoding method to achieve three-dimensional light field content generation; The three-dimensional light field content is input into a three-dimensional light field display device to generate a target three-dimensional light field.

8. A device for generating three-dimensional light field content of a circulatory neural radiation field, characterized in that: include: The acquisition module is used to use multiple RGB cameras to acquire original images of the target scene from multiple perspectives and perform camera calibration; A training module, used to complete the training of the recurrent neural radiation field by minimizing a preset error function, wherein the preset error function includes a color error function and a resampling consistency error function; A setting module, used to set a rendering camera according to preset parameters of a three-dimensional light field display device based on a light field encoding method, so that each sub-pixel emits a rendering ray; A sampling module, used for collecting a plurality of initial sampling points on the rendering light, and obtaining a first position code corresponding to each initial sampling point through a position coding module; a calculation module, configured to input the first position codes corresponding to the plurality of initial sampling points into a point density neural radiation field inside the circulatory neural radiation field, generate point density through the point density neural radiation field, and thus generate a first cumulative transparency distribution, wherein the point density neural radiation field is defined by some internal parameters of the circulatory neural radiation field; The sampling module is further used to perform resampling according to the first cumulative transparency distribution data, and obtain a second position code and a direction code corresponding to each resampling point obtained by the resampling; The calculation module is further used to calculate the color of each resampling point based on the second position encoding and direction encoding input recurrent neural radiation field corresponding to each resampling point; Calculate the sub-pixel color corresponding to the rendering light based on the color accumulation of each resampling point; A generation module is used to place the sub-pixel color calculated corresponding to each rendering light into the corresponding sub-pixel coordinates in the synthetic image based on the light field encoding method, thereby calculating the synthetic image, realizing three-dimensional light field content generation, and generating a target three-dimensional light field through a three-dimensional light field display device.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for generating three-dimensional light field content of a recurrent neural radiation field as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating three-dimensional light field content of a recurrent neural radiation field as described in any one of claims 1 to 7 is implemented.

Images