A neural radiated field network training method and related device

Abstract

The application discloses a neural radiance field network training method. In the method, in the i-th iteration process of training the neural radiance field network, the volume density and color information corresponding to each sample point in the i-th iteration process can be obtained according to the context information corresponding to each sample point in the plurality of sample points corresponding to the first view angle through the neural radiance field network in the i-th iteration process. Then, the first output image corresponding to the first view angle in the i-th iteration process is obtained through volume rendering according to the volume density and color information corresponding to each sample point in the i-th iteration process. Whether the training of the neural radiance field network is completed is determined according to the first output image and the first image. In this way, the trained neural radiance field network can fuse the context information between the sample points, learn the more accurate volume density and color information and other feature information of the sample points, and thus obtain a high-quality generated image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a neural radiance field network training method and related equipment. BACKGROUND

[0002] In various application scenarios such as autonomous driving, games, virtual reality, and augmented reality, it is often necessary to render an image of a new view in a specific scene.

[0003] At present, a traditional way to obtain an image of a new view is to perform three-dimensional scene modeling based on computer graphics, and to render a three-dimensional scene model through a rendering engine to obtain image data under a specific view. In this scheme, the image quality of the new view depends on the accuracy of the three-dimensional model and the ability of the related rendering engine, and if a large number of images under new views need to be generated, a large amount of resource cost will be spent. It can be seen that the current traditional image generation process of a new view is relatively complex and inefficient. SUMMARY

[0004] The present application provides a neural radiance field network training method, which can accurately and efficiently obtain an image under a new view in a specified scene after training. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products, etc.

[0005] The first aspect of the present application provides a neural radiance field network training method, which comprises: training a neural radiance field network according to a plurality of images and a view direction corresponding to each image, to obtain a trained neural radiance field network; wherein in the i-th iteration process of training the neural radiance field network: obtaining, by the neural radiance field network in the i-th iteration process, volume density and color information corresponding to each sample point in the i-th iteration process according to context information corresponding to each sample point in a plurality of sample points corresponding to a first view, the first view being determined based on a view direction of a first image, the plurality of sample points being obtained by sampling on at least one light ray corresponding to the first view, the first image being included in the plurality of images, and i being a positive integer; obtaining, by volume rendering, a first output image corresponding to the first view in the i-th iteration process according to the volume density and color information corresponding to each sample point in the i-th iteration process; and determining whether the training of the neural radiance field network is completed according to the first output image and the first image.

[0006] In the first aspect, through the related training process, the trained neural radiance field network can be enabled to fuse the context information between sample points, learn more accurate volume density and color information and other feature information of the sample points, obtain accurate output results, and further improve the quality of the finally generated image, reducing the influence of noise.

[0007] In a possible implementation manner of the first aspect, the context information corresponding to each sample point is obtained based on the neighborhood sample points in the plurality of sample points corresponding to the first view angle of the corresponding sample point.

[0008] In a possible implementation manner of the first aspect, the first image includes a plurality of image blocks, a plurality of light rays corresponding to each sample point are obtained by emitting light rays from a plurality of pixel points in the image block corresponding to the corresponding sample point in the first image corresponding to the camera, and the context information corresponding to the corresponding sample point is obtained based on the neighborhood sample points on the neighborhood light rays of the corresponding sample point, the neighborhood light rays are included in the plurality of light rays, and the neighborhood light rays are different from the light rays on which the corresponding sample point is located.

[0009] In this possible implementation manner, considering that in actual application, the pixel color, depth, and semantics of the image rendered at any angle for a scene have certain continuity, the image used for training can be divided into a plurality of image blocks, each image block can include a plurality of pixels, and in the training and inference process, a plurality of light rays can be obtained based on the image blocks in the image corresponding to the iteration process, the plurality of light rays are obtained by emitting light rays from a plurality of pixel points in any image block corresponding to the camera of the first image. In this way, in the training and inference process, the receptive field of the light rays corresponding to a sample point can be expanded to the range of the corresponding image block through convolution operation and the like, so as to associate the related context information on the neighborhood light rays of the light rays corresponding to the sample point to the sample point in the depth dimension, so that the neural radiance field network can fuse the information of the sample point and the related context information on the neighborhood light rays to obtain the color information and volume density of the sample point.

[0010] As can be seen, through this possible implementation manner, the receptive field of each light ray can be increased, so that the neural radiance field network can associate to the information of the neighborhood sample points on the neighborhood light rays corresponding to a sample point.

[0011] In a possible implementation manner of the first aspect, the context information corresponding to each sample point is obtained based on the corresponding neighborhood sample points on the light rays on which the corresponding sample point is located.

[0012] In a possible implementation of the first aspect, in actual application, for a view direction, the color and the volume density of a series of sample points sampled from the three-dimensional space in the view direction have certain continuity, therefore, the context information of one or more neighboring sample points corresponding to the same light ray is associated through a convolution operation or the like to obtain a more smooth estimated value corresponding to the sample point.

[0013] In a possible implementation of the first aspect, the volume density and the color information corresponding to each sample point in the i th iteration process are obtained by the neural radiance field network in the i th iteration process according to the context information corresponding to each sample point in the first view, including: the information of a corresponding sample point and the information of a corresponding neighboring sample point on a light ray where the corresponding sample point is located are fused according to the weight of the corresponding neighboring sample point on the light ray by the neural radiance field network in the i th iteration process to obtain the volume density and the color information corresponding to the corresponding sample point in the i th iteration process, wherein the weight of each neighboring sample point corresponding to the corresponding sample point on the light ray is determined based on the distance between the corresponding neighboring sample point and the corresponding sample point.

[0014] In a possible implementation of the first aspect, for a sample point, the distances between the sample point and different neighboring sample points corresponding to the sample point on the light ray where the sample point is located can be different, and it can be seen that the influence degrees of different neighboring sample points on the sample point are different. Therefore, the distance between the sample points can be used as a weight of the relevant neighboring sample points in information fusion for relevant fusion operations such as convolution operations.

[0015] In a possible implementation of the first aspect, the neural radiance field network includes at least one convolution layer, and the at least one convolution layer is configured to fuse the information of each sample point and the context information corresponding to the corresponding sample point through a convolution operation.

[0016] In a possible implementation of the first aspect, the at least one convolution layer includes a plurality of convolution layers, the plurality of convolution layers are in a serial structure, and the size of the output of each convolution layer in the plurality of convolution layers in at least one dimension is not greater than the size of the output of the corresponding next convolution layer in at least one dimension.

[0017] In a possible implementation of the first aspect, the coverage range of the plurality of light rays corresponding to the first view can be expanded, so that the information extraction range of the context information is expanded, and the neural radiance field network can learn scene information from a large-size image, and the performance of the neural radiance field network in a large scene is improved.

[0018] In a possible implementation manner of the first aspect, the determining whether the training of the neural radiance field network is completed according to the first output image and the first image comprises: obtaining a weight corresponding to each pixel point in the first output image through an output of a kth layer in the neural radiance field network in the ith iteration process, k being a positive integer less than a preset threshold; obtaining a target pixel value of at least one target pixel point in the first output image according to the first output image and the weight corresponding to each pixel point in the first output image; and determining whether the training of the neural radiance field network is completed according to the target pixel value of the at least one target pixel point in the first output image and the first image.

[0019] In a possible implementation manner of the first aspect, the determining whether the training of the neural radiance field network is completed according to the first output image and the first image comprises: obtaining first feature information of each feature mapping network in the plurality of feature mapping networks corresponding to the ith iteration process with respect to the first output image, any feature mapping network being used for mapping an input to a corresponding feature space, and different feature mapping networks corresponding to different feature spaces; obtaining second feature information of each feature mapping network in the plurality of feature mapping networks corresponding to the ith iteration process with respect to the first image; and determining whether the training of the neural radiance field network is completed according to the first feature information and the second feature information.

[0020] In the training process of the conventional neural radiance field network, the peak signal-to-noise ratio (PSNR) is usually used alone as the loss function of the neural radiance field network, which ignores the similarity between images in the local and global aspects, and may make the evaluation result of the loss function more one-sided and less accurate. If the loss function of the neural radiance field is constructed according to multiple performance indicators such as the peak signal-to-noise ratio, the structural similarity (SSIM), and the perceptual similarity (LPIPS), the multiple performance indicators (the peak signal-to-noise ratio, the structural similarity, and the perceptual similarity) are heterogeneous in the hypothesis space, and do not converge in the same direction in the network optimization process. Therefore, the effect of the loss function of the neural radiance field constructed according to the multiple performance indicators such as the peak signal-to-noise ratio, the structural similarity, and the perceptual similarity is not ideal.

[0021] In the possible implementation manner, in the training process, the predicted value and the true value of the image are mapped to different feature spaces through the plurality of feature mapping networks, the accuracy of the generated image in the high-dimensional feature space and the multiple feature spaces can be evaluated, and the performance of the trained neural radiance field network is better.

[0022] In a possible implementation manner of the first aspect, the neural radiance field network is trained according to the plurality of images and the corresponding view direction of each image, to obtain the trained neural radiance field network, including: the neural radiance field network and a plurality of initial feature mapping networks are alternately trained according to each image and the corresponding view direction of each image, to obtain a plurality of trained initial feature mapping networks and the trained neural radiance field network, and the plurality of trained initial feature mapping networks are taken as the plurality of feature mapping networks.

[0023] In the possible implementation manner, the alternately training the neural radiance field network and the plurality of initial feature mapping networks can be: first, fixing the plurality of initial feature mapping networks, and performing one or more iteration training on the neural radiance field network; then, fixing the latest updated neural radiance field network, and performing one or more iteration training on the plurality of initial feature mapping networks; then, fixing the latest updated plurality of initial feature mapping networks, and performing one or more iteration training on the latest updated neural radiance field network, and so on, until a preset iteration number is reached, or until the iteration neural radiance field network and the initial feature mapping network respectively converge to the respective expected state.

[0024] In a possible implementation manner of the first aspect, the neural radiance field network and the plurality of initial feature mapping networks are alternately trained according to each image and the corresponding view direction of each image, to obtain a plurality of trained initial feature mapping networks and the trained neural radiance field network, and the plurality of trained initial feature mapping networks are taken as the plurality of feature mapping networks, including: in the jth iteration process of training the plurality of initial feature mapping networks: the second output image in the jth iteration process is obtained through the neural radiance field network in the jth iteration process, j is a positive integer; the third feature information of each initial feature mapping network about the second output image in the plurality of initial feature mapping networks corresponding to the jth iteration process is obtained; and whether the training of the plurality of initial feature mapping networks is completed is determined based on the difference between each third feature information.

[0025] In the possible implementation manner, whether the training of the plurality of initial feature mapping networks is completed can be determined based on the difference between each third feature information and a second loss function. The second loss function is used to make each third feature information as diverse as possible, that is, to make the difference between each third feature information tend to increase. In this way, the feature spaces corresponding to the plurality of feature mapping networks obtained through the training of the plurality of initial feature mapping networks are obviously different, thereby keeping the diversity of the feature spaces corresponding to the plurality of feature mapping networks.

[0026] It can be seen that in the possible implementation manner, the plurality of trained initial feature mapping networks can accurately evaluate the image similarity between the output image of the neural radiance field network and the corresponding ground truth image at a plurality of different feature space levels, and the performance of the finally obtained trained neural radiance field network is better, and the generated image quality is better.

[0027] The second aspect of the present application provides an image generation method, which comprises: obtaining, by a neural radiance field network, volume density and color information corresponding to each sample point according to context information corresponding to each sample point in a plurality of sample points corresponding to a target view angle, the plurality of sample points being obtained by sampling on at least one light ray corresponding to the target view angle; and obtaining an output image corresponding to the target view angle by volume rendering according to the volume density and color information corresponding to each sample point.

[0028] In a possible implementation manner of the second aspect, the context information corresponding to each sample point is obtained based on neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the target view angle.

[0029] In a possible implementation manner of the second aspect, the corresponding sample point corresponds to a plurality of light rays, the context information corresponding to the corresponding sample point is obtained based on neighborhood sample points on a neighborhood light ray of the corresponding sample point, the neighborhood light ray is included in the plurality of light rays, and the neighborhood light ray is different from the light ray on which the corresponding sample point is located.

[0030] In a possible implementation manner of the second aspect, the context information corresponding to each sample point is obtained based on corresponding neighborhood sample points on the light ray on which the corresponding sample point is located.

[0031] In a possible implementation manner of the second aspect, the obtaining, by the neural radiance field network, of the volume density and color information corresponding to each sample point according to the context information corresponding to each sample point in the plurality of sample points corresponding to the target view angle comprises: fusing, by the neural radiance field network, information of the corresponding sample point and information of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located according to weights of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located, to obtain the volume density and color information corresponding to the corresponding sample point, wherein the weight of each corresponding neighborhood sample point on the light ray on which the corresponding sample point is located is determined based on a distance between the corresponding neighborhood sample point and the corresponding sample point.

[0032] In a possible implementation manner of the second aspect, the neural radiance field network comprises at least one convolution layer, and the at least one convolution layer is configured to fuse information of each sample point and context information corresponding to the corresponding sample point by a convolution operation.

[0033] In a possible implementation form of the second aspect, the at least one convolutional layer comprises a plurality of convolutional layers, the plurality of convolutional layers are in a serial structure, and a size of an output of any convolutional layer in the plurality of convolutional layers in at least one dimension is not greater than a size of an output of a corresponding next convolutional layer in at least one dimension.

[0034] The third aspect of the present application provides a neural radiance field network training apparatus, which has the function of implementing the method of the first aspect or any possible implementation form of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a training module, etc.

[0035] The fourth aspect of the present application provides an image generation apparatus, which has the function of implementing the method of the second aspect or any possible implementation form of the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a neural radiance field network module and a volume rendering module, etc.

[0036] The fifth aspect of the present application provides an electronic device, which includes at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor executes the method of the first aspect or any possible implementation form of the first aspect, and / or executes the method of the second aspect or any possible implementation form of the second aspect.

[0037] The sixth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method of the first aspect or any possible implementation form of the first aspect, and / or executes the method of the second aspect or any possible implementation form of the second aspect.

[0038] The seventh aspect of the present application provides a computer program product storing one or more computer-executable instructions. The computer program product contains computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method of the first aspect or any possible implementation form of the first aspect, and / or executes the method of the second aspect or any possible implementation form of the second aspect.

[0039] The eighth aspect of the present application provides a chip system, which comprises a processor for supporting the electronic device to realize the functions involved in the first aspect or any possible implementation manner of the first aspect, and / or realize the functions involved in the second aspect or any possible implementation manner of the second aspect. In a possible design, the chip system can further comprise a memory for storing necessary program instructions and data of the electronic device. The chip system can be composed of a chip, or can comprise a chip and other discrete devices.

[0040] The technical effects brought by the second aspect to the eighth aspect or any possible implementation manner thereof can refer to the technical effects brought by the first aspect or the related possible implementation manner of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is an exemplary schematic diagram of a neural radiance field network provided by an embodiment of the present application;

[0042] Figure 2 is an exemplary schematic diagram of a light ray in volume rendering provided by an embodiment of the present application;

[0043] Figure 3 is an exemplary schematic diagram of a neural radiance field network training method provided by an embodiment of the present application;

[0044] Figure 4 is an exemplary schematic diagram of a neural radiance field network training method provided by an embodiment of the present application;

[0045] Figure 5a is an exemplary schematic diagram of a receptive field corresponding to a light ray in a conventional neural radiance field network provided by an embodiment of the present application;

[0046] Figure 5b is an exemplary schematic diagram of context information between neighboring light rays provided by an embodiment of the present application;

[0047] Figure 6 is an exemplary schematic diagram of fusing context information between neighboring sample points on the same light ray provided by an embodiment of the present application;

[0048] Figure 7 is an exemplary schematic diagram of a neural radiance field network provided by an embodiment of the present application;

[0049] Figure 8 is an exemplary schematic diagram of a plurality of feature mapping networks provided by an embodiment of the present application;

[0050] Figure 9a is an exemplary training schematic diagram of the first stage provided by an embodiment of the present application;

[0051] Figure 9b is an exemplary training schematic diagram of a second stage provided by an embodiment of the present application;

[0052] Figure 10 is an exemplary iteration process schematic diagram provided by an embodiment of the present application;

[0053] Figure 11 is an embodiment schematic diagram of an image generation method provided by an embodiment of the present application;

[0054] Figure 12 is an embodiment schematic diagram of a neural radiation field network training apparatus provided by an embodiment of the present application;

[0055] Figure 13 is an embodiment schematic diagram of an image generation apparatus provided by an embodiment of the present application;

[0056] Figure 14 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0057] The embodiments of the present application are described below with reference to the accompanying drawings. It is obvious to those skilled in the art that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0058] In the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attribute. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the process, method, system, product or device containing a series of units does not have to be limited to those units, but can include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0059] In various application scenarios such as autonomous driving, gaming, virtual reality, and augmented reality, it is often necessary to render images of new perspectives in a specific scene.

[0060] At present, a traditional way to obtain images of new perspectives is to model a three-dimensional scene based on computer graphics, and render the three-dimensional scene model through a rendering engine to obtain image data at a specific perspective. In this scheme, the image quality of the new perspective depends on the accuracy of the three-dimensional model and the capability of the related rendering engine, and if a large number of images at new perspectives need to be generated, a large amount of resource cost will be required. It can be seen that the current traditional image generation process of new perspectives is relatively complex and inefficient.

[0061] With the rapid rise of neural radiance fields (NeRF) in the field of computer vision in the past two years, a new perspective image generation method based on neural radiance fields has emerged.

[0062] Among them, neural radiance fields is a new method of scene representation and image rendering. Neural radiance fields record scene representation in a deep neural network through implicit expression, use a deep neural network to implicitly learn the information of a three-dimensional scene, and complete tasks such as three-dimensional reconstruction and new perspective image generation of the three-dimensional scene.

[0063] Reference Figure 1 As shown in the schematic diagram, the input of the neural radiance field network F Ω may include the spatial position (x, y, z) of a spatial point and the perspective direction where θ can be the camera pose corresponding to the perspective direction, may be the camera intrinsic corresponding to the perspective direction. Through the neural radiance field network F Ω , the corresponding output result (r, g, b, σ) can be obtained. In the output result, (r, g, b) represents the color information corresponding to the corresponding spatial point, and σ represents the volume density of the corresponding spatial point. The volume density σ(x) can be understood as the probability that a ray r is terminated when passing through an infinitesimal particle at x, and the probability is differentiable. The volume density of a certain spatial point is similar to the opacity of the corresponding spatial point.

[0064] After obtaining the color information and volume density of multiple spatial points, the corresponding image data and spatial depth information can be inferred and rendered from any new perspective through volume rendering technology according to the color information and volume density of the multiple spatial points.

[0065] Volume rendering technology refers to that for any light ray emitted from the camera perspective in the scene, the volume density and color information of all points on the light ray can be accumulated in the form of integration, and finally the pixel color and depth information corresponding to the light ray are obtained. For example,Figure 2 As shown, the implemented process can realize discrete sample space modeling by sampling a limited number of sample points on the ray and in an accumulated manner to obtain corresponding image data and depth information of the image data.

[0066] At present, the direct input of the commonly used neural radiance field network is the information of the sample points on the light ray, and each sample point is trained and estimated separately through the neural radiance field network, which leads to noise in the depth of the scene reconstructed by the neural radiance field network, so that the noise of the finally synthesized image is also relatively large, and the image quality is affected.

[0067] Based on this, the neural radiance field network provided in the embodiments of the present application can obtain relatively accurate color information and volume density of a corresponding sample point according to the feature information and the corresponding context information of the sample point, so as to obtain a high-quality generated image.

[0068] The specific structure of the neural radiance field network in the embodiments of the present application is not limited here. Exemplarily, the neural radiance field network can include a multilayer perceptron (MLP) to realize the latent representation of a specified three-dimensional scene by learning a series of images of the specified three-dimensional scene, so as to generate an image corresponding to a new perspective of the specified scene.

[0069] The neural radiance field network of the embodiments of the present application can be applied to various application fields that need image generation, for example, can be applied to the field of autonomous driving, the field of virtual reality, the field of augmented reality, the field of high-precision three-dimensional map surveying and mapping, and other fields that need three-dimensional scene construction.

[0070] The embodiments of the present application relate to the training phase and the inference phase of the neural radiance field network. The training phase and the inference phase will be introduced respectively as follows.

[0071] First, the related operations of the training phase of the neural radiance field network are introduced.

[0072] In the embodiments of the present application, the related operations of the training phase of the neural radiance field network can be performed by an electronic device. The electronic device performing the related operations of the training phase can be the same as or different from the electronic device performing the related operations of the inference phase.

[0073] The specific type of the electronic device performing the related operations of the training phase is not limited here. Exemplarily, the electronic device can be a single server, a server cluster, a terminal device, or a virtual machine (VM) or a container.

[0074] Exemplarily, if the electronic device is a terminal device, the type of the terminal device can be a mobile phone, a pad, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a terminal in industrial control, a terminal in self driving, a terminal in remote medical, a terminal in smart grid, a terminal in transportation safety, a terminal in smart city, a terminal in smart home, a terminal in internet of things (IoT), and the like.

[0075] As shown in Figure 3 In the embodiments of the present application, the neural radiance field network training method performed by the electronic device can include step 301.

[0076] Step 301: training a neural radiance field network according to a plurality of images and a view direction corresponding to each image, to obtain a trained neural radiance field network.

[0077] In the embodiments of the present application, the specific type of the plurality of images can have multiple cases, and the specific acquisition method of the plurality of images can also have multiple cases, which are not limited here.

[0078] Exemplarily, any image can be a red green blue (RGB) image or a YUV image. Among them, "Y" in the YUV image represents luminance or luma, that is, a gray scale value, and "U" and "V" represent chrominance or chroma. Alternatively, any image can also be a depth image acquired by a depth camera or an image acquired by an event camera, etc. Generally, the specific type of each image in the plurality of images can be the same.

[0079] The plurality of images can be obtained by image acquisition on the same scene.

[0080] The view direction of any image can include camera pose and other parameter information such as focal length corresponding to the image.

[0081] In the prior art, the direct input of the neural radiance field network is discrete samples on the light rays emitted by the pixels of the independent images, and the context information between the samples is ignored by the neural radiance field network, which leads to that the finally rendered image is easily affected by random noise and the quality of the finally generated image is affected.

[0082] In the embodiment of the present application, through the related training process, the trained neural radiance field network can fuse the context information between the sample points, learn the more accurate volume density and color information of the sample points, and obtain accurate output results, thereby improving the quality of the finally generated image and reducing the influence of noise.

[0083] The training process will be described in detail below.

[0084] Specifically, as shown in the figure, Figure 4 The process of training the neural radiance field network can include multiple iteration processes, and in the i-th iteration process of training the neural radiance field network, steps 3011-3013 can be included.

[0085] In step 3011, the volume density and color information corresponding to each sample point in the i-th iteration process are obtained by the neural radiance field network in the i-th iteration process according to the context information corresponding to each sample point in the first view.

[0086] The first view is determined based on the view direction of the first image, the plurality of sample points are obtained by sampling at least one light ray corresponding to the first view, the first image is included in the plurality of images, and i is a positive integer.

[0087] In the embodiment of the present application, the i-th iteration process can be one of the multiple iteration processes of training the neural radiance field network, and the operations of other iteration processes in the multiple iteration processes can be the same as or similar to the i-th iteration process, or there can be differences.

[0088] The first view can be considered as the observation angle corresponding to the first image in the world coordinate system, and specifically, the pose of the camera for shooting the first image and the focal length and other intrinsic parameters of the camera can be used to determine the first view.

[0089] In the i-th iteration process, the light ray corresponding to the first view is determined based on the first image corresponding to the first view. In the i-th iteration process, the number and form of the light rays corresponding to the first view are not limited.

[0090] In an example, to facilitate the neural radiance field network to obtain context information between sample points, the first image corresponding to the first view angle can be divided into a plurality of image blocks, and each image block includes a plurality of pixel points.

[0091] In the i-th iteration process, an image block can be determined from the plurality of image blocks, and a plurality of light rays corresponding to the i-th iteration process can be obtained according to the plurality of pixel points in the image block. Any light ray can be a ray from the optical center of the camera corresponding to the first image to a pixel point in the corresponding image block. In this way, a plurality of light rays corresponding to the image block can be obtained as the plurality of light rays under the first view angle corresponding to the i-th iteration process.

[0092] In the embodiments of the present application, the plurality of sample points corresponding to the first view angle are obtained by sampling at least one light ray corresponding to the first view angle. The sampling method is not limited herein, for example, uniform sampling or non-uniform sampling can be performed.

[0093] In an example, the neural radiance field network can include a "coarsetofine" network structure, which includes a coarse network and a fine network. The structures of the coarse network and the fine network can be the same, for example, both can include a multi-layer perceptron. The input of the coarse network can be the spatial position and view direction information of the uniformly sampled sample points. According to the output of the coarse network (for example, the volume density and / or color information of the corresponding sample points) of the uniformly sampled sample points, the weight corresponding to the corresponding sample point is obtained, and the input of the fine network is obtained by non-uniform sampling according to the weight corresponding to the corresponding sample point.

[0094] In addition, in some examples, considering that the result obtained by taking the spatial position and view direction of each sample point as the input of the neural radiance field network is relatively fuzzy, the spatial position and view direction of the sample points can be mapped to a high-dimensional space by using positional encoding before being input into the neural radiance field network, so that the neural radiance field network can better fit the data containing high-frequency changes, thereby effectively improving the performance of the neural radiance field network and improving the clarity of the synthesized image.

[0095] In the embodiments of the present application, for each sample point in the plurality of sample points, the context information corresponding to the sample point can be fused by the neural radiance field network in the i-th iteration process to obtain the volume density and color information corresponding to the sample point in the i-th iteration process.

[0096] In some embodiments, the context information corresponding to each sample point is obtained based on neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the first view angle.

[0097] In this way, the neural radiance field network can fuse the feature information of a sample point and the feature information of the corresponding neighborhood sample points, so as to obtain the volume density and color information corresponding to the sample point according to the context information of the sample point.

[0098] In some embodiments, the neighborhood sample points of a sample point in the plurality of sample points corresponding to the first view angle can include sample points on a light ray where the sample point is located, one or more neighborhood sample points of the sample point, and / or sample points at the same depth or similar depth on one or more neighborhood light rays of the light ray where the sample point is located. In some embodiments, the sample points at the same depth or similar depth on the one or more neighborhood light rays of the light ray where the sample point is located can also be understood as neighborhood sample points of the sample point in the image plane dimension of the first image corresponding to the sample point.

[0099] In some embodiments, the neural radiance field network can include a convolution layer, so that the context information of the sample point can be fused by a convolution operation on the context information of the sample point. Alternatively, the context information of the sample point can also be fused by a structure similar to the context information extraction structure in the transformer network.

[0100] In some embodiments, the neural radiance field network includes at least one convolution layer, and the at least one convolution layer is configured to fuse the information of each sample point with the context information corresponding to the respective sample point by a convolution operation.

[0101] The input of the at least one convolution layer can be obtained based on the information of the sample point and the context information corresponding to the respective sample point. For example, the at least one convolution layer can be a plurality of convolution layers in series, and the input of a first convolution layer in the plurality of convolution layers in series can be a tensor obtained based on the spatial position and view angle direction of the sample point and the spatial position and view angle direction of the neighborhood sample points corresponding to the respective sample point. In this way, the context information of the respective sample point can be fused by the plurality of convolution layers in series.

[0102] The number and parameters of the convolution layer in the neural radiance field network are not limited herein. For example, the size of the convolution kernel in the at least one convolution layer can be set based on the actual scene and experience. If there are a plurality of convolution layers, the sizes of the convolution kernels of the plurality of convolution layers can be the same, for example, each being 5*5. Alternatively, the sizes of the convolution kernels of the plurality of convolution layers can be different.

[0103] In some embodiments, at least one convolutional layer includes multiple convolutional layers, which are in a serial structure, and the output of any convolutional layer in the multiple convolutional layers is no greater than the output of the corresponding next convolutional layer in at least one dimension.

[0104] The output of any convolutional layer is typically two-dimensional or higher. For example, the output of any convolutional layer can be a three-dimensional matrix, including length, width, and depth. In this embodiment, the output of any convolutional layer in a multi-layered system is at least as large as the output of the next convolutional layer in at least one dimension, where at least one dimension may include length and / or width. That is, in this embodiment, the output dimension of the multi-layered convolutional layers can gradually increase with the increase in network depth. This can improve the upper limit of the coverage of multiple light rays corresponding to the first viewpoint, expand the range of contextual information extraction, and thus facilitate the neural radiation field network to learn scene information from large-size images, improving the performance of the neural radiation field network in large scenes.

[0105] Furthermore, in some examples, the serial structure formed by the multi-layer convolutional layers may also include other layers. For example, in one example, an activation function layer may be included between any two adjacent convolutional layers.

[0106] In this embodiment, the specific content of the contextual information of the sample points fused by the neural radiation field can be varied. Examples are given below.

[0107] 1. Merge contextual information between neighboring light rays.

[0108] In some embodiments, the first image includes multiple image blocks, and each sample point corresponds to multiple light rays. The multiple light rays are obtained by the camera corresponding to the first image emitting light rays to multiple pixels in the image block corresponding to the corresponding sample point.

[0109] The context information corresponding to the sample point is obtained based on the neighboring sample points on the neighboring ray. The neighboring ray is contained in multiple rays, and the neighboring ray is different from the ray where the corresponding sample point is located.

[0110] like Figure 5a As shown, in traditional neural radiation field networks, the receptive field corresponding to a certain ray of light is the range of the corresponding single pixel point during each training and inference process.

[0111] In the embodiment of the present application, considering that in actual application, the pixel color, depth and semantics of the image rendered at any angle for a scene have certain continuity, the image used for training can be divided into multiple image blocks, each of which can include multiple pixels, and in the training and inference process, multiple light rays can be obtained based on the image blocks in the image corresponding to the iteration process, which are obtained by emitting light rays from the camera corresponding to the first image to the multiple pixels in any image block. In this way, in the training and inference process, the receptive field of the light rays corresponding to a sample point can be expanded to the range of the corresponding image block through convolution operation and the like, so as to associate the related context information on the neighborhood light rays of the sample point to the sample point in the depth dimension, so that the neural radiance field network can fuse the information of the sample point and the related context information on the neighborhood light rays to obtain the color information and the volume density of the sample point.

[0112] Take the i th iteration process as an example for introduction.

[0113] In the i th iteration process, the neighborhood sample points of a sample point on the neighborhood light rays can be sample points on one or more neighborhood light rays of the light ray where the sample point is located, which are located at the same depth or adjacent depth, and can also be understood as neighborhood sample points of the sample point in the image plane dimension of the first image corresponding to the sample point. In an example, the number of sample points on the multiple light rays corresponding to the first view angle is the same, at this time, the neighborhood sample points of a sample point on the neighborhood light rays can also be determined based on the ordering of the sample points on the multiple light rays. For example, if a sample point is the 10 th sample point on the corresponding light ray, then the 10 th sample points on the corresponding neighborhood light rays are all neighborhood sample points of the sample point.

[0114] For example, as shown in an example in Figure 5b The plane A and the plane B can be considered as planes overlapping or parallel to the image plane of the first image, or the depths of the points on the plane A relative to the camera are substantially consistent, and the depths of the points on the plane B relative to the camera are substantially consistent. The sample point a under the first view angle corresponds to 5 light rays, including the light ray where the sample point a is located and 4 neighborhood light rays. At this time, the sample point a corresponds to 4 neighborhood sample points on the neighborhood light rays, and the 4 neighborhood sample points and the sample point a can be located on the plane A. In this way, the neural radiance field network can fuse the information of the 4 neighborhood sample points and the sample point a to obtain the volume density and color information corresponding to the sample point a.

[0115] It can be seen that, by the embodiments of the present application, the receptive field of each light ray can be increased, so that the neural radiance field network can be associated with the information of the neighborhood sample points on the neighborhood light rays corresponding to a sample point.

[0116] At this time, even if the direction of the light ray is disturbed due to noise or the like, as long as the intersection point of the disturbed light ray and the pixel point on the corresponding image is located inside the same pixel point, the receptive field of this light ray is the same, the color information and the volume density of the corresponding sample point obtained are stable, and the result of the final image rendering should also be the same. Therefore, in the training process of the embodiments of the present application, the neural radiance field network can effectively aggregate the context information between the neighborhood light rays, thereby facilitating the training of the entire scene consistency and continuity.

[0117] In addition, the receptive field of the light ray corresponding to the traditional neural radiance field network is a single ray, and therefore, volume rendering is required to perform rendering once according to the color information and the volume density of the multiple sample points on the single ray to obtain the information of a pixel point in the corresponding first output image. It can be seen that, if an H*W image is rendered, H*W rendering operations are required in the traditional image generation method based on the neural radiance field network.

[0118] In the embodiments of the present application, however, the receptive field of a single light ray is expanded to the range of a corresponding image block through convolution operation or the like in the training and inference process, and therefore, after the color information and the volume density of the sample points on the multiple light rays are obtained by the neural radiance field network, the first output image block corresponding to the multiple light rays can be obtained through volume rendering. If the size of the image block is h*w, then (H*W) / (h*w) rendering operations are required to render an H*W image.

[0119] 2. fuse the context information between the neighborhood sample points on the same light ray.

[0120] In some embodiments, the context information corresponding to each sample point is obtained based on the corresponding neighborhood sample points on the light ray where the corresponding sample point is located.

[0121] In the traditional neural radiance field network, each sample point on each light ray is respectively input into the neural radiance field network to estimate the volume density and the color information in each training and inference process.

[0122] In the embodiments of the present application, it is considered that, in actual application process, for a view direction, the color and volume density of a series of sample points obtained by sampling from three-dimensional space in the view direction have certain continuity, therefore, the context information of one or more neighborhood sample points corresponding to the same light ray is associated by means of convolution operation or the like to obtain a more smooth estimated value corresponding to the sample point.

[0123] For example, in the example as shown in Figure 6 , the information of sample point B and the information of neighborhood sample points A and C of sample point B on the light ray can be fused by convolution operation to obtain the volume density and color information of sample point B.

[0124] And, as shown in the curve diagram in Figure 6 , in the training process, the neighborhood sample points A and C of sample point B on the light ray can be associated by neural radiance field network to obtain the more smooth volume density corresponding to sample point B.

[0125] In some embodiments, the above step 3011 comprises:

[0126] The information of the corresponding sample point and the information of the corresponding neighborhood sample points of the corresponding sample point on the light ray are fused by the neural radiance field network in the i th iteration process according to the weight of the corresponding neighborhood sample points on the light ray of the sample point to obtain the volume density and color information corresponding to the corresponding sample point in the i th iteration process, wherein the weight of each neighborhood sample point on the light ray of the corresponding sample point is determined based on the distance between the corresponding neighborhood sample point and the corresponding sample point.

[0127] In the embodiments of the present application, for a sample point, the distances between the sample point and different neighborhood sample points corresponding to the sample point on the light ray may be different, and it can be seen that the influence degrees between different neighborhood sample points and the sample point are different. Therefore, in the embodiments of the present application, the distance between the sample points can be used as a weight of the related neighborhood sample points in information fusion to perform related fusion operation.

[0128] For example, in one example, in the scene as shown in Figure 6 , the context information of sample point B on the light ray is associated by the neural radiance field network through convolution operation, and then the convolution operation can be performed based on the following convolution operator f(B):

[0129]

[0130] In the convolution operator f(B) about the sample point B, W(·B) is a weight in the corresponding convolution kernel, represents a multiplication operation of corresponding elements in the convolution operation, represents an element in the context feature tensor corresponding to the sample point B, the context feature tensor contains feature information of the neighborhood sample point A and / or feature information of the neighborhood sample point C, d(·B) represents a distance between the corresponding element and the sample point B, and in the example shown in the formula, the distance between the neighborhood sample point A and the sample point B is d1, and the distance between the neighborhood sample point C and the sample point B is d2. norm(·) represents normalization, and cos(·) represents a cosine function. Through the processing of normalization and the cosine function, the processing result can be relatively smooth and within [0, 1]. Figure 6

[0131] It can be seen that in the embodiments of the present application, in the training process and even in the inference process, the neural radiance field network can fuse the context information between the neighborhood light rays to increase the receptive field of the light rays, and can also fuse the context information between the neighborhood sample points on the same light ray to improve the continuity of the estimated values of the color information and the volume density of the sample points, so that the scene information learned by the trained neural radiance field network reduces the interference of noise, thereby improving the quality of the finally generated image.

[0132] It should be noted that in the embodiments of the present application, the neural radiance field network can fuse one or more of the context information between the neighborhood light rays and the context information between the neighborhood sample points on the same light ray, and the embodiments of the present application do not limit this.

[0133] In step 3012, according to the volume density and color information corresponding to each sample point in the i-th iteration process, a first output image corresponding to the first view angle in the i-th iteration process is obtained through volume rendering.

[0134] In the embodiments of the present application, after obtaining the volume density and color information corresponding to each sample point in the i-th iteration process, the pixel information corresponding to each light ray under the first view angle can be obtained through volume rendering according to the volume density and color information corresponding to each sample point in the i-th iteration process, so as to obtain the first output image corresponding to the first view angle in the i-th iteration process according to each pixel information.

[0135] The first output image can be an image corresponding to the first view angle estimated by the neural radiance field network and the volume rendering technology in the i-th iteration process.

[0136] In step 3013, whether the training of the neural radiance field network is completed is determined according to the first output image and the first image.

[0137] ​In the embodiments of the present application, the difference between the first output image and the first image can be evaluated by a preset loss function to determine whether the training of the neural radiance field network is completed.

[0138] Specifically, if the loss value of the i th iteration process is determined to converge to an expected state based on the loss function, it can be determined that the training of the neural radiance field network is completed; or if the number of training reaches a preset number, it can also be determined that the training of the neural radiance field network is completed.

[0139] If it is determined that the training is not completed, the neural radiance field network can be updated based on the corresponding loss value by means of back propagation, etc., to take the updated neural radiance field network as the neural radiance field network in the i+1 th iteration process, and continue the training by referring to the i th iteration process.

[0140] In the embodiments of the present application, there can be multiple ways to evaluate the difference between the first output image and the first image.

[0141] For example, in one example, the difference between each pixel point in the first output image and each pixel point in the first image can be compared.

[0142] In another example, the weight corresponding to the pixel point in the first output image can be determined based on the intermediate layer information of the neural radiance field network, so as to determine the target pixel value of a part of the pixel points in the first output image. Then, the image similarity between the first output image and the first image can be determined based on the target pixel value of the part of the pixel points and the pixel value of the corresponding part of the pixel points in the first image.

[0143] Specifically, in one embodiment, whether the training of the neural radiance field network is completed is determined according to the first output image and the first image, comprising:

[0144] The weight corresponding to each pixel point in the first output image is obtained by the output of the k th layer in the neural radiance field network in the i th iteration process, k is a positive integer less than a preset threshold;

[0145] The target pixel value of at least one target pixel point in the first output image is obtained according to the first output image and the weight corresponding to each pixel point in the first output image;

[0146] Whether the training of the neural radiance field network is completed is determined according to the target pixel value of at least one target pixel point in the first output image and the first image.

[0147] In an embodiment of the present application, considering that the dimension of the weight is usually low, it is empirically determined that the k-th layer is usually located at a shallow layer in the neural radiance field network. For example, the neural radiance field network includes a plurality of convolutional layers, and the k-th layer can be an output of the plurality of convolutional layers.

[0148] The embodiments of the present application are exemplarily described below by an example.

[0149] As shown in Figure 7 , the neural radiance field network includes a plurality of convolutional layers, and the weight corresponding to each pixel point in the first output image can be obtained according to an output of a first convolutional layer in the plurality of convolutional layers.

[0150] Then, for a sub-region (for example, an image region with a dimension of 3*3 in length and width as shown in Figure 7 ), the pixel values and the weights of the pixel points in the sub-region can be weighted and summed to obtain a target pixel value of a center pixel point of the sub-region, so as to aggregate the features of the sub-region on the corresponding center pixel point, and the center pixel point is taken as a target pixel point. The target pixel value of the target pixel point is used to calculate the loss value, without the need to calculate the loss value according to the color information of the entire sub-region.

[0151] At this time, the image similarity between the first output image and the first image can be determined based on the target pixel value of each target pixel point in the first output image and the pixel value of the corresponding pixel point of each target pixel point in the first image.

[0152] In an embodiment of the present application, the loss function used to evaluate the difference between the first output image and the first image can be set in multiple ways.

[0153] In an example, the error between the pixel information in the first output image and the pixel information of the corresponding first image can be evaluated based on the peak signal-to-noise ratio (PSNR).

[0154] However, in another example, considering that the PSNR is used alone as the loss function of the neural radiance field network, the similarity between images in the local and global aspects is ignored, which may make the evaluation result of the loss function more one-sided and less accurate.

[0155] Therefore, the loss function of the neural radiance field can be constructed according to multiple performance indicators.

[0156] For example, in addition to the PSNR, the difference between the first output image and the first image can also be evaluated by one or more of the performance indicators such as structural similarity (SSIM), perceptual similarity (LPIPS), etc.

[0157] The structural similarity measures the similarity of local regions of images. In general, in the implementation process, the entire image is first divided into local blocks using a sliding window, and the similarity of the true value (e.g., the pixel value in the first image) and the predicted value (e.g., the pixel value in the first output image) is compared in the image block unit. Finally, the average value of the similarity of all small blocks is taken as the structural similarity of the entire image. The structural similarity can measure the similarity between images from three aspects of brightness, contrast and structure. For specific calculation formula of structural similarity, please refer to related technologies, which will not be described here.

[0158] The perceptual similarity is used to measure the image similarity by deep features. For example, given the predicted value (e.g., the first output image in this example) and the true value (e.g., the first image in this example) of the image, input a feature extraction network (such as AlexNet, Vgg16 network, etc.), compare the similarity of the predicted value and the true value on different feature layers of the feature extraction network, and then average in the feature space and sum in the channel to obtain the perceptual similarity of the predicted value and the true value.

[0159] However, the inventors of the present application found that the above-mentioned performance indicators (peak signal-to-noise ratio, structural similarity and perceptual similarity) are heterogeneous in the hypothesis space, and do not converge in the same direction during network optimization. Therefore, in the embodiments of the present application, a method for calculating the similarity between the true value and the predicted value of an image is proposed, so that multiple performance indicators can be effectively combined to calculate the corresponding loss value during the training process of the neural radiation field network, to accurately evaluate the similarity between the true value and the predicted value of the image during the training process, and to accurately guide the training of the corresponding neural radiation field network.

[0160] In some embodiments, the step 3013 comprises:

[0161] In the i-th iteration process, a plurality of feature mapping networks are obtained, each feature mapping network is related to the first feature information of the first output image, and any feature mapping network is used to map the input to a corresponding feature space, and different feature mapping networks correspond to different feature spaces.

[0162] In the i-th iteration process, a plurality of feature mapping networks are obtained, each feature mapping network is related to the first feature information of the first output image, and any feature mapping network is used to map the input to a corresponding feature space, and different feature mapping networks correspond to different feature spaces.

[0163] Based on the first and second feature information, determine whether the training of the neural radiation field network has been completed.

[0164] In this embodiment, the specific structure of any feature mapping network is not limited. For example, any feature mapping network may include a multilayer perceptron.

[0165] The specific function of each feature mapping network is to map the corresponding input to the corresponding feature space. The specific nature of the feature space corresponding to each feature mapping network is not limited here. In one example, each feature mapping network is a machine learning network, in which case the feature space corresponding to each feature mapping network is determined based on machine learning. In another example, the feature space corresponding to any feature mapping network can refer to the feature space corresponding to performance metrics such as peak signal-to-noise ratio, structural similarity, or perceptual similarity when evaluating image similarity.

[0166] In this way, through multiple feature mapping networks, multiple first feature information about the first output image and multiple second feature information about the first image can be obtained. Based on the multiple first feature information and multiple second feature information, the difference between the first output image and the first image can be evaluated, and based on the difference, it can be determined whether the training of the neural radiation field network is complete.

[0167] For example, such as Figure 8 As shown, multiple feature mapping networks may include feature mapping network A, feature mapping network B, and feature mapping network C.

[0168] At this point, first feature information a1 and second feature information a2 of the first output image can be obtained through feature mapping network A, and first feature information b1 and second feature information b2 of the first output image can be obtained through feature mapping network B, and first feature information c1 and second feature information c2 of the first output image can be obtained through feature mapping network C.

[0169] Then, based on the differences between the first feature information a1 and the second feature information a2, the differences between the first feature information b1 and the second feature information b2, and the differences between the first feature information c1 and the second feature information c2, the loss value of the i-th iteration process can be obtained, thereby evaluating whether training is complete.

[0170] In this example, the loss value can reflect the similarity between the first output image and the first image in multiple feature spaces. If the similarity between the first output image and the first image in multiple feature spaces is high, it usually indicates that the first output image and the first image have high similarity both locally and globally. This allows for a more accurate assessment of the training progress and determination of whether the network has converged.

[0171] In addition, in the embodiments of the present application, the complete first output image and the complete first image can be respectively input into the feature mapping network to obtain the first feature information and the second feature information; or, a part of the first output image and a corresponding part of the first image can be respectively input into the feature mapping network to obtain the first feature information and the second feature information. For example, an image composed of each target pixel point in the first output image is input into the feature mapping network, and an image composed of pixel points corresponding to each target pixel point in the first image is input into the feature mapping network.

[0172] It can be seen that, in the embodiments of the present application, in the training process, the predicted value and the true value of the image are mapped to different feature spaces by the plurality of feature mapping networks, which can evaluate the accuracy of the generated image in a plurality of feature spaces such as a high-dimensional feature space, so that the performance of the trained neural radiance field network is better.

[0173] The specific acquisition manner of each feature mapping network is not limited herein.

[0174] In an example, each feature mapping network can be deployed into the electronic device after being pre-trained, and used for training of the neural radiance field network.

[0175] In another example, the plurality of feature mapping networks can be alternately trained with the neural radiance field network, and after the alternately training ends, the trained neural radiance field network and the trained plurality of feature mapping networks are obtained.

[0176] In some embodiments, the step of training the neural radiance field network according to the plurality of images and the corresponding view directions of each image to obtain the trained neural radiance field network comprises:

[0177] alternately training the neural radiance field network and the plurality of initial feature mapping networks according to each image and the corresponding view direction of each image to obtain the plurality of trained initial feature mapping networks and the trained neural radiance field network, and taking the plurality of trained initial feature mapping networks as the plurality of feature mapping networks.

[0178] In the embodiments of the present application, the alternating training of the neural radiance field network and the plurality of initial feature mapping networks can be that the plurality of initial feature mapping networks are first fixed, and the neural radiance field network is trained for one or more iterations; then, the latest updated neural radiance field network is fixed, and the plurality of initial feature mapping networks are trained for one or more iterations; then, the latest updated plurality of initial feature mapping networks are fixed, and the latest updated neural radiance field network is trained for one or more iterations, and so on until a preset number of iterations is reached, or until the neural radiance field network and the initial feature mapping networks after iteration converge to respective expected states.

[0179] In some embodiments, the neural radiance field network and the plurality of initial feature mapping networks are alternately trained according to each image and the view direction corresponding to each image, to obtain a plurality of trained initial feature mapping networks and a trained neural radiance field network, and the plurality of trained initial feature mapping networks are taken as the plurality of feature mapping networks, comprising:

[0180] In the jth iteration process of training the plurality of initial feature mapping networks:

[0181] The second output image in the jth iteration process is obtained through the neural radiance field network in the jth iteration process, and j is a positive integer;

[0182] The third feature information of each initial feature mapping network about the second output image is obtained from the plurality of initial feature mapping networks corresponding to the jth iteration process;

[0183] Based on the difference between each third feature information, it is determined whether the training of the plurality of initial feature mapping networks is completed.

[0184] The iteration process of training the plurality of initial feature mapping networks and the iteration process of training the neural radiance field network appear alternately, so as to realize the alternating training of the neural radiance field network and the plurality of initial feature mapping networks.

[0185] The specific period of the alternating training of the neural radiance field network and the plurality of initial feature mapping networks is not limited here.

[0186] For example, the plurality of initial feature mapping networks and the neural radiance field network can be periodically alternately trained in a fixed period, that is, the number of iterations of the plurality of initial feature mapping networks and the neural radiance field network is fixed in each alternating period, and at this time, the number of iterations of the plurality of initial feature mapping networks and the number of iterations of the neural radiance field network can be the same or different. Alternatively, during the alternating training of the plurality of initial feature mapping networks and the neural radiance field network, if a certain network converges in advance, the number of iterations of the network can be reduced or stopped.

[0187] Of course, there may be other variations in the alternating training cycle, which are not limited here.

[0188] The training process of multiple initial feature mapping networks and neural radiation field networks is illustrated below using an alternating cycle as an example.

[0189] The training of multiple initial feature mapping networks and neural radiation field networks can be divided into two stages within an alternating cycle.

[0190] 1. First stage: Train the neural radiation field network while fixing multiple initial feature mapping networks.

[0191] like Figure 9a As shown, in the first stage, multiple initial feature mapping networks (initial feature mapping network A, initial feature mapping network B, and initial feature mapping network C in the figure) can be fixed, and the neural radiation field network can be trained iteratively once or multiple times. The iterative process of training the neural radiation field network can be referred to the description of any of the related embodiments above. For example, as... Figure 9a In the example shown, during one iteration of training the neural radiation field network, feature mapping network A obtains first feature information a1 and second feature information a2 of the first output image. Furthermore, feature mapping network B obtains first feature information b1 and second feature information b2 of the first output image, and feature mapping network C obtains first feature information c1 and second feature information c2 of the first output image. A first loss function is set to make the output image and the corresponding ground truth image as close as possible in their respective feature spaces. For example, the first feature information a1 and second feature information a2 are made as close as possible, the first feature information b1 and second feature information b2 are made as close as possible, and the first feature information c1 and second feature information c2 are made as close as possible. Since there are multiple initial feature mapping networks, this first loss function can make the output image and the corresponding ground truth image as close as possible in multiple feature spaces.

[0192] 2. Second stage: Fix the neural radiation field network and train multiple initial feature mapping networks.

[0193] like Figure 9b As shown, in the second stage, the second output image of the j-th iteration process is obtained through the neural radiation field network in the j-th iteration process; then, the third feature information (e.g., ...) of each initial feature mapping network corresponding to the j-th iteration process with respect to the second output image is obtained. Figure 9bthe third feature information a3, the third feature information b3, and the third feature information c3), so as to update the plurality of initial feature mapping networks based on the difference between each third feature information and the second loss function.

[0194] The second loss function is used to make each third feature information as diverse as possible, that is, to make the difference between the third feature information a3, the third feature information b3, and the third feature information c3tend to increase.

[0195] In this way, the feature spaces corresponding to the plurality of feature mapping networks obtained by training the plurality of initial feature mapping networks are obviously different, thereby maintaining the diversity of the feature spaces corresponding to the plurality of feature mapping networks.

[0196] In this way, the plurality of trained initial feature mapping networks can accurately evaluate the image similarity between the output image of the neural radiance field network and the corresponding ground truth image at the level of the plurality of obviously different feature spaces, and make the performance of the finally obtained trained neural radiance field network better and the generated image quality better.

[0197] As shown in FIG. 13, it is an exemplary iterative process schematic diagram of an embodiment of the present application. Figure 10

[0198] As shown in FIG. 13, it is an exemplary iterative process schematic diagram of an embodiment of the present application. Figure 10 In the example shown in FIG. 13, the neural radiance field network can fuse the context information between the plurality of light rays corresponding to the image blocks in the image and the context information between the neighborhood sample points on the same light ray, and can evaluate the difference between the output image obtained based on the neural radiance field network and the corresponding ground truth image through the plurality of feature mapping networks to guide the training process of the neural radiance field network according to the difference, thereby improving the performance of the trained neural radiance field.

[0199] After the training of the neural radiance field network is completed, a corresponding inference task can be performed based on the trained neural radiance field network.

[0200] The related operations of the inference phase of the neural radiance field network will be introduced below.

[0201] In the embodiments of the present application, the related operations of the inference phase of the neural radiance field network can be performed by an electronic device. The electronic device performing the related operations of the training phase of the neural radiance field network in any of the above embodiments and the electronic device performing the related operations of the inference phase can be the same or different.

[0202] ​The specific type of the electronic device performing the related operations of the inference stage is not limited here. Illustratively, the electronic device can be a single server, a server cluster, a terminal device, etc., or can also be a virtual machine (VM) or a container.

[0203] The neural radiance field network deployed by the electronic device in the inference stage can be the neural radiance field network obtained after training through the training stage described above. For ease of description, the trained neural radiance field network deployed in the inference stage is referred to as a neural radiance field network.

[0204] As shown in Figure 11 In the embodiments of the present application, the image generation method related to the neural radiance field network executed by the electronic device can include steps 1101-1102.

[0205] In step 1101, the neural radiance field network is used to obtain the volume density and color information corresponding to each sample point according to the context information corresponding to each sample point in the plurality of sample points corresponding to the target view angle.

[0206] The plurality of sample points are obtained by sampling on at least one light ray corresponding to the target view angle.

[0207] In step 1102, the output image corresponding to the target view angle is obtained by volume rendering according to the volume density and color information corresponding to each sample point.

[0208] In the embodiments of the present application, the target view angle can be a view angle for which an output image needs to be generated in the inference process. At this time, the spatial position (x, y, z) and the view direction Then the plurality of light rays corresponding to the target view angle can be determined by the neural radiance field network, and the plurality of sample points corresponding to the target view angle can be obtained by sampling on the plurality of light rays corresponding to the target view angle. The specific way of sampling on the plurality of light rays corresponding to the target view angle to obtain the plurality of sample points corresponding to the target view angle in the inference stage can refer to the sampling way of the sample points in the neural radiance field network in the training stage described above. For example, the weight of the uniformly sampled sample points can be obtained by the coarse network in the neural radiance field network, and then the plurality of sample points can be obtained by non-uniform sampling based on the weight to obtain the input of the fine network in the neural radiance field network.

[0209] In the embodiments of the present application, the volume density and color information corresponding to each sample point are obtained according to the context information corresponding to each sample point in the plurality of sample points corresponding to the target view angle through the neural radiance field network, and then the output image rendered by the scene under the target view angle and the depth information of the output image are inferred through the volume rendering technology, so as to display the output image based on the depth information of the output image, thereby achieving the purpose of generating an image in a new view angle in the scene, and the three-dimensional space of the entire scene can be explicitly restored through the output image.

[0210] In some embodiments, the context information corresponding to each sample point is obtained based on the neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the target view angle.

[0211] In some embodiments, each sample point corresponds to a plurality of light rays, the context information corresponding to the corresponding sample point is obtained based on the neighborhood sample points on the neighborhood light rays of the corresponding sample point, the neighborhood light rays are included in the plurality of light rays, and the neighborhood light rays are different from the light ray on which the corresponding sample point is located.

[0212] In some embodiments, the context information corresponding to each sample point is obtained based on the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located.

[0213] In some embodiments, step 1101 comprises: fusing the information of the corresponding sample point and the information of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located through the neural radiance field network according to the weights of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located, to obtain the volume density and color information corresponding to the corresponding sample point, wherein the weight of each neighborhood sample point on the light ray on which the corresponding sample point is located is determined based on the distance between the corresponding neighborhood sample point and the corresponding sample point.

[0214] In some embodiments, the neural radiance field network comprises at least one convolution layer, and the at least one convolution layer is used to fuse the information of each sample point and the context information corresponding to the corresponding sample point through a convolution operation.

[0215] In some embodiments, the at least one convolution layer comprises a plurality of convolution layers, the plurality of convolution layers are in a serial structure, and the size of the output of any convolution layer in the plurality of convolution layers in at least one dimension is not greater than the size of the output of the corresponding next convolution layer in at least one dimension.

[0216] The features and steps in each embodiment of the reasoning stage can refer to the related content in any of the above embodiments of the training stage, for example, the specific content and fusion manner of the context information corresponding to the sample points in the reasoning stage, and the structure of the neural radiance field network, and the like, which can be similar to the related schemes in any of the above embodiments of the training stage, and will not be described here in the embodiments of the present application.

[0217] The above introduces various method embodiments from multiple aspects, and the neural radiance field network training apparatus and the image generation apparatus in the embodiments of the present application are introduced below in combination with the drawings.

[0218] As shown in Figure 12 The present application embodiment provides a neural radiance field network training apparatus 120.

[0219] The apparatus 120 comprises:

[0220] The training module 1201 is configured to train the neural radiance field network according to a plurality of images and a view direction corresponding to each image, and obtain a trained neural radiance field network.

[0221] In the i th iteration process of training the neural radiance field network, the training module is configured to:

[0222] obtain, by the neural radiance field network in the i th iteration process, volume density and color information corresponding to each sample point in the i th iteration process according to context information corresponding to each sample point in the plurality of sample points corresponding to the first view, the first view being determined based on a view direction of a first image, the plurality of sample points being obtained by sampling on at least one light ray corresponding to the first view, the first image being included in the plurality of images, and i being a positive integer;

[0223] obtain, by volume rendering, a first output image corresponding to the first view in the i th iteration process according to the volume density and color information corresponding to each sample point in the i th iteration process;

[0224] determine whether the training of the neural radiance field network is completed according to the first output image and the first image.

[0225] Optionally, the context information corresponding to each sample point is obtained based on neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the first view.

[0226] Optionally, the first image comprises a plurality of image blocks, and a plurality of light rays corresponding to each sample point, the plurality of light rays being obtained by emitting light rays from a camera corresponding to the first image to a plurality of pixel points in an image block corresponding to the corresponding sample point;

[0227] The context information corresponding to the sample point is obtained based on a neighborhood sample point on a neighborhood ray corresponding to the sample point, the neighborhood ray is included in the plurality of rays, and the neighborhood ray is different from the ray on which the sample point is located.

[0228] Optionally, the context information corresponding to each sample point is obtained based on a corresponding neighborhood sample point on a ray on which the sample point is located.

[0229] Optionally, the training module 1201 is configured to:

[0230] The information of the sample point and the information of the neighborhood sample point on the ray on which the sample point is located are fused according to the weight of the neighborhood sample point on the ray on which the sample point is located, by the neural radiance field network in the i-th iteration process, to obtain the volume density and color information corresponding to the sample point in the i-th iteration process, wherein the weight of each neighborhood sample point on the ray on which the sample point is located is determined based on the distance between the corresponding neighborhood sample point and the sample point.

[0231] Optionally, the neural radiance field network comprises at least one convolutional layer, and the at least one convolutional layer is configured to fuse the information of each sample point and the context information corresponding to the sample point by a convolution operation.

[0232] Optionally, the at least one convolutional layer comprises a plurality of convolutional layers, the plurality of convolutional layers are in a serial structure, and the size of the output of each convolutional layer in the plurality of convolutional layers in at least one dimension is not greater than the size of the output of the corresponding next convolutional layer in at least one dimension.

[0233] Optionally, the training module 1201 is configured to:

[0234] The weight corresponding to each pixel point in the first output image is obtained by the output of the k-th layer in the neural radiance field network in the i-th iteration process, k is a positive integer less than a preset threshold;

[0235] The target pixel value of at least one target pixel point in the first output image is obtained according to the first output image and the weight corresponding to each pixel point in the first output image;

[0236] Whether the training of the neural radiance field network is completed is determined according to the target pixel value of the at least one target pixel point in the first output image and the first image.

[0237] Optionally, the training module 1201 is configured to:

[0238] In the plurality of feature mapping networks corresponding to the i-th iteration process, each feature mapping network is related to the first feature information of the first output image, and any feature mapping network is used to map an input to a corresponding feature space, and different feature mapping networks correspond to different feature spaces.

[0239] In the plurality of feature mapping networks corresponding to the i-th iteration process, each feature mapping network is related to the second feature information of the first image.

[0240] According to the first feature information and the second feature information, it is determined whether the training of the neural radiance field network is completed.

[0241] Optionally, the training module 1201 is configured to:

[0242] According to each image and the corresponding view direction of each image, the neural radiance field network and the plurality of initial feature mapping networks are alternately trained to obtain a plurality of trained initial feature mapping networks and a trained neural radiance field network, and the plurality of trained initial feature mapping networks are used as the plurality of feature mapping networks.

[0243] Optionally, the training module 1201 is configured to:

[0244] In the j-th iteration process of training the plurality of initial feature mapping networks:

[0245] In the j-th iteration process, the second output image is obtained through the neural radiance field network in the j-th iteration process, and j is a positive integer.

[0246] In the plurality of initial feature mapping networks corresponding to the j-th iteration process, each initial feature mapping network is related to third feature information of the second output image.

[0247] Based on the difference between each third feature information, it is determined whether the training of the plurality of initial feature mapping networks is completed.

[0248] As shown in Figure 13 , an embodiment of the present application provides an image generation apparatus 130.

[0249] The apparatus 130 comprises:

[0250] The neural radiance field network module 1301 is configured to obtain, through a neural radiance field network, volume density and color information corresponding to each sample point according to context information corresponding to each sample point in a plurality of sample points corresponding to a target view angle, and the plurality of sample points are obtained by sampling at least one light ray corresponding to the target view angle.

[0251] The volume rendering module 1302 is configured to obtain an output image corresponding to the target view angle through volume rendering according to the volume density and color information corresponding to each sample point.

[0252] Optionally, the context information corresponding to each sample point is obtained based on the neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the target view angle.

[0253] Optionally, each sample point corresponds to a plurality of light rays, and the context information corresponding to the corresponding sample point is obtained based on the neighborhood sample points of the corresponding sample point on the neighborhood light rays, the neighborhood light rays are included in the plurality of light rays, and the neighborhood light rays are different from the light ray on which the corresponding sample point is located.

[0254] Optionally, the context information corresponding to each sample point is obtained based on the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located.

[0255] Optionally, the neural radiance field network module 1301 is configured to:

[0256] By means of the neural radiance field network, the information of the corresponding sample point and the information of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located are fused according to the weights of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located, to obtain the volume density and color information corresponding to the corresponding sample point, wherein the weight of each neighborhood sample point on the light ray on which the corresponding sample point is located is determined based on the distance between the corresponding neighborhood sample point and the corresponding sample point.

[0257] Optionally, the neural radiance field network comprises at least one convolutional layer, and the at least one convolutional layer is configured to fuse the information of each sample point and the context information corresponding to the corresponding sample point by means of a convolutional operation.

[0258] Optionally, the at least one convolutional layer comprises a plurality of convolutional layers, the plurality of convolutional layers are in a serial structure, and the size of the output of any convolutional layer in the plurality of convolutional layers in at least one dimension is not greater than the size of the output of the corresponding next convolutional layer in at least one dimension.

[0259] Figure 14 Fig. 1 shows a possible logical structure schematic diagram of an electronic device 140 provided by an embodiment of the present application. The electronic device 140 is configured to implement the functions of the electronic device of the neural radiance field network training method embodiment and / or the image generation method embodiment involved in any of the above embodiments. The electronic device 140 comprises a memory 1401, a processor 1402, a communication interface 1403, and a bus 1404. The memory 1401, the processor 1402, and the communication interface 1403 are communicatively connected to each other through the bus 1404.

[0260] The memory 1401 can be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1401 can store programs, and when the programs stored in the memory 1401 are executed by the processor 1402, the processor 1402 and the communication interface 1403 are configured to perform one or more steps of the above-mentioned neural radiance field network training method embodiments and / or image generation method embodiments.

[0261] The processor 1402 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processing (DSP), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any combination thereof, configured to execute a related program to implement the functions required to be performed by the training module in the neural radiance field network training apparatus in the above embodiments, or to implement the functions required to be performed by the neural radiance field network module and the volume rendering module in the image generation apparatus in the above embodiments, or to perform one or more steps of each method embodiment of the present application. The steps of the method disclosed in the embodiments of the present application can be performed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory, an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the memory 1401, and the processor 1402 reads the information in the memory 1401 and combines the hardware to perform one or more steps of the above-mentioned neural radiance field network training method embodiments and / or image generation method embodiments.

[0262] The communication interface 1403 uses a transceiver such as, but not limited to, a transceiver to realize the communication between the electronic device 140 and other devices or communication networks.

[0263] Bus 1404 enables the transmission of information between various components of electronic device 140 (e.g., memory 1401, processor 1402, and communication interface 1403). Bus 1404 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0264] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figure 14 The steps performed by the processor in the process.

[0265] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium; when the processor of the device executes the computer-executable instructions, the device performs the above-described... Figure 14 The steps performed by the processor in the process.

[0266] In another embodiment of this application, a chip system is also provided, the chip system including a processor for implementing the above. Figure 14 The steps performed by the processor. In one possible design, the chip system may also include memory for storing program instructions and data necessary for the electronic device. The chip system may consist of chips or may include chips and other discrete components.

[0267] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0268] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0269] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed system, device and method can be implemented by other manners. For example, the device embodiments described above are merely illustrative, for example, the division of units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0270] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0271] In addition, each functional unit in each embodiment of the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0272] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and various program code storage media.

[0273] The above is only a specific implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this.

Claims

1. A neural radiated field network training method, characterized in that, The method comprises the following steps: training a neural radiance field network according to a plurality of images and a corresponding view direction of each of the images, to obtain a trained neural radiance field network; wherein in the i-th iteration process of training the neural radiance field network: obtaining, by the neural radiance field network in the i-th iteration process, volume density and color information corresponding to each of a plurality of sample points in a first view direction according to context information corresponding to each of the sample points, the first view direction being determined based on a view direction of a first image, the plurality of sample points being obtained by sampling on at least one light ray corresponding to the first view direction, the first image being included in the plurality of images, i being a positive integer, the context information corresponding to each of the sample points being obtained based on neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the first view direction, the first image comprising a plurality of image blocks, a plurality of light rays corresponding to each of the sample points, the plurality of light rays being obtained by the camera corresponding to the first image emitting light rays to a plurality of pixel points in the image block corresponding to the corresponding sample point, the context information corresponding to the corresponding sample point being obtained based on neighborhood sample points of the corresponding sample point on the neighborhood light ray, the neighborhood light ray being included in the plurality of light rays, and the neighborhood light ray being different from the light ray on which the corresponding sample point is located; obtaining, by volume rendering, a first output image corresponding to the first view direction in the i-th iteration process according to the volume density and color information corresponding to each of the sample points in the i-th iteration process; determining whether the training of the neural radiance field network is completed according to the first output image and the first image.

2. The method of claim 1, wherein, The context information corresponding to each of the sample points is obtained based on the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located.

3. The method of claim 2, wherein, The obtaining, by the neural radiance field network in the i-th iteration process, volume density and color information corresponding to each of the plurality of sample points in the first view direction according to the context information corresponding to each of the sample points comprises: fusing, by the neural radiance field network in the i-th iteration process, information of the corresponding sample point and information of the corresponding neighborhood sample point on the light ray on which the corresponding sample point is located according to a weight of the corresponding neighborhood sample point on the light ray on which the corresponding sample point is located, to obtain the volume density and color information corresponding to the corresponding sample point in the i-th iteration process, wherein the weight of each of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located is determined based on a distance between the corresponding neighborhood sample point and the corresponding sample point.

4. The method of claim 1, wherein, The neural radiance field network comprises at least one convolution layer, and the at least one convolution layer is configured to fuse information of each of the sample points and the context information corresponding to the corresponding sample point by a convolution operation.

5. The method of claim 4, wherein, The at least one convolutional layer comprises a plurality of convolutional layers, the plurality of convolutional layers are in a serial structure, and a size of an output of any convolutional layer in the plurality of convolutional layers in at least one dimension is not greater than a size of an output of a corresponding next convolutional layer in the at least one dimension.

6. The method of claim 1, wherein, The determining whether the training of the neural radiance field network is completed according to the first output image and the first image comprises: obtaining, through an output of a kth layer in the neural radiance field network in the ith iteration process, a weight corresponding to each pixel point in the first output image, k being a positive integer less than a preset threshold; obtaining a target pixel value of at least one target pixel point in the first output image according to the first output image and the weight corresponding to each pixel point in the first output image; determining whether the training of the neural radiance field network is completed according to the target pixel value of the at least one target pixel point in the first output image and the first image.

7. The method according to any one of claims 1 to 6, characterized in that, The determining whether the training of the neural radiance field network is completed according to the first output image and the first image comprises: obtaining, in the plurality of feature mapping networks corresponding to the ith iteration process, first feature information of each feature mapping network with respect to the first output image, any feature mapping network being configured to map an input to a corresponding feature space, and different feature mapping networks corresponding to different feature spaces; obtaining, in the plurality of feature mapping networks corresponding to the ith iteration process, second feature information of each feature mapping network with respect to the first image; determining whether the training of the neural radiance field network is completed according to the first feature information and the second feature information.

8. The method of claim 7, wherein, The training of the neural radiance field network according to the plurality of images and the viewing angle direction corresponding to each image comprises: alternately training the neural radiance field network and a plurality of initial feature mapping networks according to each image and the viewing angle direction corresponding to each image, obtaining a plurality of trained initial feature mapping networks and a trained neural radiance field network, and taking the plurality of trained initial feature mapping networks as the plurality of feature mapping networks.

9. The method of claim 8, wherein, The alternately training the neural radiance field network and the plurality of initial feature mapping networks according to each image and the viewing angle direction corresponding to each image, obtaining the plurality of trained initial feature mapping networks and the trained neural radiance field network, and taking the plurality of trained initial feature mapping networks as the plurality of feature mapping networks comprises: in a jth iteration process of training the plurality of initial feature mapping networks: obtaining, through the neural radiance field network in the jth iteration process, a second output image in the jth iteration process, j being a positive integer; obtaining, in the plurality of initial feature mapping networks corresponding to the jth iteration process, third feature information of each initial feature mapping network with respect to the second output image; determining whether the training of the plurality of initial feature mapping networks is completed based on a difference between each third feature information.

10. An image generation method characterized by, comprises: The volume density and color information corresponding to each sample point are obtained by a neural radiance field network according to context information corresponding to each sample point in a plurality of sample points corresponding to a target view angle, the plurality of sample points being obtained by sampling on at least one light ray corresponding to the target view angle, the context information corresponding to each sample point being obtained based on neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the target view angle, the plurality of light rays corresponding to any sample point, the context information corresponding to the corresponding sample point being obtained based on neighborhood sample points of the corresponding sample point on a neighborhood light ray, the neighborhood light ray being included in the plurality of light rays and being different from the light ray on which the corresponding sample point is located. The output image corresponding to the target view angle is obtained by volume rendering according to the volume density and color information corresponding to each sample point.

11. The method of claim 10, wherein, The context information corresponding to each sample point is obtained based on the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located.

12. The method of claim 11, wherein, The obtaining of the volume density and color information corresponding to each sample point by the neural radiance field network according to the context information corresponding to each sample point in the plurality of sample points corresponding to the target view angle comprises: The information of the corresponding sample point and the information of the neighborhood sample points on the light ray on which the corresponding sample point is located are fused according to the weights of the neighborhood sample points on the light ray on which the corresponding sample point is located by the neural radiance field network to obtain the volume density and color information corresponding to the corresponding sample point, wherein the weight of each neighborhood sample point on the light ray on which the corresponding sample point is located is determined based on the distance between the corresponding neighborhood sample point and the corresponding sample point.

13. The method according to any one of claims 10-12, characterized in that, The neural radiance field network comprises at least one convolution layer, and the at least one convolution layer is configured to fuse the information of each sample point and the context information corresponding to the corresponding sample point by a convolution operation.

14. The method of claim 13, wherein, The at least one convolution layer comprises a plurality of convolution layers, the plurality of convolution layers are in a serial structure, and the size of the output of any convolution layer in the plurality of convolution layers in at least one dimension is not greater than the size of the output of the corresponding next convolution layer in the at least one dimension.

15. A neural radiation field network training apparatus, comprising: The training module is configured to train the neural radiance field network according to a plurality of images and a view direction corresponding to each image to obtain the trained neural radiance field network. In the i th iteration process of training the neural radiance field network, the training module is configured to: ​ obtaining, by the neural radiance field network in the i-th iteration process, volume density and color information corresponding to each of the sample points in the i-th iteration process according to context information corresponding to each of the sample points in the first view, the first view being determined based on a view direction of a first image, the sample points being obtained by sampling at least one light ray corresponding to the first view, the first image being included in the plurality of images, i being a positive integer, the context information corresponding to each of the sample points being obtained based on neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the first view, the first image including a plurality of image blocks, the corresponding sample point corresponding to a plurality of light rays, the plurality of light rays being obtained by the camera corresponding to the first image emitting light rays to a plurality of pixel points in the image block corresponding to the corresponding sample point, the context information corresponding to the corresponding sample point being obtained based on neighborhood sample points of the corresponding sample point on the neighborhood light ray, the neighborhood light ray being included in the plurality of light rays, and the neighborhood light ray being different from the light ray on which the corresponding sample point is located; obtaining, by volume rendering, a first output image corresponding to the first view in the i-th iteration process according to the volume density and color information corresponding to each of the sample points in the i-th iteration process; determining whether the training of the neural radiance field network is completed according to the first output image and the first image.

16. The apparatus of claim 15, wherein, The context information corresponding to each of the sample points is obtained based on the corresponding neighborhood sample point on the light ray on which the corresponding sample point is located.

17. The apparatus of claim 16, wherein the training module is configured to: fuse, by the neural radiance field network in the i-th iteration process, information of the corresponding sample point with information of the corresponding neighborhood sample point on the light ray on which the corresponding sample point is located according to a weight of the corresponding neighborhood sample point on the light ray on which the corresponding sample point is located to obtain the volume density and color information corresponding to the corresponding sample point in the i-th iteration process, wherein the weight of each of the corresponding neighborhood sample points on the light ray on which the corresponding sample point is located is determined based on a distance between the corresponding neighborhood sample point and the corresponding sample point.

18. The apparatus of claim 15, wherein, The neural radiance field network includes at least one convolutional layer, and the at least one convolutional layer is configured to fuse, by a convolution operation, the information of each of the sample points with the context information corresponding to the corresponding sample point.

19. The apparatus of claim 18, wherein, The at least one convolutional layer includes a plurality of convolutional layers, the plurality of convolutional layers are in a serial structure, and a size of an output of each of the convolutional layers in at least one dimension is not greater than a size of an output of a corresponding next convolutional layer in the at least one dimension.

20. The apparatus of claim 15, wherein the training module is configured to: obtain a weight corresponding to each of the pixel points in the first output image according to an output of a k-th layer in the neural radiance field network in the i-th iteration process, k being a positive integer less than a preset threshold. obtaining a target pixel value of at least one target pixel in the first output image according to the first output image and the weight corresponding to each pixel in the first output image; determining whether the training of the neural radiance field network is completed according to the target pixel value of at least one target pixel in the first output image and the first image.

21. The apparatus of any one of claims 15-20, wherein, the training module is configured to: obtain, in each of the plurality of feature mapping networks corresponding to the i th iteration process, first feature information about the first output image, any of the feature mapping networks being configured to map an input to a corresponding feature space, different feature mapping networks corresponding to different feature spaces; obtain, in each of the plurality of feature mapping networks corresponding to the i th iteration process, second feature information about the first image; determine whether the training of the neural radiance field network is completed according to the first feature information and the second feature information.

22. The apparatus of claim 21, wherein, the training module is configured to: perform alternating training on the neural radiance field network and a plurality of initial feature mapping networks according to each image and a corresponding view direction of each of the images, to obtain a plurality of trained initial feature mapping networks and a trained neural radiance field network, and use the plurality of trained initial feature mapping networks as the plurality of feature mapping networks.

23. The apparatus of claim 22, wherein, the training module is configured to: in a j th iteration process of training the plurality of initial feature mapping networks: obtain, by the neural radiance field network in the j th iteration process, a second output image in the j th iteration process, j being a positive integer; obtain, in each of the plurality of initial feature mapping networks corresponding to the j th iteration process, third feature information about the second output image; determine whether the training of the plurality of initial feature mapping networks is completed based on a difference between each of the third feature information.

24. An image generation apparatus characterized by comprising: comprising: a neural radiance field network module configured to obtain, by a neural radiance field network, volume density and color information corresponding to each of a plurality of sample points corresponding to a target view direction according to context information corresponding to each of the sample points, the plurality of sample points being obtained by sampling on at least one light ray corresponding to the target view direction, the context information corresponding to each sample point being obtained based on neighborhood sample points of the corresponding sample point in the plurality of sample points corresponding to the target view direction, a plurality of light rays corresponding to any sample point, the context information corresponding to the corresponding sample point being obtained based on neighborhood sample points of the corresponding sample point on a neighborhood light ray, the neighborhood light ray being included in the plurality of light rays and being different from the light ray on which the corresponding sample point is located; a volume rendering module configured to obtain, by volume rendering, an output image corresponding to the target view direction according to the volume density and the color information corresponding to each of the sample points.

25. The apparatus of claim 24, wherein, The context information corresponding to each sample point is obtained based on the corresponding neighboring sample points on the light ray on which the sample point is located.

26. The apparatus of claim 25, wherein, The neural radiance field network module is configured to: The neural radiance field network is configured to: obtain the volume density and color information of each sample point by fusing the information of the sample point and the information of the corresponding neighboring sample points on the light ray on which the sample point is located according to the weights of the corresponding neighboring sample points, wherein the weight of each neighboring sample point on the light ray on which the sample point is located is determined based on the distance between the corresponding neighboring sample point and the sample point.

27. The apparatus of any of claims 24-26, wherein, The neural radiance field network comprises at least one convolution layer, and the at least one convolution layer is configured to fuse the information of each sample point and the context information corresponding to the sample point by a convolution operation.

28. The apparatus of claim 27, wherein, The at least one convolution layer comprises a plurality of convolution layers, the plurality of convolution layers are in a serial structure, and the size of the output of any convolution layer in the plurality of convolution layers in at least one dimension is not greater than the size of the output of the corresponding next convolution layer in the at least one dimension.

29. An electronic device, comprising: The electronic device comprises at least one processor, a memory, and instructions stored on the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method of any one of claims 1-14.

30. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-14.

Images