A processing method of an image reconstruction model and a related device

Abstract

The application discloses a processing method of an image reconstruction model and a related device. The method comprises the following steps: inputting a plurality of sampling points on each sampling ray corresponding to a sample image block of a target scene into an initial reconstruction model based on a neural radiation field to perform color prediction and density prediction, and outputting a predicted color and a predicted density corresponding to each sampling point on each sampling ray; inputting the predicted color and the predicted density corresponding to each sampling point on each sampling ray into the initial reconstruction model to perform volume rendering, and outputting a rendered image block corresponding to the sample image block; and training the initial reconstruction model to obtain an image reconstruction model through color loss and structural loss between the rendered image block and the sample image block, so as to ensure that each rendered pixel point in the rendered image block is consistent with a corresponding sample pixel point in the sample image block in color, and ensure that the rendered image block is consistent with the sample image block in structure, and improve the reconstruction performance of the image reconstruction model in image structure.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a processing method and related apparatus for an image reconstruction model. Background Technology

[0002] With the rapid development of computer vision technology, an initial reconstruction model based on neural radiation fields is trained using sample images of specific scenes to obtain an image reconstruction model. This allows the image reconstruction model to render a new 2D image of a specific scene from a new perspective, or to reconstruct a 3D model of a specific scene, based on 2D images from multiple perspectives in a specific scene.

[0003] In related technologies, the training method for image reconstruction models refers to: randomly selecting multiple sampling rays from sample images of a specific scene, outputting the predicted color and predicted density corresponding to each sampling point on each sampling ray through an initial reconstruction model based on neural radiation fields, and rendering a rendered image by volume. According to the color loss between the rendered image and the sample image, the initial reconstruction model is trained to obtain the image reconstruction model.

[0004] However, in the above method, multiple sampling rays are randomly selected based on sample images of a specific scene. These multiple sampling rays are relatively discrete. The neural radiation field is trained according to the color loss between the rendered image and the sample image, which fails to focus on learning the structural information of the sample image, resulting in poor reconstruction performance of the image reconstruction model in terms of image structure. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a method and related apparatus for processing image reconstruction models, thereby improving the reconstruction performance of image reconstruction models on image structures.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On one hand, embodiments of this application provide a method for processing an image reconstruction model, the method comprising:

[0008] Based on multiple sampling points on each sampling ray corresponding to the sample image block of the target scene, color prediction and density prediction are performed through an initial reconstruction model based on neural radiation field to obtain the predicted color and predicted density corresponding to each sampling point on each sampling ray.

[0009] Based on the predicted color and predicted density of each sampling point on each sampling ray, volume rendering is performed through the initial reconstruction model to obtain the rendered image block corresponding to the sample image block;

[0010] The initial reconstruction model is trained based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch, to obtain the image reconstruction model.

[0011] On the other hand, embodiments of this application provide a processing apparatus for an image reconstruction model, the apparatus comprising: a prediction unit, a rendering unit, and a training unit;

[0012] The prediction unit is used to perform color prediction and density prediction based on an initial reconstruction model based on neural radiation field, according to multiple sampling points on each sampling ray corresponding to the sample image block of the target scene, so as to obtain the predicted color and predicted density corresponding to each sampling point on each sampling ray.

[0013] The rendering unit is used to perform volume rendering based on the predicted color and predicted density corresponding to each sampling point on each sampling ray, through the initial reconstruction model, to obtain the rendered image block corresponding to the sample image block;

[0014] The training unit is used to train the initial reconstruction model to obtain an image reconstruction model based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch.

[0015] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0016] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0017] The processor is configured to execute the method described in any of the foregoing aspects according to instructions in the computer program.

[0018] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program that, when run on a computer device, causes the computer device to perform the methods described in any of the foregoing aspects.

[0019] On the other hand, embodiments of this application provide a computer program product, including a computer program that, when run on a computer device, causes the computer device to perform the method described in any of the foregoing aspects.

[0020] As can be seen from the above technical solution, firstly, multiple sampling points on each sampling ray corresponding to the sample image block of the target scene are input into an initial reconstruction model based on neural radiation fields for color and density prediction. The predicted color and predicted density corresponding to each sampling point on each sampling ray are output, representing the three-dimensional structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image block along each sample pixel in the sample image block, thus representing the three-dimensional structural information of a portion of the scene corresponding to the sample image block of the target scene. Then, the predicted color and predicted density corresponding to each sampling point on each sampling ray are input into the initial reconstruction model for volume rendering, outputting the rendered image block corresponding to the sample image block. This realizes the conversion of the three-dimensional structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image block along each pixel in the sample image block into the two-dimensional visual information of the rendered pixels in the rendered image block corresponding to each sample pixel in the sample image block, thereby mapping the three-dimensional structural information of a portion of the scene corresponding to the sample image block of the target scene into two-dimensional visual information. Finally, by using the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch, the initial reconstruction model is trained to obtain the image reconstruction model. This ensures that each rendered pixel in the rendered image patch is consistent with the corresponding sample pixel in the sample image patch in terms of color, and also ensures that the rendered image patch and the sample image patch are consistent in terms of structure. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch, but also the structural information of the sample image patch, thereby improving the reconstruction performance of the image reconstruction model in terms of image structure. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A system schematic diagram of an image reconstruction model processing method provided in an embodiment of this application;

[0023] Figure 2 A flowchart illustrating a processing method for an image reconstruction model provided in this application embodiment;

[0024] Figure 3 A schematic diagram showing the shooting points of a sample image block along the viewing direction of sample pixels in the sample image block, as provided in an embodiment of this application;

[0025] Figure 4A schematic diagram of a sample image and a rendered image provided for an embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating a real depth image corresponding to a sample image and a rendered depth image corresponding to a rendered image, provided in an embodiment of this application.

[0027] Figure 6 A schematic diagram of another sample image, the gradient image corresponding to the sample image, and the rendering depth image corresponding to the rendered image provided in an embodiment of this application;

[0028] Figure 7 A structural diagram of a processing apparatus for an image reconstruction model provided in an embodiment of this application;

[0029] Figure 8 A structural diagram of a server provided in an embodiment of this application;

[0030] Figure 9 This is a structural diagram of a terminal provided in an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application will now be described with reference to the accompanying drawings.

[0032] Currently, multiple sampling rays are randomly selected from sample images of a specific scene. An initial reconstruction model based on neural radiation fields outputs the predicted color and density corresponding to each sampling point on each sampling ray and renders the volumetric image. The initial reconstruction model is then trained according to the color loss between the rendered image and the sample image to obtain an image reconstruction model. This allows the image reconstruction model to render a new 2D image of a specific scene from multiple perspectives, or to reconstruct a 3D model of a specific scene, based on 2D images from multiple perspectives within that scene. However, research has found that randomly selecting multiple sampling rays from sample images of a specific scene results in a relatively discrete structure. Training the neural radiation field based on the color loss between the rendered image and the sample image fails to focus on learning the structural information of the sample images, leading to poor reconstruction performance of the image reconstruction model in terms of image structure.

[0033] This application provides a processing method for an image reconstruction model. Multiple sampling points on each sampling ray corresponding to a sample image patch of the target scene are input into an initial reconstruction model based on a neural radiation field for color and density prediction. The model outputs the predicted color and density corresponding to each sampling point on each sampling ray. The predicted color and density corresponding to each sampling point on each sampling ray are then input into the initial reconstruction model for volume rendering, outputting a rendered image patch corresponding to the sample image patch. The initial reconstruction model is trained using color loss between the rendered image patch and the sample image patch, as well as structural loss between the rendered image patch and the sample image patch, to obtain an image reconstruction model. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch but also the structural information of the sample image patch, thereby improving the image reconstruction performance in terms of image structure.

[0034] Next, the system architecture of the image reconstruction model processing method will be introduced. See [link / reference] Figure 1 ,Should Figure 1 This is a schematic diagram of a system for processing an image reconstruction model according to an embodiment of the present application. The system includes a computer device 100, which is used to execute the image reconstruction model processing method.

[0035] The computer device 100 uses an initial reconstruction model based on neural radiation fields to predict color and density based on multiple sampling points on each sampling ray corresponding to the sample image block of the target scene, thereby obtaining the predicted color and predicted density corresponding to each sampling point on each sampling ray.

[0036] The target scene is the scene to be learned, rendered, and reconstructed; the sample image patch of the target scene is a random image patch in the sample image of the target scene; each sampling ray corresponding to the sample image patch is a sampling ray emitted from the shooting point of the sample image patch along each sample pixel in the sample image patch; the initial reconstruction model based on the neural radiation field is a deep learning model that uses a neural network to represent the three-dimensional scene and render the two-dimensional image; the predicted color and predicted density corresponding to each sampling point are the color and density of each sampling point predicted by the initial reconstruction model based on the neural radiation field, along the viewing direction of each sample pixel in the sample image patch and the spatial position of each sampling point.

[0037] As an example, the sample image patch of the target scene can be image patch 1 in the point 1 image of indoor scene A. Then, the computer device 100 inputs multiple sampling points on each sampling ray corresponding to image patch 1 in the point 1 image of indoor scene A into the initial reconstruction model based on the neural radiation field to perform color prediction and density prediction, and outputs the predicted color and predicted density corresponding to each sampling point on each sampling ray.

[0038] Computer device 100 performs volume rendering based on the predicted color and predicted density corresponding to each sampling point on each sampling ray, and obtains the rendered image block corresponding to the sample image block through the initial reconstruction model.

[0039] The rendered image block corresponding to the sample image block is formed by the color of the rendered pixel corresponding to each sample pixel in the sample image block obtained by rendering based on the predicted color and predicted density of each sampling point on each sampling ray and the color of the rendered pixel corresponding to each sample pixel in the sample image block rendered by the initial reconstruction model.

[0040] As an example, based on the above example, the computer device 100 inputs the predicted color and predicted density corresponding to each sampling point on each sampling ray into the initial reconstruction model for volume rendering, and outputs the rendered image block corresponding to image block 1, for example, image block 1'.

[0041] Computer device 100 trains an initial reconstruction model to obtain an image reconstruction model based on the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch.

[0042] Among them, the color loss between the rendered image patch and the sample image patch is the difference in color between each rendered pixel in the rendered image patch and the corresponding sample pixel in the sample image patch; the structural loss between the rendered image patch and the sample image patch is the difference in structure between the rendered image patch and the sample image patch; the image reconstruction model is a deep learning model that uses a neural network to represent the target scene and render a two-dimensional image.

[0043] As an example, based on the above example, the color loss could be L color Structural loss can be L structure Then computer device 100 connects image block 1' with image block 1 via L... color and the L between image block 1' and image block 1 structure The initial reconstruction model is trained to obtain the image reconstruction model.

[0044] In other words, multiple sampling points on each sampling ray corresponding to a sample image patch of the target scene are input into an initial reconstruction model based on neural radiation fields for color and density prediction. The output is the predicted color and predicted density corresponding to each sampling point on each sampling ray, representing the 3D structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image patch along each sample pixel in the sample image patch, thus representing the 3D structural information of a portion of the scene corresponding to the sample image patch of the target scene. The predicted color and predicted density corresponding to each sampling point on each sampling ray are input into the initial reconstruction model for volume rendering, outputting the rendered image patch corresponding to the sample image patch. This converts the 3D structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image patch along each pixel in the sample image patch into the 2D visual information of the rendered pixels in the rendered image patch corresponding to each sample pixel in the sample image patch, thus mapping the 3D structural information of a portion of the scene corresponding to the sample image patch of the target scene into 2D visual information. By training the initial reconstruction model using the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch, an image reconstruction model is obtained. This ensures that each rendered pixel in the rendered image patch is consistent with the corresponding sample pixel in the sample image patch in terms of color, and also ensures that the rendered image patch and the sample image patch are consistent in terms of structure. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch, but also the structural information of the sample image patch, thereby improving the image reconstruction performance of the image reconstruction model in terms of image structure.

[0045] It should be noted that, in the embodiments of this application, the computer device can be a server or a terminal. The method provided in the embodiments of this application can be executed by the terminal or the server alone, or it can be executed by the terminal and the server in cooperation. Specifically, when the method provided in the embodiments of this application is executed by the terminal or the server alone, its execution method is similar to... Figure 1 The corresponding embodiments are similar, mainly replacing the computer device with a terminal or server. Furthermore, when the method provided in this application is executed by a terminal and a server, steps that need to be displayed on the front-end interface can be executed by the terminal, while steps that require background calculations and do not need to be displayed on the front-end interface can be executed by the server.

[0046] The terminal can be a smartphone, tablet, laptop, desktop computer, intelligent voice interaction device, vehicle terminal, extended reality device, or aircraft, but is not limited to these. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, but is not limited to these. The terminal and server can be connected directly or indirectly through wired or wireless communication, and this application does not impose any restrictions. For example, the terminal and server can be connected through a network, which can be wired or wireless.

[0047] Next, taking the method provided in the embodiments of this application as an example of a computer device executing the method, and in conjunction with the accompanying drawings, the processing method of the image reconstruction model provided in the embodiments of this application will be described in detail.

[0048] See Figure 2 ,Should Figure 2 The flowchart illustrates a processing method for an image reconstruction model provided in this application embodiment, the method including the following steps S201-S203.

[0049] S201: Based on multiple sampling points on each sampling ray corresponding to the sample image patch of the target scene, color prediction and density prediction are performed through an initial reconstruction model based on neural radiation field to obtain the predicted color and predicted density corresponding to each sampling point on each sampling ray.

[0050] In related technologies, the training method for image reconstruction models involves randomly selecting multiple sampling rays from sample images of the target scene. An initial reconstruction model based on a neural radiation field outputs the predicted color and density corresponding to each sampling point on each sampling ray and renders a volumetric image. The initial reconstruction model is then trained according to the color loss between the rendered image and the sample image to obtain the image reconstruction model. This allows the image reconstruction model to render a new 2D image of the target scene from multiple perspectives, or to reconstruct a 3D model of the target scene, based on 2D images from multiple viewpoints. However, research has found that randomly selecting multiple sampling rays from sample images of the target scene results in a relatively discrete model. Training the neural radiation field based on the color loss between the rendered image and the sample image fails to focus on learning the structural information of the sample images, leading to poor reconstruction performance of the image reconstruction model in terms of image structure.

[0051] In this embodiment, to address the aforementioned issues and enable the image reconstruction model trained using the initial reconstruction model based on the neural radiation field to learn the structural information of the sample images, it is first necessary to ensure the concentration of multiple sampling rays selected from the sample images of the target scene. This can be achieved by randomly selecting multiple sampling rays corresponding to sample image blocks from the sample images of the target scene, and then using the initial reconstruction model based on the neural radiation field to realize a three-dimensional representation. This outputs the predicted color and predicted density corresponding to each sampling point on each sampling ray. Therefore, multiple sampling points on each sampling ray corresponding to the sample image block of the target scene are input into the initial reconstruction model based on the neural radiation field for color and density prediction, outputting the predicted color and predicted density corresponding to each sampling point on each sampling ray.

[0052] The target scene is the scene to be learned, rendered, and reconstructed; the sample image patch of the target scene is a random image patch in the sample image of the target scene; each sampling ray corresponding to the sample image patch is a sampling ray emitted from the shooting point of the sample image patch along each sample pixel in the sample image patch; the initial reconstruction model based on the neural radiation field is a deep learning model that uses a neural network to represent the three-dimensional scene and render the two-dimensional image; the predicted color and predicted density corresponding to each sampling point are the color and density of each sampling point predicted by the initial reconstruction model based on the neural radiation field, based on the shooting point of the sample image patch along the viewing direction of each sample pixel in the sample image patch and the spatial position of each sampling point, representing the visual color and opacity of each sampling point under the viewing direction of each sample pixel in the sample image patch at the shooting point of the sample image patch.

[0053] See Figure 3 ,Should Figure 3 This is a schematic diagram illustrating the shooting point of a sample image block along the viewing direction of sample pixels in the sample image block, as provided in an embodiment of this application. The sample pixels in the sample image block are (i, j), and the shooting point of the sample image block along the viewing direction of sample pixels (i, j) is (θ, φ), where θ = πi / H, φ = 2πj / W, H is the height of the sample image, and W is the width of the sample image.

[0054] In practical applications, based on multiple sampling points on each sampling ray corresponding to the sample image patch of the target scene, color and density prediction are performed using an initial reconstruction model based on neural radiation fields to obtain the predicted color and density corresponding to each sampling point on each sampling ray. This can be achieved by: based on the shooting point of the sample image patch along the viewing direction of each sample pixel in the sample image patch, and the spatial position of each sampling point on each sampling ray emanating from the shooting point of the sample image patch along each sample pixel in the sample image patch, color and density prediction are performed using an initial reconstruction model based on neural radiation fields, that is, predicting the color and density of each sampling point on each sampling ray to obtain the predicted color and density corresponding to each sampling point on each sampling ray.

[0055] The above-mentioned S201, based on the initial reconstruction model based on the neural radiation field, is a deep learning model that uses a neural network to represent a three-dimensional scene and render a two-dimensional image. It achieves three-dimensional representation through the initial reconstruction model based on the neural radiation field, and can output the predicted color and predicted density of each sampling point on each sampling ray corresponding to the sample image block of the target scene. It represents the three-dimensional structural information of each sampling point on each sampling ray emitted from the shooting point of the sample image block along each sample pixel in the sample image block, so as to represent the three-dimensional structural information of the part of the scene corresponding to the sample image block of the target scene.

[0056] As an example of S201, the computer device inputs multiple sampling points on each sampling ray r corresponding to the sample image block p of the target scene into an initial reconstruction model based on the neural radiation field to perform color prediction and density prediction, and outputs the predicted color and predicted density corresponding to each sampling point on each sampling ray r.

[0057] S202: Based on the predicted color and predicted density of each sampling point on each sampling ray, volume rendering is performed through the initial reconstruction model to obtain the rendered image block corresponding to the sample image block.

[0058] In this embodiment, after executing S201 to achieve 3D representation through an initial reconstruction model based on neural radiation fields and outputting the predicted color and predicted density of each sampling point on each sampling ray corresponding to the sample image patch of the target scene, considering that the initial reconstruction model based on neural radiation fields is a deep learning model that represents the 3D scene and renders the 2D image through a neural network, volume rendering can be achieved through the initial reconstruction model based on neural radiation fields, and the rendered image patch corresponding to the sample image patch can be output. Therefore, the predicted color and predicted density corresponding to each sampling point on each sampling ray are input into the initial reconstruction model for volume rendering, and the rendered image patch corresponding to the sample image patch is output.

[0059] The rendered image block corresponding to the sample image block is formed by the color of the rendered pixel corresponding to each sample pixel in the sample image block obtained by rendering based on the predicted color and predicted density of each sampling point on each sampling ray and the color of the rendered pixel corresponding to each sample pixel in the sample image block rendered by the initial reconstruction model.

[0060] In practical applications, based on the predicted color and predicted density corresponding to each sampling point on each sampling ray, volume rendering is performed through an initial reconstruction model to obtain the rendered image block corresponding to the sample image block. This can be achieved by: performing volume rendering through an initial reconstruction model based on the predicted color and predicted density corresponding to each sampling point on each sampling ray, that is, accumulating the predicted color and predicted density corresponding to each sampling point on each sampling ray to obtain the color of the rendered pixel corresponding to each sample pixel in the sample image block, thus forming the rendered image block corresponding to the sample image block.

[0061] The above-mentioned S202, based on the initial reconstruction model based on the neural radiation field, is a deep learning model that uses neural networks to represent three-dimensional scenes and render two-dimensional images. It uses the predicted color and predicted density of each sampling point on each sampling ray corresponding to the sample image block of the target scene to represent the three-dimensional structural information of the part of the scene corresponding to the sample image block of the target scene. It achieves volume rendering through the initial reconstruction model based on the neural radiation field and can output the rendered image block corresponding to the sample image block. This realizes the conversion of the three-dimensional structural information of each sampling point on each sampling ray emitted from the shooting point of the sample image block along each pixel in the sample image block into the two-dimensional visual information of the rendered pixel in the rendered image block corresponding to each sample pixel in the sample image block. That is, it realizes the mapping of the three-dimensional structural information of the part of the scene corresponding to the sample image block of the target scene into two-dimensional visual information.

[0062] As an example of S202, based on the example of S201 above, the computer device inputs the predicted color and predicted density corresponding to each sampling point on each sampling ray r into the initial reconstruction model for volume rendering, and outputs the rendered image block corresponding to the sample image block p.

[0063] See Figure 4 ,Should Figure 4 This is a schematic diagram of a sample image and a rendered image provided for an embodiment of this application. Wherein, Figure 4 In the middle (a), the sample image is shown. Figure 4 In the middle (b), the image is rendered.

[0064] S203: Train the initial reconstruction model to obtain the image reconstruction model based on the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch.

[0065] In this embodiment, after executing S202 to predict the color and density of each sampling point on each sampling ray corresponding to the sample image block of the target scene, and to achieve volume rendering through an initial reconstruction model based on neural radiation fields, outputting the rendered image block corresponding to the sample image block, in order to train the initial reconstruction model based on neural radiation fields to obtain the image reconstruction model, so that the image reconstruction model not only learns the color information of each sample pixel in the sample image block, but also learns the structural information of the sample image block, it is necessary to ensure that each rendered pixel in the rendered image block is consistent with the sample pixel in the corresponding sample image block in color, and to ensure that the rendered image block is consistent with the sample image block in structure. Therefore, the initial reconstruction model is trained to obtain the image reconstruction model through the color loss between the rendered image block and the sample image block, and the structural loss between the rendered image block and the sample image block.

[0066] Among them, the color loss between the rendered image patch and the sample image patch is the difference in color between each rendered pixel in the rendered image patch and the corresponding sample pixel in the sample image patch; the structural loss between the rendered image patch and the sample image patch is the difference in structure between the rendered image patch and the sample image patch; the image reconstruction model is a deep learning model that uses a neural network to represent the target scene and render a two-dimensional image.

[0067] In practical applications, an initial reconstruction model is trained to obtain an image reconstruction model based on the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch. This can be done by iteratively training the model parameters of the initial reconstruction model based on the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch, until the initial reconstruction model converges or the number of iterations reaches a preset number, thus determining that the initial reconstruction model training is complete, and using the completed initial reconstruction model as the image reconstruction model.

[0068] The above-mentioned S203 trains the initial reconstruction model by using the color loss between the rendered image patch and the sample image patch to ensure that each rendered pixel in the rendered image patch is consistent with the sample pixel in the corresponding sample image patch in terms of color. It also trains the initial reconstruction model by using the structural loss between the rendered image patch and the sample image patch to ensure that the rendered image patch and the sample image patch are consistent in terms of structure. The trained initial reconstruction model is used as the image reconstruction model, so that the image reconstruction model not only learns the color information of each sample pixel in the sample image patch, but also learns the structural information of the sample image patch, thereby improving the reconstruction performance of the image reconstruction model in terms of image structure.

[0069] As an example of S203, based on the example of S202 above, the color loss is L color Structural loss is L structureComputer devices render image blocks L between sample image patch p color and rendering image blocks L between sample image patch p structure The initial reconstruction model is trained to obtain the image reconstruction model.

[0070] As can be seen from the above technical solution, firstly, multiple sampling points on each sampling ray corresponding to the sample image block of the target scene are input into an initial reconstruction model based on neural radiation fields for color and density prediction. The predicted color and predicted density corresponding to each sampling point on each sampling ray are output, representing the three-dimensional structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image block along each sample pixel in the sample image block, thus representing the three-dimensional structural information of a portion of the scene corresponding to the sample image block of the target scene. Then, the predicted color and predicted density corresponding to each sampling point on each sampling ray are input into the initial reconstruction model for volume rendering, outputting the rendered image block corresponding to the sample image block. This realizes the conversion of the three-dimensional structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image block along each pixel in the sample image block into the two-dimensional visual information of the rendered pixels in the rendered image block corresponding to each sample pixel in the sample image block, thereby mapping the three-dimensional structural information of a portion of the scene corresponding to the sample image block of the target scene into two-dimensional visual information. Finally, by using the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch, the initial reconstruction model is trained to obtain the image reconstruction model. This ensures that each rendered pixel in the rendered image patch is consistent with the corresponding sample pixel in the sample image patch in terms of color, and also ensures that the rendered image patch and the sample image patch are consistent in terms of structure. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch, but also the structural information of the sample image patch, thereby improving the reconstruction performance of the image reconstruction model in terms of image structure.

[0071] In this embodiment, the structural loss between the rendered image patch and the sample image patch is the structural difference between them. This difference can be a difference in gradient, a difference in structural similarity, or a difference in perceptual features. Alternatively, it can be any two or three of the following differences: gradient, structural similarity, or perceptual features. Based on this, this application provides a possible implementation where the structural loss between the rendered image patch and the sample image patch includes one or more of the following: gradient loss, structural similarity loss, or perceptual feature loss.

[0072] Among them, the gradient loss between the rendered image patch and the sample image patch is the difference in gradient between each rendered pixel in the rendered image patch and the corresponding sample pixel in the sample image patch; the structural similarity loss between the rendered image patch and the sample image patch is the inverse of the structural similarity between the rendered image patch and the sample image patch; and the perceptual feature loss between the rendered image patch and the sample image patch is the difference in perceptual features between the rendered image patch and the sample image patch.

[0073] This method trains the initial reconstruction model by using the gradient loss between the rendered image patch and the sample image patch when the structural loss between the rendered image patch and the sample image patch includes the gradient loss between the rendered image patch and the sample image patch. This ensures that each rendered pixel in the rendered image patch is consistent with the corresponding sample pixel in the sample image patch in terms of gradient. This allows the image reconstruction model obtained by training the initial reconstruction model to learn the gradient information of the sample image patch, thereby improving the reconstruction performance of the image reconstruction model on the image gradient in the image structure.

[0074] When the structural loss between the rendered image patch and the sample image patch includes the structural similarity loss between the rendered image patch and the sample image patch, the initial reconstruction model is trained by the structural similarity loss between the rendered image patch and the sample image patch. This ensures that the rendered image patch and the sample image patch are consistent in brightness, contrast and structure, so that the image reconstruction model obtained by training the initial reconstruction model can learn the brightness information, contrast information and structural information of the sample image patch, thereby improving the reconstruction performance of the image reconstruction model in terms of image brightness, contrast and structure in the image structure.

[0075] When the structural loss between the rendered image patch and the sample image patch includes the perceptual feature loss between the rendered image patch and the sample image patch, the initial reconstruction model is trained by the perceptual feature loss between the rendered image patch and the sample image patch. This ensures that the rendered image patch and the sample image patch are consistent in perceptual features, so that the image reconstruction model obtained by training the initial reconstruction model can learn the perceptual feature information of the sample image patch, thereby improving the reconstruction performance of the image reconstruction model in the image perceptual features of the image structure.

[0076] In this embodiment, since the gradient loss between the rendered image block and the sample image block is the difference in gradient between each rendered pixel in the rendered image block and the corresponding sample pixel in the sample image block, the step of determining the gradient loss between the rendered image block and the sample image block refers to: calculating the gradient loss between the rendered image block and the sample image block using the gradient loss function based on the rendering gradient of each rendered pixel in the rendered image block and the sample gradient of each sample pixel in the sample image block. Based on this, this application provides a possible implementation where the structural loss between the rendered image block and the sample image block includes the gradient loss between the rendered image block and the sample image block; the step of determining the gradient loss may include, for example, S1 (not shown in the figure): determining the gradient loss based on the rendering gradient of each rendered pixel in the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient loss function.

[0077] The rendering gradient of each rendered pixel in the rendered image block is the rate of change of each rendered pixel in the spatial dimension; the sample gradient of each sample pixel in the sample image block is the rate of change of each sample pixel in the spatial dimension.

[0078] In practical applications, the gradient loss is determined based on the rendering gradient of each rendered pixel in the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient loss function. This can be achieved by inputting the rendering gradient of each rendered pixel in the rendered image block and the sample gradient of each sample pixel in the sample image block into the gradient loss function, calculating the average difference in gradient between multiple rendered pixels in the rendered image block and the corresponding sample pixels in the sample image block, and obtaining the gradient loss between the rendered image block and the sample image block.

[0079] The above-mentioned S1 utilizes the gradient loss function to accurately calculate the difference in gradient between each rendered pixel in the rendered image block and the corresponding sample pixel in the sample image block by using the rendering gradient of each rendered pixel in the rendered image block and the sample gradient of each sample pixel in the sample image block. This accurately determines the gradient loss between the rendered image block and the sample image block, laying the foundation for ensuring that the gradient of each rendered pixel in the rendered image block is consistent with that of the corresponding sample pixel in the sample image block.

[0080] As an example of S1, based on the example of S203 above, the gradient loss can be L. grad Computer devices render image blocks Rendering gradient for each rendered pixel And the sample gradient g of each sample pixel in the sample image block p. i,jThe gradient loss function is used to calculate the gradient loss between the rendered image patch and the sample image patch. Where N is the total number of sample pixels in the sample image block, that is, the total number of rendered pixels in the rendered image block.

[0081] In this embodiment, since the structural similarity loss between the rendered image patch and the sample image patch is the inverse of the structural similarity between the rendered image patch and the sample image patch, the step of determining the structural similarity loss between the rendered image patch and the sample image patch refers to: firstly, calculating the structural similarity between the rendered image patch and the sample image patch using the rendered image patch and the sample image patch, and then using the structural similarity loss function to calculate the inverse of the structural similarity between the rendered image patch and the sample image patch as the structural similarity loss between the rendered image patch and the sample image patch. Based on this, this application provides a possible implementation, wherein the structural loss between the rendered image patch and the sample image patch includes the gradient loss between the rendered image patch and the sample image patch; the step of determining the gradient loss may, for example, include the following S2-S3 (not shown in the figure).

[0082] S2: Perform structural similarity calculations on the rendered image block and the sample image block to determine the structural similarity between the rendered image block and the sample image block.

[0083] S3: Determine the structural similarity loss based on the structural similarity and the structural similarity loss function.

[0084] The structural similarity between the rendered image patch and the sample image patch is the similarity between the rendered image patch and the sample image patch in terms of brightness, contrast, and structure.

[0085] In practical applications, structural similarity calculations are performed on rendered image patches and sample image patches to determine the structural similarity between them. This can be achieved by: calculating the structural similarity between the rendered image patch and the sample image patch based on the average pixel value of the rendered image patch, the average pixel value of the sample image patch, the pixel variance of the rendered image patch, the pixel variance of the sample image patch, and the pixel covariance of the rendered image patch and the sample image patch. The structural similarity loss is then determined based on the structural similarity and the structural similarity loss function. This can be achieved by: inputting the structural similarity between the rendered image patch and the sample image patch into the structural similarity loss function, calculating the inverse of the structural similarity, and obtaining the structural similarity loss between the rendered image patch and the sample image patch.

[0086] The above steps S2-S3 first calculate the structural similarity between the rendered image patch and the sample image patch, representing the similarity between the rendered image patch and the sample image patch in terms of brightness, contrast, and structure. Then, by using the structural similarity loss function, the inverse of the structural similarity between the rendered image patch and the sample image patch can be accurately calculated, thereby accurately determining the structural similarity loss between the rendered image patch and the sample image patch. This lays the foundation for accurately ensuring that the rendered image patch and the sample image patch are consistent in terms of brightness, contrast, and structure in the future.

[0087] As an example of S2-S3, based on the example of S203 above, the gradient loss can be L. SSIM Computer equipment first renders image blocks. Calculate the rendered image block using the sample image block p. Structural similarity between the sample image patch p and the sample image patch p Then use the structural similarity loss function to calculate The inverse of this is used as the structural similarity loss between the rendered image patch and the sample image patch.

[0088] In this embodiment, since the perceptual feature loss between the rendered image patch and the sample image patch is the difference in perceptual features between the rendered image patch and the sample image patch, the step of determining the perceptual feature loss between the rendered image patch and the sample image patch refers to: calculating the perceptual feature loss between the rendered image patch and the sample image patch using the rendering perceptual features of the rendered image patch and the sample perceptual features of the sample image patch, and utilizing the perceptual feature loss function. Based on this, this application provides a possible implementation, wherein the structural loss between the rendered image patch and the sample image patch includes the perceptual feature loss between the rendered image patch and the sample image patch; the step of determining the perceptual feature loss may include, for example, S4 (not shown in the figure): determining the perceptual feature loss based on the rendering perceptual features of the rendered image patch, the sample perceptual features of the sample image patch, and the perceptual feature loss function.

[0089] Among them, the rendering perceptual features of the rendered image patch are the perceptual features of the rendered image patch extracted based on the perceptual feature network; the sample perceptual features of the sample image patch are the perceptual features of the sample image patch extracted based on the perceptual feature network.

[0090] In practical applications, the perceptual feature loss is determined based on the rendering perceptual features of the rendered image patch, the sample perceptual features of the sample image patch, and the perceptual feature loss function. This can be achieved by inputting the rendering perceptual features of the rendered image patch and the sample perceptual features of the sample image patch into the perceptual feature loss function, calculating the difference between the rendering perceptual features of the rendered image patch and the sample perceptual features of the sample image patch, and obtaining the perceptual feature loss between the rendered image patch and the sample image patch.

[0091] The above-mentioned S4 utilizes the perceptual feature function to accurately calculate the difference in perceptual features between the rendered image block and the sample image block by using the rendered perceptual features of the rendered image block and the sample perceptual features of the sample image block. This accurately determines the perceptual feature loss between the rendered image block and the sample image block, laying the foundation for ensuring that the rendered image block and the sample image block are consistent in perceptual features in the future.

[0092] As an example of S4, based on the example of S203 above, the computer device perception feature loss can be L VGG Computer devices render image blocks Rendering-aware features And the sample perceptual features v(p) of the sample image patch p are used to calculate the perceptual feature loss between the rendered image patch and the sample image patch using the perceptual feature loss function.

[0093] In this embodiment, since the color loss between the rendered image block and the sample image block is the color difference between each rendered pixel in the rendered image block and the corresponding sample pixel in the sample image block, which is actually the color difference between the rendered pixel and the sample pixel corresponding to each sampling ray, the step of determining the color loss between the rendered image block and the sample image block refers to: firstly, determining the predicted color of the rendered pixel corresponding to each sampling ray using the predicted color and predicted density corresponding to each sampling point on each sampling ray; then, calculating the color loss between the rendered image block and the sample image block using the color loss function using the predicted color of the rendered pixel corresponding to each sampling ray and the sample color of the sample pixel corresponding to each sampling ray. Based on this, this application provides a possible implementation, and the color loss determination step may include, for example, the following S5-S6 (not shown in the figure).

[0094] S5: Determine the predicted color of the rendered pixel corresponding to each sampling ray based on the predicted color and predicted density of each sampling point on each sampling ray.

[0095] S6: Determine the color loss based on the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function.

[0096] In practical applications, the predicted color of the rendered pixel corresponding to each sampling ray is determined based on the predicted color and predicted density corresponding to each sampling point on each sampling ray. This can be achieved by accumulating the predicted color and predicted density corresponding to each sampling point on each sampling ray as the predicted color of the rendered pixel corresponding to each sampling ray. The color loss is determined based on the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function. This can be achieved by inputting the predicted color of the rendered pixel corresponding to each sampling ray and the sample color of the sample pixel corresponding to each sampling ray into the color loss function, and accumulating the color difference between the rendered pixel and the sample pixel corresponding to each sampling ray to obtain the color loss between the rendered image patch and the sample image patch.

[0097] The above steps S5-S6 first calculate the predicted color of the rendered pixel corresponding to each sampling point on each sampling ray based on the predicted color and predicted density. This converts the three-dimensional structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image block along each pixel in the sample image block into the two-dimensional visual information of the rendered pixel in the rendered image block corresponding to each sample pixel in the sample image block. Then, using the color loss function, the difference in color between the rendered pixel and the sample pixel corresponding to each sampling ray can be accurately calculated through the predicted color of the rendered pixel and the sample pixel corresponding to each sampling ray. That is, the difference in color between each rendered pixel in the rendered image block and the sample pixel in the corresponding sample image block can be accurately calculated, thereby accurately determining the color loss between the rendered image block and the sample image block. This lays the foundation for accurately ensuring that each rendered pixel in the rendered image block is consistent with the sample pixel in the corresponding sample image block in color.

[0098] As an example of S5-S6, based on the example of S203 above, the computer device first determines the predicted color of the rendered pixel corresponding to each sampling ray r by using the predicted color and predicted density corresponding to each sampling point on each sampling ray r. Then through The sample color C(r) of the sample pixel corresponding to each sampling ray r is calculated using a color loss function to determine the color loss between the rendered image patch and the sample image patch. in, It is a collection of multiple sampling rays.

[0099] In this embodiment, to ensure that multiple sampling points on each sampling ray not only capture the preliminary geometric information of a portion of the scene corresponding to the sample image patch, but also capture the geometric information of the surface details of that portion of the scene more precisely, preliminary sampling can be performed on each sampling ray first, followed by dense sampling near the true depth on each sampling ray to determine multiple sampling points on each sampling ray. Therefore, the step of determining multiple sampling points on each sampling ray refers to: first, preliminary sampling is performed on each sampling ray to obtain multiple sampling points; according to the true depth corresponding to each sampling point, each sampling ray is divided into a first depth region with a smaller true depth and a second depth region with a larger true depth; then, dense sampling is performed in the second depth region with a larger true depth to obtain multiple sampling points. Based on this, this application provides a possible implementation method, where the step of determining multiple sampling points may include, for example, the following S7-S9 (not shown in the figure).

[0100] S7: Determine multiple sampling points on each sampling ray.

[0101] S8: Based on the true depth corresponding to each sampling point, determine the first depth region and the second depth region on each sampling ray; the true depth of the second depth region is greater than the true depth of the first depth region.

[0102] S9: Determine multiple sampling points again in the second depth region.

[0103] The above steps S7-S9 initially sample multiple sampling points on each sampling ray, which helps the image reconstruction model learn the preliminary geometric information of the part of the scene corresponding to the sample image patch. In the depth region with large real depth on each sampling ray, multiple sampling points are densely sampled, which helps the image reconstruction model further learn the geometric information of the surface details of the part of the scene corresponding to the sample image patch, thereby further improving the reconstruction accuracy of the image reconstruction model.

[0104] As an example, color loss between rendered image patches and sample image patches in, The color is obtained by accumulating the predicted color corresponding to each sampling point in the initial sampling on the sampling ray. It is the color obtained by accumulating dense sampling in the second depth region on the sampling ray to obtain the predicted color corresponding to each sampling point.

[0105] Furthermore, in this embodiment, when the sample image block is a panoramic image block, considering that panoramic images have image distortion in the top and bottom regions due to the isometric projection characteristics, and based on the fact that each sampling ray corresponding to the sample image block is a sampling ray emitted from the shooting point of the sample image block along each sample pixel in the sample image block, the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere is different. The larger the coverage area, the greater the image distortion. In order to reduce the impact of image distortion on the training process of the image reconstruction model, it is necessary to determine that the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere. In the specific implementation of S6 above, when calculating the color loss between the rendered image block and the sample image block using the color loss function through the predicted color of the rendered pixel corresponding to each sampling ray and the sample color of the sample pixel corresponding to each sampling ray, the sampling probability corresponding to each sampling ray is further considered. Therefore, the sampling probability of each sampling ray is further determined by the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere, such that the sampling probability of each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere. Based on the sampling probability of each sampling ray, the color loss between the rendered image patch and the sample image patch can be calculated using the color loss function by using the predicted color of the rendered pixel corresponding to each sampling ray and the sample color of the sample pixel corresponding to each sampling ray.

[0106] Based on this, this application provides a possible implementation, where the sample image patch is a panoramic image patch, and the method may further include, for example, S10 (not shown in the figure): determining the sampling probability corresponding to each sampling ray based on the coverage area of ​​the sample pixel corresponding to each sampling ray on a unit sphere; the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixel corresponding to each sampling ray on a unit sphere; correspondingly, the above S6 may include, for example, S6a (not shown in the figure): determining the color loss based on the sampling probability corresponding to each sampling ray, the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function.

[0107] The area covered by the sample pixels corresponding to each sampling ray on the unit sphere is the area covered by the sample pixels corresponding to each sampling ray projected equidistantly onto the unit sphere.

[0108] In practical applications, the color loss is determined based on the sampling probability corresponding to each sampling ray, the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function. This can be achieved by inputting the sampling probability corresponding to each sampling ray, the predicted color of the rendered pixel corresponding to each sampling ray, and the sample color of the sample pixel corresponding to each sampling ray into the color loss function. Under the sampling probability corresponding to each sampling ray, the difference between the predicted color of the rendered pixel corresponding to each sampling ray and the sample color of the sample pixel corresponding to each sampling ray is accumulated to obtain the color loss between the rendered image patch and the sample image patch.

[0109] When the sample image patch is a panoramic image patch, S10 and S6a above, based on the negative correlation between the sampling probability corresponding to each sampling ray and the coverage area of ​​the sample pixel corresponding to each sampling ray on a unit sphere, utilize the color loss function. By considering the sampling probability corresponding to each sampling ray, the predicted color of the rendered pixel corresponding to each sampling ray, and the sample color of the sample pixel corresponding to each sampling ray, more attention can be paid to the color difference between the rendered pixel corresponding to each sampling ray and the sample pixel corresponding to the middle region of the sample image, while less attention is paid to the color difference between the rendered pixel corresponding to each sampling ray and the sample pixel corresponding to the top and bottom regions of the sample image. This reduces the impact of image distortion on the training process of the image reconstruction model, and more accurately calculates the color difference between the rendered pixel corresponding to each sampling ray and the sample pixel, that is, more accurately calculates the color difference between each rendered pixel in the rendered image patch and the sample pixel in the corresponding sample image patch. This more accurately determines the color loss between the rendered image patch and the sample image patch, laying the foundation for more accurately ensuring that each rendered pixel in the rendered image patch and the sample pixel in the corresponding sample image patch are consistent in color.

[0110] As an example of S10 and S6a, based on the examples of S5-S6 above, the computer device covers the area S on a unit sphere by the sample pixel point (i, j) corresponding to each sampling ray r. d (i, j), determine the sampling probability corresponding to each sampling ray. pass C(r) is calculated using a color loss function to determine the color loss between the rendered image patch and the sample image patch.

[0111] in, θ∈[θ1, θ2] is the latitude of the sample pixel (i, j) on the unit sphere, and φ∈[φ1, φ2] is the longitude of the sample pixel (i, j) on the unit sphere.

[0112] Furthermore, in this embodiment, to enable the image reconstruction model to further learn the depth information of each sample pixel in the sample image block, it is also necessary to ensure that each rendered pixel in the rendered image block is consistent in depth with the sample pixel in the corresponding sample image block. Therefore, the depth loss between the rendered image block and the sample image block needs to be calculated using a depth loss function based on the rendered depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, and the sampling point distance variance corresponding to each sampling ray. In the aforementioned S203, when training the initial reconstruction model to obtain the specific implementation of the image reconstruction model using the color loss and structural loss between the rendered image block and the sample image block, the depth loss between the rendered image block and the sample image block is further considered. Therefore, the depth loss between the rendered image block and the sample image block can be calculated using a depth loss function based on the rendered depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, and the sampling point distance variance corresponding to each sampling ray. This allows the initial reconstruction model to be trained to obtain the image reconstruction model using the depth loss, color loss, and structural loss between the rendered image block and the sample image block.

[0113] Based on this, this application provides a possible implementation method, which may further include S11 (not shown in the figure): determining the depth loss between the rendered image patch and the sample image patch based on the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function; correspondingly, the above S203 may include S203a (not shown in the figure): training an initial reconstruction model to obtain an image reconstruction model based on the depth loss, color loss, and structural loss.

[0114] Among them, the rendering depth corresponding to each sampling ray is the weighted average depth of multiple sampling points on each sampling ray; the true depth corresponding to each sampling ray is the true depth of the sample pixel corresponding to each sampling ray; the distance variance of the sampling points corresponding to each sampling ray is the distance variance of multiple sampling points on each sampling ray; the depth loss between the rendered image patch and the sample image patch is the difference in depth between each rendered pixel in the rendered image patch and the sample pixel in the corresponding sample image patch.

[0115] The above S11 and S203a utilize the depth loss function to accurately calculate the difference between the rendering depth and the true depth corresponding to each sampling ray by using the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, and the variance of the sampling point distance corresponding to each sampling ray. In other words, they accurately calculate the difference in depth between each rendering pixel in the rendered image block and the sample pixel in the corresponding sample image block, thereby accurately determining the depth loss between the rendered image block and the sample image block.

[0116] The initial reconstruction model is trained by using color loss between rendered image patches and sample image patches to ensure that each rendered pixel in the rendered image patch is consistent in color with the corresponding sample pixel in the sample image patch. Based on structural loss between rendered image patches and sample image patches to ensure structural consistency, the initial reconstruction model is further trained by using depth loss between rendered image patches and sample image patches to accurately ensure that each rendered pixel in the rendered image patch is consistent in depth with the corresponding sample pixel in the sample image patch. This trained initial reconstruction model is then used as the image reconstruction model. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch and the structural information of the sample image patch, but also the depth information of the sample image patch, thereby improving the image reconstruction performance in terms of image depth.

[0117] As an example of S11 and S203a, based on the example of S203 above, the computer device uses the rendering depth corresponding to each sampling ray r. The true depth D(r) corresponding to each sampling ray r, and the variance of the sampling point distance corresponding to each sampling ray r. The depth loss function can be used to calculate the depth loss between the rendered image patch and the sample image patch. By rendering image blocks The depth loss L between the sample image patch p and the sample image patch p depth Rendering image blocks L between sample image patch p color and rendering image blocks L between sample image patch p structure The initial reconstruction model is trained to obtain the image reconstruction model.

[0118] See Figure 5 ,Should Figure 5 This is a schematic diagram illustrating a real depth image corresponding to a sample image and a rendered depth image corresponding to a rendered image, provided in an embodiment of this application. In the above... Figure 4 Based on this, among which, Figure 5 In the image (a), the true depth image corresponding to the sample image is shown. Figure 5In the middle (b), the rendering depth image is represented.

[0119] Furthermore, in this embodiment, when the sample image block is a panoramic image block, considering that panoramic images have image distortion in the top and bottom regions due to the isometric projection characteristics, and based on the fact that each sampling ray corresponding to the sample image block is a sampling ray emitted from the shooting point of the sample image block along each sample pixel in the sample image block, the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere is different. The larger the coverage area, the greater the image distortion. In order to reduce the impact of image distortion on the training process of the image reconstruction model, it is necessary to determine that the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere. In the specific implementation of S11, the depth loss between the rendered image block and the sample image block can be calculated using the depth loss function through the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, and the sampling point distance variance corresponding to each sampling ray, the sampling probability corresponding to each sampling ray is further considered. Therefore, the sampling probability of each sampling ray is further determined by the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere, such that the sampling probability of each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere. Based on the sampling probability of each sampling ray, the depth loss between the rendered image patch and the sample image patch can be calculated using the depth loss function by using the rendering depth, the true depth, and the sampling point distance variance of each sampling ray.

[0120] Based on this, this application provides a possible implementation, where the sample image patch is a panoramic image patch, and the method may further include, for example, S12 (not shown in the figure): determining the sampling probability corresponding to each sampling ray based on the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere; the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere; correspondingly, the above S11 may include, for example, S11a (not shown in the figure): determining the depth loss based on the sampling probability corresponding to each sampling ray, the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function.

[0121] The area covered by the sample pixels corresponding to each sampling ray on the unit sphere is the area covered by the sample pixels corresponding to each sampling ray projected equidistantly onto the unit sphere.

[0122] In practical applications, the depth loss is determined based on the sampling probability, rendering depth, true depth, sampling point distance variance, and depth loss function corresponding to each sampling ray. This can be achieved by inputting these parameters into the depth loss function, and then summing the differences between the rendering depth and true depth of each sampling ray under the sampling probability and sampling point distance variance, to obtain the depth loss between the rendered image patch and the sample image patch.

[0123] When the sample image patch is a panoramic image patch, S12 and S11a above, based on the determination that the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixel point corresponding to each sampling ray on a unit sphere, utilize the depth loss function. By considering the sampling probability corresponding to each sampling ray, the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, and the variance of the sampling point distance corresponding to each sampling ray, more attention can be paid to the difference in depth between the rendered pixel point corresponding to each sampling ray and the sample pixel point corresponding to the central region of the sample image, while less attention is paid to the top and bottom regions of the sample image. The difference in depth between the rendered pixel and the sample pixel corresponding to each sampling ray is calculated to reduce the impact of image distortion on the training process of the image reconstruction model. This allows for a more accurate calculation of the difference in depth between the rendered pixel and the sample pixel corresponding to each sampling ray, i.e., a more accurate calculation of the difference in depth between each rendered pixel in the rendered image block and the sample pixel in the corresponding sample image block. This, in turn, allows for a more accurate determination of the depth loss between the rendered image block and the sample image block, laying the foundation for ensuring that each rendered pixel in the rendered image block is consistent in depth with the sample pixel in the corresponding sample image block.

[0124] As an example of S12 and S11a, based on the examples of S11 and S203a above, the computer device covers the area S on a unit sphere by the sample pixel point (i, j) corresponding to each sampling ray r. d (i, j), determine the sampling probability corresponding to each sampling ray. pass D(r) is calculated using the color loss function to determine the depth loss between the rendered image patch and the sample image patch.

[0125] Furthermore, in this embodiment, in order to enable the image reconstruction model to further learn the gradient smoothing information of the sample image patch, that is, to ensure that the rendering depth map of the rendered image patch and the sample image patch are consistent and smooth in gradient, it is also necessary to ensure that each depth pixel in the rendering depth map of the rendered image patch is consistent with the corresponding sample pixel in the sample image patch in gradient. Therefore, it is necessary to calculate the gradient smoothing loss between the rendered image patch and the sample image patch using the gradient smoothing loss function through the depth gradient of each depth pixel in the rendering depth map of the rendered image patch and the sample gradient of each sample pixel in the sample image patch. And when the initial reconstruction model is trained to obtain the specific implementation of the image reconstruction model in S203 through the color loss between the rendered image patch and the sample image patch and the structural loss between the rendered image patch and the sample image patch, the gradient smoothing loss between the rendered image patch and the sample image patch is further considered. Therefore, by further utilizing the depth gradient of each depth pixel in the rendering depth map of the rendered image patch and the sample gradient of each sample pixel in the sample image patch, the gradient smoothing function can be used to calculate the gradient smoothing loss between the rendered image patch and the sample image patch. In order to obtain the image reconstruction model by training the initial reconstruction model through the gradient smoothing loss between the rendered image patch and the sample image patch, the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch.

[0126] Based on this, this application provides a possible implementation method, which may further include S13 (not shown in the figure): determining the gradient smoothing loss between the rendered image block and the sample image block based on the depth gradient of each depth pixel in the rendered depth map of the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient smoothing loss function; correspondingly, the above S203 may include S203b (not shown in the figure): training an initial reconstruction model to obtain an image reconstruction model based on the gradient smoothing loss, color loss, and structural loss.

[0127] In this context, the depth gradient of each depth pixel in the rendering depth map of the rendered image patch is the rate of change of each depth pixel in the rendering depth map of the rendered image patch in the spatial dimension; the sample gradient of each sample pixel in the sample image patch is the rate of change of each sample pixel in the spatial dimension; and the gradient smoothing loss between the rendered image patch and the sample image patch is the difference in gradient between each depth pixel in the rendering depth map of the rendered image patch and the corresponding sample pixel in the sample image patch.

[0128] The above S13 and S203b utilize the gradient smoothing loss function to accurately calculate the difference in depth between each depth pixel in the rendering depth map of the rendered image block and the corresponding sample pixel in the sample image block by using the depth gradient of each depth pixel in the rendering depth map of the rendered image block and the sample gradient of each sample pixel in the sample image block, thereby accurately determining the gradient smoothing loss between the rendered image block and the sample image block.

[0129] The initial reconstruction model is trained by using color loss between rendered image patches and sample image patches to ensure that each rendered pixel in the rendered image patch is consistent in color with the corresponding sample pixel in the sample image patch. Based on structural loss between rendered image patches and sample image patches to ensure structural consistency, the initial reconstruction model is further trained by gradient smoothing loss between rendered image patches and sample image patches to accurately ensure that each depth pixel in the rendered depth map of the rendered image patch is consistent in gradient with the corresponding sample pixel in the sample image patch. This trained initial reconstruction model is then used as the image reconstruction model. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch and the structural information of the sample image patch, but also the gradient smoothing information of the sample image patch, thereby further improving the reconstruction performance of the image reconstruction model in terms of gradient smoothing.

[0130] As an example of S13 and S203b, based on the example of S203 above, the computer device uses the depth gradient of each depth pixel in the rendering depth map of the rendered image block. Sample gradient of each sample pixel in the sample image patch The gradient smoothing loss function can be used to calculate the gradient smoothing loss between the rendered image patch and the sample image patch. By rendering image blocks Gradient smoothing loss L between sample image patch p smooth Rendering image blocks L between sample image patch p color and rendering image blocks L between sample image patch p structure The initial reconstruction model is trained to obtain the image reconstruction model.

[0131] See Figure 6 ,Should Figure 6 This is a schematic diagram illustrating another sample image, the gradient image corresponding to the sample image, and the rendering depth image corresponding to the rendered image, provided as an embodiment of this application. Figure 6 In the middle (a), the sample image is shown. Figure 6 In the middle (b), the gradient image corresponding to the sample image is shown. Figure 6 In the image, (c) represents the rendering depth image corresponding to the rendered image.

[0132] Furthermore, in this embodiment, in order for the image reconstruction model to further learn the depth information of each sample pixel in the sample image block, it is also necessary to ensure that each rendered pixel in the rendered image block is consistent in depth with the sample pixel in the corresponding sample image block. This requires calculating the depth loss between the rendered image block and the sample image block using the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, and the variance of the sampling point distance corresponding to each sampling ray, using a depth loss function. To further enable the image reconstruction model to learn the gradient smoothing information of the sample image block, i.e., to ensure that the rendering depth map of the rendered image block is consistent and smooth in gradient with the sample image block, it is also necessary to ensure that the rendering depth map of the rendered image block... Since each depth pixel in the depth map has the same gradient as the corresponding sample pixel in the sample image patch, it is necessary to calculate the gradient smoothing loss between the rendered image patch and the sample image patch using the gradient smoothing loss function, based on the depth gradient of each depth pixel in the rendered depth map of the rendered image patch and the sample gradient of each sample pixel in the sample image patch. Furthermore, in step S203, when training the initial reconstruction model to obtain the specific implementation of the image reconstruction model using the color loss and structural loss between the rendered image patch and the sample image patch, the depth loss and gradient smoothing loss between the rendered image patch and the sample image patch are further considered. Based on this, this application provides a possible implementation method, which may further include steps S14-S15 (not shown in the figure).

[0133] S14: Determine the depth loss between the rendered image patch and the sample image patch based on the rendering depth corresponding to each sampled ray, the true depth corresponding to each sampled ray, the sampling point distance variance corresponding to each sampled ray, and the depth loss function.

[0134] S15: Determine the gradient smoothing loss between the rendered image block and the sample image block based on the depth gradient of each depth pixel in the rendered depth map of the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient smoothing loss function.

[0135] Correspondingly, the above S203 may include, for example, S203c (not shown in the figure): training an initial reconstruction model to obtain an image reconstruction model based on depth loss, gradient smoothing loss, color loss, and structural loss.

[0136] The above steps S14-S15 and S203c utilize the depth loss function. By using the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, and the variance of the sampling point distance corresponding to each sampling ray, the difference between the rendering depth and the true depth corresponding to each sampling ray can be accurately calculated. In other words, the difference in depth between each rendered pixel in the rendered image block and the sample pixel in the corresponding sample image block can be accurately calculated, thereby accurately determining the depth loss between the rendered image block and the sample image block.

[0137] By utilizing the gradient smoothing loss function, the difference in depth between each depth pixel in the rendering depth map of the rendered image patch and the corresponding sample pixel in the sample image patch can be accurately calculated by using the depth gradient of each depth pixel in the rendering depth map of the rendered image patch and the sample gradient of each sample pixel in the sample image patch. This allows for the accurate determination of the gradient smoothing loss between the rendered image patch and the sample image patch.

[0138] The initial reconstruction model is trained using color loss between rendered image patches and sample image patches to ensure that each rendered pixel in the rendered image patch is consistent in color with the corresponding sample pixel in the sample image patch. It is also trained using structural loss between rendered image patches and sample image patches to ensure structural consistency. Further training using depth loss between rendered image patches and sample image patches ensures that each rendered pixel in the rendered image patch is consistent in depth with the corresponding sample pixel in the sample image patch. Finally, training using gradient smoothing loss ensures that each depth pixel in the rendered depth map of the rendered image patch is consistent in gradient with the corresponding sample pixel in the sample image patch. This trained initial reconstruction model is then used as the image reconstruction model. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch and the structural information of the sample image patch, but also the depth information and gradient smoothing information of the sample image patch, thereby further improving the reconstruction performance of the image reconstruction model in terms of image depth and gradient smoothing.

[0139] As an example of S14-S15 and S203c, based on the example of S203 above, the computer device uses the rendering depth corresponding to each sampling ray r. The true depth D(r) corresponding to each sampling ray r, and the variance of the sampling point distance corresponding to each sampling ray r. The depth loss function can be used to calculate the depth loss between the rendered image patch and the sample image patch. The depth gradient of each depth pixel in the rendering depth map of the rendered image block is used. Sample gradient of each sample pixel in the sample image patch The gradient smoothing loss function can be used to calculate the gradient smoothing loss between the rendered image patch and the sample image patch. By rendering image blocks The depth loss L between the sample image patch p and the sample image patch p depth Rendering image blocks Gradient smoothing loss L between sample image patch p smooth Rendering image blocks L between sample image patch p color and rendering image blocks L between sample image patch p structure The initial reconstruction model is trained to obtain the image reconstruction model. This involves rendering image patches. L between sample image patch p structure This can include the gradient loss L between the rendered image patch and the sample image patch. grad The structural similarity loss between the rendered image patch and the sample image patch, L SSIM And the perceptual feature loss L between the rendered image patch and the sample image patch. VGG .

[0140] Furthermore, in this embodiment, since the image reconstruction model is a deep learning model that represents the target scene and renders a two-dimensional image through a neural network, based on multiple target rays from a new perspective of the target scene, the image reconstruction model achieves a three-dimensional representation and can output the predicted color and predicted density corresponding to each target point on each target ray; the image reconstruction model also achieves volume rendering and can output a rendered image of the target scene from a new perspective. Therefore, multiple target points on each target ray from a new perspective of the target scene are input into the image reconstruction model for color prediction and density prediction, and the predicted color and predicted density corresponding to each target point on each target ray are output; the predicted color and predicted density corresponding to each sampling point on each sampling ray are input into the initial reconstruction model for volume rendering, and the rendered image of the target scene from a new perspective is output. Based on this, this application provides a possible implementation method, which may further include the following S16-S17 (not shown in the figure).

[0141] S16: Based on multiple target points on each target ray under the new perspective of the target scene, color and density prediction are performed through the image reconstruction model to obtain the predicted color and predicted density corresponding to each target point on each target ray.

[0142] S17: Based on the predicted color and predicted density of each target point on each target ray, volume rendering is performed using an image reconstruction model to obtain a rendered image of the target scene from a new perspective.

[0143] The predicted color and density of each target point are obtained by using the image reconstruction model based on the new viewpoint direction and the spatial position of each target point, representing the visual color and opacity of each target point under the new viewpoint direction. The rendered image of the target scene under the new viewpoint is formed by using the predicted color and density of each target point on each target ray, and the color of the rendered pixel corresponding to each target ray rendered by the image reconstruction model.

[0144] The above steps S16-S17, based on the deep learning model that uses a neural network to represent the target scene and render a 2D image, achieve 3D representation through the image reconstruction model. This model outputs the predicted color and density of each target point on each target ray from the new perspective of the target scene, representing the 3D structural information of each target point on each target ray emitted from the new perspective. Volumetric rendering is achieved through the image reconstruction model, outputting a rendered image of the target scene from the new perspective. This converts the 3D structural information of each target point on each target ray emitted from the new perspective into the 2D visual information of the corresponding rendered pixels of each target ray, thus mapping the 3D structural information of the 3D scene from the new perspective to 2D visual information.

[0145] Furthermore, in this embodiment, the predicted color and predicted density corresponding to each target point on each target ray under the new perspective of the target scene represent the three-dimensional structural information of each target point on each target ray emitted under the new perspective of the target scene, thus representing the three-dimensional structural information of the three-dimensional scene under the new perspective of the target scene. Therefore, by realizing three-dimensional reconstruction through the image reconstruction model, a three-dimensional model of the target scene under the new perspective can be obtained. Based on this, this application provides a possible implementation method, which may further include, for example, the following S18 (not shown in the figure): performing three-dimensional reconstruction on the predicted color and predicted density corresponding to each target point on each target ray to obtain a three-dimensional model of the target scene under the new perspective.

[0146] The above-mentioned S18 uses the predicted color and predicted density of each target point on each target ray to achieve three-dimensional reconstruction through the image reconstruction model, thereby obtaining a three-dimensional model of the target scene from a new perspective. This enables the display of the three-dimensional structural information of each target point on each target ray emitted from the new perspective of the target scene, that is, the display of the three-dimensional structural information of each target point.

[0147] In summary, the image reconstruction model processing method provided in this application can render highly realistic 2D images from new perspectives of a specific scene based on a very small number of 2D images from multiple viewpoints in a specific scenario, or can reconstruct highly realistic 3D models of a specific scene. This reduces computational costs and data acquisition difficulties, while significantly reducing computational resource consumption. It can run on ordinary consumer-grade hardware without requiring expensive professional equipment. It is suitable not only for high-end applications but can also be widely applied to ordinary applications, giving it high market competitiveness and greatly improving the application efficiency and practicality of image reconstruction models. Furthermore, the image reconstruction model processing method provided in this application significantly improves rendering and reconstruction speeds, allowing for the generation of highly realistic 2D images or 3D models of a specific scene from new perspectives without long waiting times.

[0148] The image reconstruction model processing method provided in this application embodiment, combined with the optimization method of the sampling probability corresponding to the sampling ray, can render a highly realistic panoramic image from a new perspective of a specific scene based on a very small number of panoramic images from multiple perspectives in a specific scene.

[0149] The image reconstruction model processing method provided in this application embodiment, combined with depth loss, has significant robustness against inaccurate depth estimation and camera pose deviation. It can better adapt to different input conditions in practical application scenarios. Even if the depth information of the two-dimensional image under multiple perspectives in a specific scene has errors, or the camera pose data is not accurate enough, it can still stably render a highly realistic two-dimensional image under a new perspective of a specific scene, or reconstruct a highly realistic three-dimensional model of a specific scene, without distortion or blurring.

[0150] The image reconstruction model processing method provided in this application embodiment, combined with gradient smoothing loss, can reduce artifacts in highly realistic 2D images from new perspectives in specific scenes, making highly realistic 2D images from new perspectives in specific scenes clearer and more natural.

[0151] Based on this, the image reconstruction model processing method provided in this application embodiment can be applied to various application scenarios such as indoor scenes and outdoor scenes. It can render highly realistic two-dimensional images of a specific scene from a new perspective based on a very small number of two-dimensional images from multiple perspectives in a specific scene, or can reconstruct highly realistic three-dimensional models of a specific scene.

[0152] In indoor scenarios, such as virtual house viewing, indoor navigation, virtual museums, and indoor virtual games, the highly realistic 2D images from a new perspective not only preserve the details of the indoor scene but also accurately present the changes in light and shadow and the geometric structure of indoor objects, making it feel as if you are in a real indoor scene, greatly enhancing the sense of immersion.

[0153] In outdoor scenarios, such as virtual tourism, outdoor navigation, and outdoor virtual games, the highly realistic 2D images from the new perspective are close-up 2D images that clearly display the details and textures of outdoor objects, allowing for close observation of every detail in the outdoor scene; the highly realistic 2D images from the new perspective are also distant 2D images that can still accurately present the geometric structure and depth information of the outdoor scene, avoiding blurring and distortion problems.

[0154] It should be noted that, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0155] based on Figure 2 In accordance with the image reconstruction model processing method provided in the corresponding embodiments, this application also provides an image reconstruction model processing apparatus, see [link to relevant documentation]. Figure 7 ,Should Figure 7 This is a structural diagram of a processing device for an image reconstruction model provided in an embodiment of this application. The processing device 700 for the image reconstruction model includes: a prediction unit 701, a rendering unit 702, and a training unit 703.

[0156] The prediction unit 701 is used to predict color and density based on multiple sampling points on each sampling ray corresponding to the sample image block of the target scene, and to obtain the predicted color and predicted density corresponding to each sampling point on each sampling ray through an initial reconstruction model based on the neural radiation field.

[0157] The rendering unit 702 is used to perform volume rendering based on the predicted color and predicted density corresponding to each sampling point on each sampling ray, through the initial reconstruction model, to obtain the rendered image block corresponding to the sample image block;

[0158] Training unit 703 is used to train an initial reconstruction model to obtain an image reconstruction model based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch.

[0159] In one possible implementation, the structural loss between the rendered image patch and the sample image patch includes one or more of the following: gradient loss between the rendered image patch and the sample image patch, structural similarity loss between the rendered image patch and the sample image patch, or perceptual feature loss between the rendered image patch and the sample image patch.

[0160] In one possible implementation, the structural loss between the rendered image patch and the sample image patch includes the gradient loss between the rendered image patch and the sample image patch; the processing device 700 of the image reconstruction model includes: a first determining unit;

[0161] The first determining unit is used to determine the gradient loss based on the rendering gradient of each rendered pixel in the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient loss function.

[0162] In one possible implementation, the structural loss between the rendered image patch and the sample image patch includes the structural similarity loss between the rendered image patch and the sample image patch; the processing device 700 of the image reconstruction model includes: a second determining unit;

[0163] The second determining unit is used to perform structural similarity calculation on the rendered image block and the sample image block to determine the structural similarity between the rendered image block and the sample image block.

[0164] The structural similarity loss is determined based on the structural similarity and the structural similarity loss function.

[0165] In one possible implementation, the structural loss between the rendered image patch and the sample image patch includes the perceptual feature loss between the rendered image patch and the sample image patch; the processing device 700 of the image reconstruction model includes: a third determining unit;

[0166] The third determining unit is used to determine the perceptual feature loss based on the rendering perceptual features of the rendered image patch, the sample perceptual features of the sample image patch, and the perceptual feature loss function.

[0167] In one possible implementation, the image reconstruction model processing device 700 includes: a fourth determining unit;

[0168] The fourth determining unit is used to determine the predicted color of the rendered pixel corresponding to each sampling ray based on the predicted color and predicted density corresponding to each sampling point on each sampling ray; and to determine the color loss based on the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function.

[0169] In one possible implementation, the fourth determining unit is further used for:

[0170] Multiple sampling points are determined on each sampling ray;

[0171] Based on the true depth corresponding to each sampling point, a first depth region and a second depth region are determined on each sampling ray; the true depth of the second depth region is greater than the true depth of the first depth region.

[0172] Multiple sampling points were identified again in the second depth region.

[0173] In one possible implementation, the sample image patch is a panoramic image patch, and the image reconstruction model processing device 700 includes: a fifth determining unit;

[0174] The fifth determining unit is used to determine the sampling probability corresponding to each sampling ray based on the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere; the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere.

[0175] The fourth determining unit is used to determine the color loss based on the sampling probability corresponding to each sampling ray, the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function.

[0176] In one possible implementation, the image reconstruction model processing device 700 includes: a sixth determining unit;

[0177] The sixth determining unit is used to determine the depth loss between the rendered image patch and the sample image patch based on the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function.

[0178] Training unit 703 is used to train an initial reconstruction model based on depth loss, color loss, and structural loss to obtain an image reconstruction model.

[0179] In one possible implementation, the sample image patch is a panoramic image patch, and the image reconstruction model processing device 700 includes: a seventh determining unit;

[0180] The seventh determining unit is used to determine the sampling probability corresponding to each sampling ray based on the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere; the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere.

[0181] The sixth determining unit is used to determine the depth loss based on the sampling probability corresponding to each sampling ray, the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function.

[0182] In one possible implementation, the image reconstruction model processing device 700 includes: an eighth determining unit;

[0183] The eighth determining unit is used to determine the gradient smoothing loss between the rendered image block and the sample image block based on the depth gradient of each depth pixel in the rendered depth map of the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient smoothing loss function.

[0184] Training unit 703 is used to train an initial reconstruction model to obtain an image reconstruction model based on gradient smoothing loss, color loss, and structural loss.

[0185] In one possible implementation, the image reconstruction model processing device 700 includes: a ninth determining unit;

[0186] The ninth determining unit is used to determine the depth loss between the rendered image block and the sample image block based on the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function; and to determine the gradient smoothing loss between the rendered image block and the sample image block based on the depth gradient of each depth pixel in the rendering depth map of the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient smoothing loss function.

[0187] Training unit 703 is used to train an initial reconstruction model to obtain an image reconstruction model based on depth loss, gradient smoothing loss, color loss, and structural loss.

[0188] In one possible implementation, the prediction unit 701 is further configured to:

[0189] Based on multiple target points on each target ray from a new perspective of the target scene, color and density prediction are performed through an image reconstruction model to obtain the predicted color and predicted density corresponding to each target point on each target ray.

[0190] Rendering unit 702 is also used for:

[0191] Based on the predicted color and density of each target point on each target ray, volumetric rendering is performed using an image reconstruction model to obtain a rendered image of the target scene from a new perspective.

[0192] In one possible implementation, the image reconstruction model processing device 700 includes: a reconstruction determination unit;

[0193] The reconstruction unit is used to perform three-dimensional reconstruction of the predicted color and predicted density of each target point on each target ray, so as to obtain a three-dimensional model of the target scene from a new perspective.

[0194] As can be seen from the above technical solution, the image reconstruction model processing device includes a prediction unit, a rendering unit, and a training unit. The prediction unit inputs multiple sampling points on each sampling ray corresponding to a sample image patch of the target scene into an initial reconstruction model based on neural radiation fields for color and density prediction. It outputs the predicted color and predicted density corresponding to each sampling point on each sampling ray, representing the three-dimensional structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image patch along each sample pixel in the sample image patch, thus representing the three-dimensional structural information of a portion of the scene corresponding to the sample image patch of the target scene. The rendering unit inputs the predicted color and predicted density corresponding to each sampling point on each sampling ray into the initial reconstruction model for volume rendering, outputting the rendered image patch corresponding to the sample image patch. This converts the three-dimensional structural information of each sampling point on each sampling ray emanating from the shooting point of the sample image patch along each pixel in the sample image patch into the two-dimensional visual information of the rendered pixels in the rendered image patch corresponding to each sample pixel in the sample image patch, thereby mapping the three-dimensional structural information of a portion of the scene corresponding to the sample image patch of the target scene into two-dimensional visual information. The training unit trains the initial reconstruction model to obtain the image reconstruction model by using the color loss between the rendered image patch and the sample image patch, as well as the structural loss between the rendered image patch and the sample image patch. This ensures that each rendered pixel in the rendered image patch is consistent with the sample pixel in the corresponding sample image patch in terms of color, and also ensures that the rendered image patch and the sample image patch are consistent in terms of structure. This allows the image reconstruction model to learn not only the color information of each sample pixel in the sample image patch, but also the structural information of the sample image patch, thereby improving the reconstruction performance of the image reconstruction model in terms of image structure.

[0195] This application also provides a computer device, which may be a server, see [link to previous document]. Figure 8 ,Should Figure 8 This application provides a structural diagram of a server 800. The server 800 can vary significantly due to different configurations or performance characteristics. It may include one or more processors, such as a central processing unit (CPU) 822, and a memory 832, as well as one or more storage media 830 (e.g., one or more mass storage devices) for storing application programs 842 or data 844. The memory 832 and storage media 830 can be temporary or persistent storage. The program stored in the storage media 830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 822 may be configured to communicate with the storage media 830 and execute the series of instruction operations stored in the storage media 830 on the server 800.

[0196] Server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input / output interfaces 858, and / or one or more operating systems 841.

[0197] In this embodiment, the central processing unit 822 in the server 800 can execute the methods provided in the various optional implementations of the above embodiments.

[0198] The computer device provided in this application embodiment can also be a terminal, see [link to relevant documentation]. Figure 9 ,Should Figure 9 This is a structural diagram of a terminal provided in an embodiment of this application. Taking a smartphone as an example, the smartphone includes components such as a radio frequency (RF) circuit 910, a memory 920, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a wireless Fidelity (WiFi) module 970, a processor 980, and a power supply 990. The input unit 930 may include a touch panel 931 and other input devices 932, the display unit 940 may include a display panel 941, and the audio circuit 960 may include a speaker 961 and a microphone 962. Those skilled in the art will understand that... Figure 9 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0199] The memory 920 can be used to store software programs and modules. The processor 980 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 920 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0200] The processor 980 is the control center of the smartphone, connecting various parts of the smartphone via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 920, and by accessing data stored in the memory 920. Optionally, the processor 980 may include one or more processing units; preferably, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 980.

[0201] In this embodiment, the processor 980 in the smartphone can execute the methods provided in the various optional implementations of the above embodiments.

[0202] According to one aspect of this application, a computer-readable storage medium is provided for storing a computer program that, when run on a computer device, causes the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0203] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0204] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0205] The terms "first," "second," etc., used in this application's specification and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0209] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), RAM, magnetic disks, or optical disks.

[0210] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0211] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing an image reconstruction model, characterized in that, The method includes: Based on multiple sampling points on each sampling ray corresponding to the sample image block of the target scene, color prediction and density prediction are performed through an initial reconstruction model based on neural radiation field to obtain the predicted color and predicted density corresponding to each sampling point on each sampling ray. Based on the predicted color and predicted density of each sampling point on each sampling ray, volume rendering is performed through the initial reconstruction model to obtain the rendered image block corresponding to the sample image block; The initial reconstruction model is trained based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch, to obtain the image reconstruction model.

2. The method according to claim 1, characterized in that, The structural loss between the rendered image patch and the sample image patch includes one or more of the following: gradient loss between the rendered image patch and the sample image patch, structural similarity loss between the rendered image patch and the sample image patch, or perceptual feature loss between the rendered image patch and the sample image patch.

3. The method according to claim 2, characterized in that, The structural loss between the rendered image patch and the sample image patch includes the gradient loss between the rendered image patch and the sample image patch; the step of determining the gradient loss includes: The gradient loss is determined based on the rendering gradient of each rendered pixel in the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient loss function.

4. The method according to claim 2, characterized in that, The structural loss between the rendered image patch and the sample image patch includes the structural similarity loss between the rendered image patch and the sample image patch; the steps for determining the structural similarity loss include: Structural similarity calculations are performed on the rendered image block and the sample image block to determine the structural similarity between the rendered image block and the sample image block; The structural similarity loss is determined based on the structural similarity and the structural similarity loss function.

5. The method according to claim 2, characterized in that, The structural loss between the rendered image patch and the sample image patch includes the perceptual feature loss between the rendered image patch and the sample image patch; the step of determining the perceptual feature loss includes: The perceptual feature loss is determined based on the rendering perceptual features of the rendered image patch, the sample perceptual features of the sample image patch, and the perceptual feature loss function.

6. The method according to claim 1, characterized in that, The steps for determining the color loss include: The predicted color of the rendered pixel corresponding to each sampling ray is determined based on the predicted color and predicted density corresponding to each sampling point on each sampling ray. The color loss is determined based on the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function.

7. The method according to claim 6, characterized in that, The steps for determining the multiple sampling points include: Multiple sampling points are determined on each sampling ray; Based on the true depth corresponding to each sampling point, a first depth region and a second depth region are determined on each sampling ray; the true depth of the second depth region is greater than the true depth of the first depth region. Multiple sampling points were determined again in the second depth region.

8. The method according to claim 6, characterized in that, The sample image patch is a panoramic image patch, and the method further includes: The sampling probability corresponding to each sampling ray is determined based on the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere; the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere. The step of determining the color loss based on the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function includes: The color loss is determined based on the sampling probability corresponding to each sampling ray, the predicted color of the rendered pixel corresponding to each sampling ray, the sample color of the sample pixel corresponding to each sampling ray, and the color loss function.

9. The method according to claim 1, characterized in that, The method further includes: The depth loss between the rendered image block and the sample image block is determined based on the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function. The step of training the initial reconstruction model to obtain the image reconstruction model based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch, includes: The initial reconstruction model is trained based on the depth loss, the color loss, and the structural loss to obtain the image reconstruction model.

10. The method according to claim 9, characterized in that, The sample image patch is a panoramic image patch, and the method further includes: The sampling probability corresponding to each sampling ray is determined based on the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere; the sampling probability corresponding to each sampling ray is negatively correlated with the coverage area of ​​the sample pixels corresponding to each sampling ray on a unit sphere. The step of determining the depth loss based on the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the variance of the sampling point distance corresponding to each sampling ray, and the depth loss function includes: The depth loss is determined based on the sampling probability corresponding to each sampling ray, the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function.

11. The method according to claim 1, characterized in that, The method further includes: The gradient smoothing loss between the rendered image block and the sample image block is determined based on the depth gradient of each depth pixel in the rendered depth map of the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient smoothing loss function. The step of training the initial reconstruction model to obtain the image reconstruction model based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch, includes: The image reconstruction model is obtained by training the initial reconstruction model based on the gradient smoothing loss, the color loss, and the structural loss.

12. The method according to claim 1, characterized in that, The method further includes: The depth loss between the rendered image block and the sample image block is determined based on the rendering depth corresponding to each sampling ray, the true depth corresponding to each sampling ray, the sampling point distance variance corresponding to each sampling ray, and the depth loss function. The gradient smoothing loss between the rendered image block and the sample image block is determined based on the depth gradient of each depth pixel in the rendered depth map of the rendered image block, the sample gradient of each sample pixel in the sample image block, and the gradient smoothing loss function. The step of training the initial reconstruction model to obtain the image reconstruction model based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch, includes: The initial reconstruction model is trained using the depth loss, the gradient smoothing loss, the color loss, and the structural loss to obtain the image reconstruction model.

13. The method according to any one of claims 1-12, characterized in that, The method further includes: Based on multiple target points on each target ray from a new perspective of the target scene, color and density prediction are performed using the image reconstruction model to obtain the predicted color and predicted density corresponding to each target point on each target ray. Based on the predicted color and predicted density of each target point on each target ray, volume rendering is performed using the image reconstruction model to obtain a rendered image of the target scene from a new perspective.

14. The method according to claim 13, characterized in that, The method further includes: The predicted color and predicted density of each target point on each target ray are reconstructed in three dimensions to obtain a three-dimensional model of the target scene from a new perspective.

15. A processing apparatus for an image reconstruction model, characterized in that, The device includes: a prediction unit, a rendering unit, and a training unit; The prediction unit is used to perform color prediction and density prediction based on an initial reconstruction model based on neural radiation field, according to multiple sampling points on each sampling ray corresponding to the sample image block of the target scene, so as to obtain the predicted color and predicted density corresponding to each sampling point on each sampling ray. The rendering unit is used to perform volume rendering based on the predicted color and predicted density corresponding to each sampling point on each sampling ray, through the initial reconstruction model, to obtain the rendered image block corresponding to the sample image block; The training unit is used to train the initial reconstruction model to obtain an image reconstruction model based on the color loss between the rendered image patch and the sample image patch, and the structural loss between the rendered image patch and the sample image patch.

16. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute the method according to any one of claims 1-14 according to instructions in the computer program.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer device, causes the computer device to perform the method according to any one of claims 1-14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is run on a computer device, it causes the computer device to perform the method according to any one of claims 1-14.

Images