A model training method, a three-dimensional scene reconstruction method and device

Abstract

The application relates to a model training method in the field of artificial intelligence, which comprises the following steps: acquiring a first image; determining the loss of each pixel point in the first image based on at least a first model to obtain N losses, wherein the loss of any one pixel point in the first image is used to represent the deviation between the color value of the any one pixel point in the first image and the color value corresponding to the any one pixel point obtained by processing the first model; acquiring the weight of each loss in the N losses, wherein the weights of at least two losses are different; processing the N losses based on the weight of each loss in the N losses to obtain the loss corresponding to the first image; and training the first model based on the loss corresponding to the first image. In this way, in the process of training the model, the pixel points in the image are treated differently, the important pixel points can be paid more attention to and the unimportant pixel points can be ignored in the process of training the model, and the effect of model training is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method, a three-dimensional scene reconstruction method and apparatus. Background Technology

[0002] Neural radiance fields (NeRF) is a technique for reconstructing 3D scenes using multiview images, and it has become a popular technique in 3D scene reconstruction in recent years. NeRF's task is to capture a scene from a known viewpoint (including captured images and corresponding camera intrinsics and extrinsic parameters), synthesize images from new viewpoints, and then perform 3D reconstruction. NeRF has achieved excellent results on dense viewpoint training data (i.e., capturing a sufficient number of images from different viewpoints of a 3D scene), but its performance is poor on sparse viewpoint training data (e.g., capturing only 3 images of a scene). However, in real-world applications, it is often difficult to collect large amounts of training data. For example, when a mobile phone user reconstructs a target 3D scene by taking photos, only a few training images from a limited number of viewpoints are often available, making it difficult to reconstruct a high-quality 3D scene. Therefore, how to reconstruct high-quality 3D scenes under sparse viewpoints (i.e., with limited training data) is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This application provides a model training method, a 3D scene reconstruction method, an apparatus, a computing device, a computing device cluster, a computer storage medium, and a computer product, which can reconstruct high-quality 3D scenes under sparse perspective.

[0004] In a first aspect, this application provides a model training method, which may include: acquiring a first sample set, the first sample set including a first image; determining the loss of each pixel in the first image based at least on a first model to obtain N losses, wherein the loss of any pixel in the first image is used to characterize the deviation between the color value of any pixel in the first image and the color value corresponding to any pixel obtained by processing the first model; acquiring the weight of each of the N losses, wherein at least two of the N losses have different weights; processing the N losses based on the weights of each of the N losses to obtain the loss corresponding to the first image; and training the first model based on the loss corresponding to the first image.

[0005] In this way, by treating pixels in the image differently during model training, more attention can be paid to important pixels while ignoring unimportant ones, thus improving the training effect. Furthermore, this can enhance the subsequent 3D scene reconstruction results using the model.

[0006] In one possible implementation, the weight of each of the N losses is obtained. Specifically, this can include: for any one of the N losses, the ratio of that loss to the sum of the N losses is used as the weight of that loss. When the loss of a pixel is large, it indicates that the pixel is difficult to optimize and therefore requires more attention. In this case, the ratio between the loss of that pixel and the sum of the losses of all pixels in the image will be large, meaning that the loss of that pixel has a large weight. This allows for greater attention to that pixel in subsequent optimization processes, thus differentiating it from other pixels.

[0007] In one possible implementation, the weights of each of the N losses are obtained. Specifically, this can include: for any one of the N losses, determining the world coordinates of a first point on the surface of an object in space, based on the pixel coordinates of the first pixel corresponding to that loss, the depth of the first pixel, and the intrinsic and extrinsic parameters of the camera that captured the first image; determining the first depth of the second pixel based on the world coordinates of the first point, the pixel coordinates of the second pixel corresponding to the first point in the second image, and the intrinsic and extrinsic parameters of the camera that captured the second image, where the first and second images are images from different viewpoints; and determining the weight of any one loss based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image. Thus, by using the depth of the same point in the physical world in different images, the importance of the pixel corresponding to that point in the image can be determined, and an appropriate weight can be assigned to that pixel, thereby differentiating it from other pixels.

[0008] In one possible implementation, the weight of any one loss is determined based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image. Specifically, this can include: obtaining the deviation value between the first and second depths; when the deviation value is less than or equal to a preset threshold, determining the weight of any one loss as the first weight; when the deviation value is greater than the preset threshold, determining the weight of any one loss as the second weight, where the second weight is less than the first weight. For example, when the deviation value is less than or equal to the threshold, it indicates that the depth obtained by the point corresponding to the first pixel on the object satisfies consistency in the transformation of different viewpoints, i.e., the point has a geometric correspondence, and therefore the loss of the pixel in the 2D image corresponding to that point can be assigned a higher weight. When the deviation value is greater than the threshold, it indicates that the depth obtained by the point corresponding to the first pixel on the object does not satisfy consistency in the transformation of different viewpoints, i.e., the point does not have a geometric correspondence, and therefore the loss of the pixel in the 2D image corresponding to that point can be assigned a lower weight. This achieves differentiated treatment of each pixel.

[0009] In one possible implementation, after training the first model, the method may further include: obtaining the world coordinates of Q locations input by the user through the interface and the orientation of each of the Q locations in the camera coordinate system, where Q ≥ 1; processing the world coordinates and orientations of each of the Q locations using the first model to obtain color values ​​at the Q locations; reconstructing the 3D scene based on the color values ​​at the Q locations; and outputting the reconstructed 3D scene. Thus, after training the required model, the corresponding locations in the scene to be reconstructed can be input into the model to obtain the reconstructed 3D scene.

[0010] Secondly, this application provides a three-dimensional scene reconstruction method, which may include: obtaining the world coordinates of Q locations and the orientation of each of the Q locations in the camera coordinate system, where Q≥1, input by a user through an interface; processing the world coordinates and orientation in the camera coordinate system of each of the Q locations using at least a first model to obtain color values ​​at the Q locations, wherein the first model is trained based on the loss of each pixel contained in a first image, at least two of the losses in the loss of each pixel contained in the first image have different weights, and the loss of any pixel contained in the first image is used to characterize the deviation between the color value of any pixel in the first image and the color value corresponding to any pixel obtained by the processing of the first model; reconstructing a 3D scene based on the color values ​​at the Q locations, and outputting the reconstructed 3D scene.

[0011] Thirdly, this application provides a model training apparatus, which may include an acquisition module and a processing module. The acquisition module may be used to acquire a first sample set, which includes a first image. The processing module may be used to determine the loss of each pixel in the first image, based at least on a first model, to obtain N losses, wherein the loss of any pixel in the first image is used to characterize the deviation between the color value of any pixel in the first image and the color value corresponding to any pixel obtained by processing with the first model. The processing module may also be used to acquire the weight of each of the N losses, wherein at least two losses have different weights; and to process the N losses based on their respective weights to obtain the loss corresponding to the first image. After obtaining the loss corresponding to the first image, the processing module may also be used to train the first model based on the loss corresponding to the first image.

[0012] In one possible implementation, when the processing module obtains the weight of each of the N losses, it can specifically use the ratio of any one of the N losses to the sum of the N losses as the weight of that one loss.

[0013] In one possible implementation, when the processing module obtains the weight of each of the N losses, it can specifically: for any one of the N losses, based on the pixel coordinates of the first pixel corresponding to the first pixel, the depth of the first pixel, and the intrinsic and extrinsic parameters of the camera that captured the first image, determine the world coordinates of the first point on the surface of the object in space; based on the world coordinates of the first point, the pixel coordinates of the second pixel corresponding to the first point in the second image, and the intrinsic and extrinsic parameters of the camera that captured the second image, determine the first depth of the second pixel, wherein the first image and the second image are images from different perspectives; and based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image, determine the weight of any one of the losses.

[0014] In one possible implementation, when the processing module determines the weight of any one loss based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image, it can specifically be used to: obtain the deviation value between the first depth and the second depth; when the deviation value is less than or equal to a preset threshold, determine the weight of any one loss as the first weight; when the deviation value is greater than the preset threshold, determine the weight of any one loss as the second weight, wherein the second weight is less than the first weight.

[0015] In one possible implementation, after training the first model, the acquisition module can also acquire the world coordinates of Q locations input by the user through the interface and the orientation of each of the Q locations in the camera coordinate system, where Q ≥ 1. At this point, the processing module can further process the world coordinates and orientations of each of the Q locations using the first model to obtain the color values ​​at the Q locations, and based on these color values, reconstruct the 3D scene and output the reconstructed 3D scene.

[0016] Fourthly, this application provides a three-dimensional scene reconstruction device, which may include an acquisition module and a processing module. The acquisition module can be used to acquire the world coordinates of Q locations and the orientation of each of the Q locations in the camera coordinate system, where Q ≥ 1, input by a user through an interface. The processing module can be used to process the world coordinates and orientations in the camera coordinate system of each of the Q locations using at least a first model to obtain color values ​​at the Q locations. The first model is trained based on the loss of each pixel in a first image, where at least two losses in the first image have different weights, and the loss of any pixel in the first image is used to characterize the deviation between the color value of any pixel in the first image and the color value corresponding to any pixel obtained by the first model. The processing module can also be used to reconstruct a 3D scene based on the color values ​​at the Q locations and output the reconstructed 3D scene.

[0017] Fifthly, this application provides a computing device, comprising: at least one memory for storing a program and at least one processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof, or to execute the method described in the second aspect.

[0018] In a sixth aspect, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect, or performs the method described in the second aspect.

[0019] In a seventh aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation of the first aspect, or to perform the method described in the second aspect.

[0020] Eighthly, this application provides a computer program product that, when run on a processor, causes the processor to execute the method described in the first aspect or any possible implementation of the first aspect, or to execute the method described in the second aspect.

[0021] It is understood that the beneficial effects of the second to eighth aspects mentioned above can be found in the relevant descriptions in the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a pixel coordinate system, an image coordinate system, and a camera coordinate system provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram illustrating a process of reconstructing a 3D scene using NeRF, as provided in an embodiment of this application.

[0024] Figure 3 This is a schematic flowchart of a model training method provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating the steps for determining the weight of a pixel's loss according to an embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating the correspondence between a point on an object in space and pixels in different images, provided in an embodiment of this application.

[0027] Figure 6 This is a flowchart illustrating a 3D scene reconstruction method provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0031] Figure 10 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0032] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0033] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0034] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0036] First, the technical terms involved in the embodiments of this application will be introduced.

[0037] (1) Pixel coordinate system

[0038] The pixel coordinate system represents the projection of a three-dimensional object onto the image plane. Pixels are discretized, with their origin at the top-left corner of the image plane. Their horizontal axis is parallel to the image plane and extends horizontally to the right of the image, while their vertical axis is perpendicular to the horizontal axis and points downwards towards the image. For example, as shown... Figure 1 As shown, the pixel coordinate system may include a u-axis, a v-axis, and an origin O1. O1 is the top-left vertex of image a, the u-axis is parallel to the plane of image a and extends horizontally to the right of image a, and the v-axis is perpendicular to the u-axis and points downwards.

[0039] (2) Image coordinate system

[0040] The origin of the image coordinate system is the center point of the image, its horizontal axis is parallel to the horizontal axis of the pixel coordinate system, and its vertical axis is parallel to the vertical axis of the pixel coordinate system. For example, as shown... Figure 1 As shown, the image coordinate system may include an x-axis, a y-axis, and an origin O2. O2 is the center point of image a, the x-axis is parallel to the u-axis of the pixel coordinate system, and the y-axis is parallel to the v-axis of the pixel coordinate system. (Continue reading...) Figure 1If the origin O2 of the image coordinate system has coordinates (u0, v0) in the pixel coordinate system, and dx and dy represent the physical dimensions of each pixel on the horizontal axis x and vertical axis y, respectively, then the transformation relationship between the pixel coordinate system and the image coordinate system is as follows:

[0041] (Formula 1)

[0042] Formula 1 can be written in matrix form as follows:

[0043] (Formula 2)

[0044] (3) Camera coordinate system

[0045] The camera coordinate system is a three-dimensional coordinate system with its origin at the camera's optical center. Its horizontal axis is parallel to the horizontal axis of the image coordinate system, and its vertical axis is parallel to the vertical axis of the image coordinate system. The camera's optical axis is the Z-axis. The camera coordinate system follows the right-hand rule. The camera's optical center can be understood as the geometric center of the camera lens. For example,... Figure 1 As shown, the camera coordinate system can include X c Axis, Y c Axis, Z c The axis and origin O3. Where O3 is the optical center of the camera, X... c The axis is parallel to the x-axis of the image coordinate system, and the Y-axis is parallel to the x-axis. c The axis is parallel to the y-axis of the image coordinate system. See further... Figure 1 If the camera's focal length is f, then the transformation relationship between the image coordinate system and the camera coordinate system is:

[0046] (Formula 3)

[0047] Formula 3 can be written in matrix form as follows:

[0048] (Formula 4)

[0049] (4) World coordinate system

[0050] The world coordinate system is used to represent the absolute coordinates of objects in space. The camera coordinate system can be obtained by rotation and translation from the world coordinate system. For example, if the coordinates of the world coordinate system are represented by (X... w Y w Z w Let R be the rotation matrix and t be the 3D translation vector. Then the transformation relationship between the camera coordinate system and the world coordinate system is:

[0051] (Formula 5))

[0052] From Formula 2, Formula 4, and Formula 5, we can obtain:

[0053] (Formula 6)

[0054] In Formula 6, This is the intrinsic parameter matrix of the camera. This is the extrinsic parameter matrix of the camera.

[0055] Next, the technical solutions provided in the embodiments of this application will be introduced.

[0056] For example, Figure 2 This demonstrates a process for reconstructing a 3D scene using NeRF. For example... Figure 2 As shown, the process of reconstructing a 3D scene using NeRF can include steps S201 to S209, specifically:

[0057] S201. Obtain the training sample set, which includes at least one image.

[0058] In this embodiment, the training sample set may include at least one image. This training sample set can be a dataset pre-defined by the user, or a dataset input by the user through an interface provided by the device or system. When the training sample set is a pre-defined dataset, it can be obtained, but is not limited to, from a storage area specified by the user.

[0059] S202. For a pixel in an image, sample it on the light ray that starts from the origin of the camera coordinate system and passes through the pixel to obtain multiple sampling points.

[0060] In this embodiment, for each pixel in any image of the training sample set, sampling can be performed on the light ray originating from the origin of the camera coordinate system and passing through the corresponding pixel to obtain multiple sampling points corresponding to the pixel. For example, continue reading... Figure 1 Taking pixel b in image a as an example, a ray can be emitted from the origin O3 of the camera coordinate system to pixel b, and this ray passes through pixel b to obtain ray c. Then, a series of sampling points can be obtained from ray c within a pre-defined boundary.

[0061] S203. Input the world coordinates of each sampling point and the direction of the light in the camera coordinate system into the neural network model so that the neural network model can output the voxel density and color value of each sampling point.

[0062] In this embodiment, after obtaining the sampling points corresponding to a pixel, the world coordinates of these sampling points and the direction of the light in the camera coordinate system can be input into the neural network model so that the neural network model can output the voxel density and color value of each sampling point.

[0063] S204. Determine the pixel color value of the corresponding pixel based on the voxel density and color value of each sampling point.

[0064] In this embodiment, after obtaining the voxel density and color value of each sampling point, the pixel color value of the corresponding pixel can be calculated using the pixel color calculation formula. For example, the pixel color calculation formula can be as follows:

[0065] (Formula 7)

[0066] in, and These are the near and far boundaries corresponding to the light rays; Let be the voxel density at the t-th sampling point; Let t be the color value of the t-th sampling point; Light from arrive Transparency.

[0067] S205. Determine the loss of a pixel based on its pixel color value and the true value of its color.

[0068] In this embodiment, after obtaining the pixel color value, it and the true value of the pixel's color can be input into the loss function to obtain the loss for that pixel. For example, the true value of the pixel's color can refer to the pixel's color value in the image. For example, the loss for each pixel is used to characterize the deviation between the pixel's color value in the image and the color value obtained through the neural network model.

[0069] S206. Assign the same weight to the loss of all pixels in the image to obtain the loss corresponding to the image.

[0070] In this embodiment, after obtaining the loss of all pixels in an image, the loss of all pixels in the image can be assigned the same weight, and the loss corresponding to the image can be obtained by weighted summation or other calculation methods.

[0071] S207. Train the neural network model based on the loss corresponding to at least one image in the training sample set.

[0072] In this embodiment, after obtaining the losses corresponding to one or more images, the neural network model can be trained with the goal of minimizing these losses. For example, after obtaining the losses corresponding to one or more images, the parameters in the neural network model can be updated using, but are not limited to, the backpropagation algorithm, to complete the training of the neural network model.

[0073] S208. After training is completed, a position and orientation are input into the neural network model so that the neural network model outputs the voxel density and color value corresponding to the position and orientation.

[0074] In this embodiment, after training the neural network model, any position and direction can be input into the neural network so that the neural network model can output the voxel density and color value corresponding to that position and direction.

[0075] S209. Based on the voxel density and color values ​​corresponding to each position and direction output by the neural network model, render the synthetic viewpoint image.

[0076] In this embodiment, after obtaining the voxel density and color value corresponding to each position and direction, the color value of the pixel corresponding to each position and direction can be calculated by the aforementioned "Formula 7", and then the composite view image can be rendered.

[0077] Therefore, the neural network model is first trained using a training sample set to obtain the desired model. Then, this model can be used to render a clear scene image from any angle. However, this method assigns the same weight to every pixel in the image when calculating the image loss. This makes it difficult to distinguish which pixels are more important and which are less important, thus preventing differentiated treatment during training. When training data is insufficient, it is often necessary to focus more on important pixels and ignore less important ones to improve the model's training effect and consequently, the subsequent 3D scene reconstruction. Therefore, this method struggles to achieve satisfactory 3D scene reconstruction results under sparse perspectives (i.e., with limited training data).

[0078] In view of this, this application provides a model training method that can treat pixels in an image differently during model training, thereby focusing more on important pixels and ignoring unimportant pixels, thus improving the model training effect and subsequently improving the 3D scene reconstruction effect.

[0079] For example, Figure 3 A model training method is illustrated. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Among them, Figure 3 S301 in the above can be found in the previous text. Figure 2 S201 in Figure 3 S302 in the above can be found in the previous text. Figure 2 S202 to S205 are not detailed here. Figure 3As shown, the model training method includes the following steps S301 to S305, specifically:

[0080] S301. Obtain the first sample set, which includes the first image.

[0081] S302. Based at least on the first model, determine the loss of each pixel in the first image to obtain N losses. For example, the loss of any pixel in the first image is used to characterize the deviation between the color value of that pixel in the first image and the color value corresponding to that pixel obtained by processing the first model.

[0082] S303. Obtain the weight of each of the N losses, wherein at least two of the N losses have different weights.

[0083] In this embodiment, weights can be assigned to N losses, ensuring that at least two of these N losses have different weights. This allows for differentiated treatment of pixels in the image. For example, the weight of each loss can be the ratio of that loss to the sum of all losses. For instance, if N=3, and the first loss is 2, the second loss is 4, and the third loss is 4, then the weight of the first loss is 2 / (2+4+4)=0.2, and the weights of the second and third losses are both 4 / (2+4+4)=0.4. When a pixel has a large loss, it indicates that the pixel is difficult to optimize and therefore requires more attention. In this case, the ratio between the loss of that pixel and the sum of the losses of all pixels in the image will be large, meaning the loss of that pixel has a large weight. This allows for greater attention to that pixel in subsequent optimization processes, thus achieving differentiated treatment compared to other pixels.

[0084] One possible implementation is to determine the importance of a pixel in an image by analyzing the depth of a point on an object in space across different images, in order to assign weights to that pixel. Specifically, such as... Figure 4 As shown, determining the weight of any one of N losses can include the following steps:

[0085] S3031. Based on the pixel coordinates of the first pixel in the first image, determine the world coordinates of the first point corresponding to the first pixel on the surface of the object.

[0086] In this embodiment, the pixel coordinates of the first pixel (i.e., the coordinates of the first pixel in the pixel coordinate system) can be transformed using the aforementioned "Formula 6" to obtain the world coordinates of the first point corresponding to the first pixel on the object surface. At this time, during the calculation process, Z in "Formula 6"... cThe value of can be the depth of the first pixel, which can be obtained by the depth sensor (such as LiDAR) on the camera that captured the first image. In addition, the intrinsic and extrinsic parameter matrices of the camera required in "Formula 6" can be preset or obtained by processing the first image using a preset algorithm. The specific method depends on the actual situation and is not limited here.

[0087] For example, such as Figure 5 As shown, the first image can be image 51, the first pixel can be pixel 511, the object 53 is an object in space, the point corresponding to pixel 511 on the surface of object 53 is point 531, and the pixel coordinates of pixel 511 can be... The depth of pixel 511 can be Pixel coordinates based on pixel 511 The depth of pixel 511 By combining the camera's intrinsic and extrinsic parameters, we can obtain the world coordinates of point 531. .in, . It is the extrinsic parameter matrix of the camera that captured image 51. It is the intrinsic parameter matrix of the camera that captured image 51.

[0088] S3032. Based on the world coordinates of the first point and the pixel coordinates of the second pixel point corresponding to the first point in the second image, determine the depth of the second pixel point, wherein the first image and the second image are images from different perspectives.

[0089] In this embodiment, the world coordinates of the first point and the pixel coordinates of the second pixel corresponding to the first point in the second image can be processed using the aforementioned "Formula 6" to obtain the depth of the second pixel. During the calculation process, the camera's intrinsic and extrinsic parameter matrices required in "Formula 6" can be preset or obtained by processing the first image using a pre-defined algorithm. The specific method depends on the actual situation and is not limited here.

[0090] For example, see further. Figure 5 The second image is image 52, the second pixel can be pixel 521, and the pixel coordinates of pixel 511 can be... The first point is point 531. The pixel coordinates based on pixel point 521 can be... The world coordinates of point 531 In addition to the camera's intrinsic and extrinsic parameters, the depth of pixel 521 can be obtained. .in, . It is the extrinsic parameter matrix of the camera that captures the 52 images. It is the intrinsic parameter matrix of the camera that captured image 52.

[0091] S3033. Based on the obtained depth of the second pixel and the true depth of the second pixel, determine the weight of the loss of the first pixel.

[0092] In this embodiment, the deviation (such as a ratio or difference) between the obtained depth of the second pixel and the true depth of the second pixel can be compared with a preset threshold. When the deviation is less than or equal to the threshold, it indicates that the depth obtained by the point corresponding to the first pixel on the object satisfies the consistency in the transformation of different viewpoints, that is, the point has a geometric correspondence, so the pixel loss of the corresponding 2D image can be given a higher weight. When the deviation is greater than the threshold, it indicates that the depth obtained by the point corresponding to the first pixel on the object does not satisfy the consistency in the transformation of different viewpoints, that is, the point does not have a geometric correspondence, so the pixel loss of the corresponding 2D image can be given a lower weight. For example, see below. Figure 5 , For the true depth of pixel 521, through and the calculated depth The weight of the loss of pixel 511 can be determined. In some embodiments, when the deviation value is less than or equal to a threshold, the weight of the loss of the first pixel can be determined as a first weight; when the deviation value is greater than the threshold, the weight of the loss of the first pixel can be determined as a second weight, wherein the first weight is greater than the second weight. For example, both the first weight and the second weight can be preset.

[0093] This allows us to determine the weight of the loss at the first pixel in the first image. The method for determining the weight of the loss at other pixels in the first image can be the same as that used to determine the weight of the loss at the first pixel, and will not be repeated here.

[0094] After obtaining the weights of the loss for each pixel in the first image, S304 can be executed.

[0095] S304. Based on the weight of each of the N losses, process the N losses to obtain the loss corresponding to the first image.

[0096] In this embodiment, the N losses can be summed using the weights of each loss to obtain the loss corresponding to the first image. For example, if the image contains 3 pixels (N=3), and the first loss is a, the second loss is b, and the third loss is c, with a weight of 0.2 for the first loss, 0.3 for the second loss, and 0.5 for the third loss, then the loss corresponding to the image is: 0.2a + 0.3b + 0.5c. Of course, other methods can also be used, such as summing and averaging, depending on the actual situation, and are not limited here.

[0097] S305. Train the first model based on the loss corresponding to the first image.

[0098] In this embodiment, after obtaining the loss corresponding to the first image, the first model can be trained based on the loss corresponding to the first image. For example, but not limited to, the parameters in the first model can be updated using the backpropagation algorithm to complete the training of the first model. In some embodiments, when the first sample set also includes other images, the first model can also be trained using the loss corresponding to the first image and the losses corresponding to the other images after obtaining the losses corresponding to the other images. For example, the loss corresponding to the first image can be added to the losses corresponding to the other images, and the parameters in the first model can be updated using the result.

[0099] Therefore, by treating pixels in an image differently during model training, more attention can be paid to important pixels while ignoring unimportant ones, thus improving the training effect and consequently enhancing the subsequent 3D scene reconstruction.

[0100] After training the first model, it can be used to reconstruct 3D scenes.

[0101] For example, Figure 6 A 3D scene reconstruction method is illustrated. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 6 As shown, this 3D scene reconstruction method includes the following steps:

[0102] S601. Obtain the world coordinates of Q positions and the orientation of each of the Q positions in the camera coordinate system, where Q ≥ 1.

[0103] In this embodiment, the user can input the world coordinates of the location to be rendered and its orientation in the camera coordinate system through the interface provided by the device or system. This yields the world coordinates of Q locations and the orientation of each of the Q locations in the camera coordinate system.

[0104] S602. The world coordinates of each of the Q positions and the orientation of each position in the camera coordinate system are processed by the first model to obtain the color values ​​at the Q positions.

[0105] In this embodiment, the world coordinates of each of the Q positions and the orientation of each position in the camera coordinate system can be input into the first model for processing, and the color values ​​at the Q positions can be output.

[0106] S603: Based on the color values ​​at Q locations, reconstruct the 3D scene and output the reconstructed 3D scene.

[0107] In this embodiment, after obtaining the color values ​​at Q positions, the colors corresponding to these color values ​​can be displayed to complete the reconstruction of the 3D scene. After reconstructing the 3D scene, the reconstructed 3D scene can be output, such as displaying the reconstructed 3D scene to the user.

[0108] Thus, the 3D scene can be reconstructed using the aforementioned trained model.

[0109] Based on the methods in the above embodiments, this application also provides a model training device.

[0110] For example, Figure 7 The structure of a model training device is shown. For example... Figure 7 As shown, the model training device 700 may include an acquisition module 701 and a processing module 702. The acquisition module 701 can be used to acquire a first sample set, which includes a first image. The processing module 702 can be used to determine the loss of each pixel in the first image, at least based on a first model, to obtain N losses, wherein the loss of any pixel in the first image is used to characterize the deviation between the color value of any pixel in the first image and the color value corresponding to any pixel obtained by processing with the first model. The processing module 702 can also be used to acquire the weight of each of the N losses, wherein at least two losses have different weights; and to process the N losses based on their respective weights to obtain the loss corresponding to the first image. After obtaining the loss corresponding to the first image, the processing module 702 can also be used to train the first model based on the loss corresponding to the first image.

[0111] In some embodiments, when the processing module 702 obtains the weight of each loss among the N losses, it can specifically be used to: for any one of the N losses, take the ratio of the random loss to the sum of the N losses as the weight of the random loss.

[0112] In some embodiments, when the processing module 702 obtains the weight of each of the N losses, it may specifically be used to: for any one of the N losses, determine the world coordinates of a first point on the surface of an object in space based on the pixel coordinates of the first pixel corresponding to the loss, the depth of the first pixel, and the intrinsic and extrinsic parameters of the camera that captured the first image; determine the first depth of the second pixel based on the world coordinates of the first point, the pixel coordinates of the second pixel corresponding to the first point in the second image, and the intrinsic and extrinsic parameters of the camera that captured the second image, wherein the first image and the second image are images from different perspectives; and determine the weight of any one loss based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image.

[0113] In some embodiments, when the processing module 702 determines the weight of any one loss based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image, it may specifically be used to: obtain the deviation value between the first depth and the second depth; when the deviation value is less than or equal to a preset threshold, determine the weight of any one loss as the first weight; when the deviation value is greater than the preset threshold, determine the weight of any one loss as the second weight, wherein the second weight is less than the first weight.

[0114] In some embodiments, after training the first model is completed, the acquisition module 701 can also be used to acquire the world coordinates of Q positions and the orientation of each of the Q positions in the camera coordinate system, where Q ≥ 1, as input by the user through the interface. At this time, the processing module 702 can also be used to process the world coordinates and orientation of each of the Q positions in the camera coordinate system using the first model to obtain the color values ​​at the Q positions, and reconstruct the 3D scene based on the color values ​​at the Q positions, and output the reconstructed 3D scene.

[0115] It should be understood that the above-described device is used to execute the model training method described in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above method. The working process of the device can be referred to the corresponding process in the above method, and will not be repeated here.

[0116] Based on the methods in the above embodiments, this application also provides a three-dimensional scene reconstruction device.

[0117] For example, Figure 8 The structure of a three-dimensional scene reconstruction device is shown. For example... Figure 8As shown, the 3D scene reconstruction device 800 may include an acquisition module 801 and a processing module 802. The acquisition module 801 can acquire the world coordinates of Q locations and the orientation of each of the Q locations in the camera coordinate system, where Q ≥ 1, input by the user through an interface. The processing module 802 can process the world coordinates and orientations in the camera coordinate system of each of the Q locations using at least a first model to obtain color values ​​at the Q locations. The first model is trained based on the loss of each pixel in the first image. At least two of the losses in the first image have different weights, and the loss of any pixel in the first image is used to characterize the deviation between the color value of any pixel in the first image and the color value corresponding to any pixel obtained by the first model. The processing module 802 can also reconstruct the 3D scene based on the color values ​​at the Q locations and output the reconstructed 3D scene.

[0118] It should be understood that the above-mentioned device is used to execute the 3D scene reconstruction method described in the above embodiments. The corresponding program modules in the device are similar in implementation principle and technical effect to those described in the above method. The working process of the device can be referred to the corresponding process in the above method, and will not be repeated here.

[0119] In some embodiments, for Figure 7 The acquisition module 701 and processing module 702 in the middle, and Figure 8 Both the acquisition module 801 and the processing module 802 can be implemented in software or in hardware. For example, the following will use... Figure 7 Taking the acquisition module 701 as an example, the implementation method of the acquisition module 701 is introduced. Similarly, Figure 7 The processing module 702 in the middle, and Figure 8 The implementation methods of the acquisition module 801 and processing module 802 can be referred to Figure 7 The implementation method of the acquisition module 701 in the document.

[0120] As an example of a software functional unit, module 701 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0121] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0122] As an example of a hardware functional unit, the acquisition module 701 may include at least one computing device, such as a server. Alternatively, the acquisition module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0123] The multiple computing devices included in the acquisition module 701 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 701 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 701 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0124] It should be noted that, in other embodiments, the acquisition module 701 can be used to perform... Figure 3 The processing module 702 can be used to execute any step in the model training method described herein. Figure 3 In the model training method described herein, any step, the step implemented by the acquisition module 701 and the processing module 702, can be specified as needed and implemented by the acquisition module 701 and the processing module 702 respectively. Figure 3 The different steps in the model training method described herein implement all the functions of the model training device 700. Additionally, in other embodiments, the acquisition module 801 can be used to perform... Figure 6 The processing module 802 can be used to execute any step in the 3D scene reconstruction method described herein. Figure 6 Any step in the 3D scene reconstruction method described herein, the steps implemented by the acquisition module 801 and the processing module 802 can be specified as needed, and implemented by the acquisition module 801 and the processing module 802 respectively. Figure 6 The different steps in the 3D scene reconstruction method described herein enable the full functionality of the 3D scene reconstruction device 800.

[0125] This application also provides a computing device 900. For example... Figure 9 As shown, the computing device 900 includes a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0126] The 902 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9The bus 902 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 902 may include a path for transmitting information between various components of the computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0127] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0128] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0129] The memory 906 stores executable program code, and the processor 904 executes the executable program code to implement the aforementioned functions respectively. Figure 7 The functions of the acquisition module 701 and the processing module 702 are implemented to achieve the above. Figure 3 All or part of the steps in the model training method described herein. That is, memory 106 stores the steps for performing the above-described steps. Figure 3 Instructions for all or part of the steps in the model training method described herein. In some embodiments, the memory 906 stores executable program code, which the processor 904 executes to implement the aforementioned steps respectively. Figure 8 The functions of the acquisition module 801 and the processing module 802 are implemented to achieve the above. Figure 6 All or part of the steps in the 3D scene reconstruction method described herein. That is, memory 106 stores the steps for performing the above-described steps. Figure 6 Instructions for all or part of the steps in the 3D scene reconstruction method described herein.

[0130] Alternatively, the memory 906 stores executable code, and the processor 904 executes the executable code to implement the functions of the aforementioned model training device 700, thereby achieving the above-mentioned goals. Figure 3All or part of the steps in the model training method described herein. That is, memory 906 stores the steps for performing the above. Figure 3 Instructions for all or part of the steps in the model training method described herein. In some embodiments, the memory 906 stores executable code, and the processor 904 executes the executable code to implement the functions of the aforementioned 3D scene reconstruction apparatus 800, thereby achieving the above-described... Figure 6 All or part of the steps in the 3D scene reconstruction method described herein. That is, memory 906 stores the steps for performing the above-described steps. Figure 6 Instructions for all or part of the steps in the 3D scene reconstruction method described herein.

[0131] The communication interface 903 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 100 and other devices or communication networks.

[0132] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0133] like Figure 10 As shown, the computing device cluster includes at least one computing device 900. The memory 906 of one or more computing devices 900 in the computing device cluster may store the same instructions for performing all or part of the steps in the methods of the above embodiments.

[0134] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for performing all or part of the steps in the methods of the above embodiments. In other words, a combination of one or more computing devices 900 can jointly execute instructions for performing all or part of the steps in the methods of the above embodiments.

[0135] It should be noted that the memories 906 in the different computing devices 900 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the model training device 700. That is, the instructions stored in the memories 906 of the different computing devices 900 can implement the aforementioned... Figure 7 The acquisition module 701 and processing module 702 provide the functionality of one or more modules. In some embodiments, the memory 906 in different computing devices 900 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the 3D scene reconstruction device 800. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the aforementioned functions. Figure 8 The functions of one or more modules in the acquisition module 801 and processing module 802.

[0136] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0137] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0138] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0139] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0140] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0141] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0142] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain a first sample set, which includes a first image, wherein the first image is a 3D scene image acquired from a sparse viewpoint; Based at least on the first model, the loss of each pixel in the first image is determined to obtain N losses. The loss of any pixel in the first image is used to characterize the deviation between the color value of the pixel in the first image and the color value corresponding to the pixel obtained by the first model. The first model is used for three-dimensional scene reconstruction. Obtain the weight of each of the N losses, wherein at least two of the N losses have different weights; Based on the weight of each of the N losses, the N losses are processed to obtain the loss corresponding to the first image; The first model is trained based on the loss corresponding to the first image.

2. The method according to claim 1, characterized in that, The step of obtaining the weight of each of the N losses specifically includes: For any one of the N losses, the ratio of that loss to the sum of the N losses is used as the weight of that loss.

3. The method according to claim 1, characterized in that, The step of obtaining the weight of each of the N losses specifically includes: For any one of the N losses, based on the pixel coordinates of the first pixel corresponding to the first pixel, the depth of the first pixel, and the intrinsic and extrinsic parameters of the camera that captured the first image, determine the world coordinates of the first point corresponding to the first pixel on the surface of the object in space. Based on the world coordinates of the first point, the pixel coordinates of the second pixel point corresponding to the first point in the second image, and the intrinsic and extrinsic parameters of the camera that captured the second image, the first depth of the second pixel point is determined, wherein the first image and the second image are images from different perspectives; The weight of any one of the losses is determined based on the first depth of the second pixel and the second depth of the second pixel obtained when the second image is captured.

4. The method according to claim 3, characterized in that, The determination of the weight of any one of the losses based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image specifically includes: Obtain the deviation value between the first depth and the second depth; When the deviation value is less than or equal to a preset threshold, the weight of any one of the losses is determined as the first weight. When the deviation value is greater than the preset threshold, the weight of any one of the losses is determined as the second weight, wherein the second weight is less than the first weight.

5. The method according to any one of claims 1-4, characterized in that, After training the first model is completed, the method further includes: Obtain the world coordinates of Q locations input by the user through the interface and the orientation of each of the Q locations in the camera coordinate system, where Q≥1; The first model is used to process the world coordinates and orientation in the camera coordinate system of each of the Q positions to obtain the color values ​​at the Q positions. Based on the color values ​​at the Q locations, reconstruct the 3D scene and output the reconstructed 3D scene.

6. A method for reconstructing a three-dimensional scene, characterized in that, The method includes: Obtain the world coordinates of Q locations input by the user through the interface and the orientation of each of the Q locations in the camera coordinate system, where Q≥1; At least the world coordinates and orientation in the camera coordinate system of each of the Q positions are processed by a first model to obtain the color values ​​at the Q positions. The first model is trained based on the loss of each pixel in the first image. At least two of the losses in the loss of each pixel in the first image have different weights. The loss of any pixel in the first image is used to characterize the deviation between the color value of the pixel in the first image and the color value corresponding to the pixel obtained by the first model. Based on the color values ​​at the Q locations, reconstruct the 3D scene and output the reconstructed 3D scene.

7. A model training device, characterized in that, The device includes: The acquisition module is used to acquire a first sample set, which includes a first image, wherein the first image is a three-dimensional scene image acquired from a sparse viewpoint; The processing module is used to determine the loss of each pixel in the first image based at least on the first model, so as to obtain N losses, wherein the loss of any pixel in the first image is used to characterize the deviation between the color value of the arbitrary pixel in the first image and the color value corresponding to the arbitrary pixel obtained by processing the first model, and the first model is used for three-dimensional scene reconstruction. The processing module is further configured to obtain the weight of each of the N losses, wherein at least two of the N losses have different weights. The processing module is further configured to process the N losses based on the weight of each loss among the N losses, so as to obtain the loss corresponding to the first image; The processing module is further configured to train the first model based on the loss corresponding to the first image.

8. The apparatus according to claim 7, characterized in that, When obtaining the weight of each of the N losses, the processing module is specifically used for: For any one of the N losses, the ratio of that loss to the sum of the N losses is used as the weight of that loss.

9. The apparatus according to claim 7, characterized in that, When obtaining the weight of each of the N losses, the processing module is specifically used for: For any one of the N losses, based on the pixel coordinates of the first pixel corresponding to the first pixel, the depth of the first pixel, and the intrinsic and extrinsic parameters of the camera that captured the first image, determine the world coordinates of the first point corresponding to the first pixel on the surface of the object in space. Based on the world coordinates of the first point, the pixel coordinates of the second pixel point corresponding to the first point in the second image, and the intrinsic and extrinsic parameters of the camera that captured the second image, the first depth of the second pixel point is determined, wherein the first image and the second image are images from different perspectives; The weight of any one of the losses is determined based on the first depth of the second pixel and the second depth of the second pixel obtained when the second image is captured.

10. The apparatus according to claim 9, characterized in that, When determining the weight of any one of the losses based on the first depth of the second pixel and the second depth of the second pixel obtained when capturing the second image, the processing module is specifically used for: Obtain the deviation value between the first depth and the second depth; When the deviation value is less than or equal to a preset threshold, the weight of any one of the losses is determined as the first weight. When the deviation value is greater than the preset threshold, the weight of any one of the losses is determined as the second weight, wherein the second weight is less than the first weight.

11. The apparatus according to any one of claims 7-10, characterized in that, After training the first model is completed, the acquisition module is further configured to acquire the world coordinates of Q positions input by the user through the interface and the orientation of each of the Q positions in the camera coordinate system, where Q≥1; The processing module is further configured to process the world coordinates and orientation in the camera coordinate system of each of the Q positions using the first model to obtain the color values ​​at the Q positions, and to reconstruct the 3D scene based on the color values ​​at the Q positions, and to output the reconstructed 3D scene.

12. A three-dimensional scene reconstruction device, characterized in that, The device includes: The acquisition module is used to acquire the world coordinates of Q locations input by the user through the interface and the orientation of each of the Q locations in the camera coordinate system, where Q≥1; The processing module is used to process the world coordinates and orientation in the camera coordinate system of each of the Q positions using at least a first model to obtain the color values ​​at the Q positions. The first model is trained based on the loss of each pixel in the first image. At least two of the losses in the loss of each pixel in the first image have different weights. The loss of any pixel in the first image is used to characterize the deviation between the color value of the pixel in the first image and the color value corresponding to the pixel obtained by the first model. The processing module is also used to reconstruct a 3D scene based on the color values ​​at the Q positions, and to output the reconstructed 3D scene.

13. A computing device, characterized in that, include: At least one memory for storing programs; At least one processor for executing the program stored in the memory; When the program stored in the memory is executed, the processor is used to execute the method as described in any one of claims 1-5, or to execute the method as described in claim 6.

14. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-5, or to perform the method as described in claim 6.

15. A computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method as claimed in any one of claims 1-5, or to perform the method as claimed in claim 6.

16. A computer program product, characterized in that, When the computer program product is run on a processor, the processor performs the method as described in any one of claims 1-5, or performs the method as described in claim 6.

Images