Scene reconstruction model training method and device, scene reconstruction method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a scene reconstruction model training method and device, a scene reconstruction method and device, computer equipment, a computer readable storage medium and a computer program product, and the method comprises the steps: determining each first sampling point corresponding to each calibration image, and determining a first feature value of each first sampling point; determining a pose of a camera corresponding to each view angle in a plurality of view angles of the scene, and determining each first target view angle corresponding to each first rendering view angle and a first weight corresponding to each first target view angle based on the pose of the camera corresponding to each view angle; for each first rendering view angle, performing spatial clustering based on the appearance embedding of each first weight and each first target view angle to obtain the appearance embedding of the first rendering view angle, the appearance embedding representing the features of the corresponding view angle image related to ambient illumination and / or brightness; and performing model training based on the appearance embedding of each first rendering view angle and the first feature value of each first sampling point to obtain a scene reconstruction model. By adopting the method, the robustness of illumination variation can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of large scene reconstruction, and in particular to a scene reconstruction model training method, scene reconstruction method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] Large-scale complex scene reconstruction is an important research direction in the field of computer vision and computer graphics. This field covers the reconstruction of various complex environments, from cities, natural landscapes to indoor buildings, aiming to create highly realistic three-dimensional scene models. These models can be used in many fields such as virtual reality, augmented reality, game development, cultural heritage protection, urban planning and security monitoring. Generally speaking, large-scale three-dimensional scene models are usually represented by explicit structures such as point clouds, triangle patches, and voxel grids. In contrast, with the rapid development of neural radiance field (NeRF) technology, reconstruction methods based on implicit expression and neural rendering have attracted more and more attention from researchers and companies. Compared with explicit expression that requires a large amount of storage space to restore fine details, implicit expression can express more complete models in a smaller volume. At the same time, compared with traditional rendering that inevitably introduces errors using explicit lighting models, neural rendering based on neural networks can obtain highly realistic rendering images in a data-driven manner.

[0003] Neural implicit field (NeRF) methods have achieved fast and high-fidelity reconstruction in small scenes. However, in large-scale scenes or even city-level scenes, when faced with complex outdoor environments and drastically changing lighting conditions, NeRF methods still have limitations in terms of robustness to lighting changes. Summary of the invention

[0004] Based on this, it is necessary to provide a scene reconstruction model training method, scene reconstruction method, device, computer equipment, computer-readable storage medium and computer program product that can improve the robustness to lighting changes in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a scene reconstruction model training method, the method comprising:

[0006] Determine each first sampling point corresponding to each calibration image, and determine a first eigenvalue corresponding to each first sampling point, wherein each calibration image is obtained by performing image acquisition on a scene to be reconstructed;

[0007] Determine a position and posture of a camera corresponding to each of a plurality of perspectives of a scene, and determine first target perspectives corresponding to first rendering perspectives and first weights corresponding to first target perspectives based on the position and posture of the camera corresponding to each of the perspectives;

[0008] For each of the first rendering perspectives, spatial clustering is performed based on the first weights and the appearance embeddings corresponding to the first target perspectives to obtain an appearance embedding corresponding to the first rendering perspective, where the appearance embedding represents features of the corresponding perspective image related to ambient lighting and / or brightness;

[0009] Model training is performed based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalues ​​corresponding to each of the first sampling points to obtain a scene reconstruction model.

[0010] In one of the embodiments, determining each first target perspective corresponding to each first rendering perspective based on the position and posture of the camera corresponding to each perspective, and first weights corresponding to each first target perspective, includes:

[0011] Based on the position and posture of the camera corresponding to each of the perspectives, obtaining a first distance between the camera corresponding to each of the first rendering perspectives and cameras corresponding to other perspectives;

[0012] Based on the first distance, selecting k other perspectives as first target perspectives for each of the first rendering perspectives, where k is a positive integer greater than 0;

[0013] A first weight corresponding to each first target viewing angle is determined based on the first distance.

[0014] In one embodiment, before determining each first sampling point corresponding to each calibration image, the method includes:

[0015] Acquire parameters of each camera and a first sparse point cloud corresponding to the scene, and determine a first space occupied by the scene based on the parameters of each camera and the first sparse point cloud corresponding to the scene;

[0016] Dividing the first space into a plurality of sub-blocks, and determining calibration images corresponding to each sub-block;

[0017] The determining of each first sampling point corresponding to each calibration image includes:

[0018] For each sub-block, determining each first sampling point corresponding to each calibration image;

[0019] The performing model training based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalue corresponding to each of the first sampling points to obtain a scene reconstruction model includes:

[0020] For each sub-block, based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalues ​​corresponding to each of the first sampling points, a neural network model training is performed to obtain a scene reconstruction model corresponding to each sub-block.

[0021] In one embodiment, determining the first eigenvalue corresponding to each of the first sampling points includes:

[0022] Based on the hash table, the first eigenvalue represented by the hash grid corresponding to each of the first sampling points is queried, wherein each of the hash grids corresponds to one of the sub-blocks, and each hash grid stores the information of the calibration image of the corresponding sub-block in the corresponding hash table.

[0023] In one embodiment, after determining each first sampling point based on each calibration image, the method further includes:

[0024] Determining an initial bounded area based on a first space occupied by the scene;

[0025] In the case that the first sampling point is not within the initial bounded region, the first sampling point is nonlinearly mapped to a difference region between a target bounded region and the initial bounded region, and the initial bounded region is a subset of the target bounded region.

[0026] In one embodiment, the performing model training based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalue corresponding to each of the first sampling points to obtain the scene reconstruction model includes:

[0027] Inputting the first characteristic value corresponding to each of the first sampling points into the first neural network to obtain the model volume density and model characteristics of each of the first sampling points;

[0028] Obtaining a model color corresponding to each of the first sampling points based on the model features of each of the first sampling points, the directions of each of the first sampling points, and the appearance embedding corresponding to each of the first rendering perspectives;

[0029] The model volume density and the model color of each of the first sampling points are synthesized to obtain the model color of the corresponding pixel point on the calibration image plane;

[0030] Based on the real colors of the pixels in the calibration image and the model colors, model training is performed to obtain a scene reconstruction model.

[0031] In a second aspect, the present application also provides a scene reconstruction method, the method comprising:

[0032] Determine each second sampling point corresponding to the image to be processed, and determine a second eigenvalue corresponding to each second sampling point, wherein the image to be processed is obtained by performing image acquisition on the scene to be reconstructed;

[0033] Determine the pose of a camera corresponding to each of the multiple perspectives of the scene to be reconstructed, and determine second target perspectives corresponding to each second rendering perspective based on the pose of the camera corresponding to each of the perspectives, and second weights corresponding to each of the second target perspectives;

[0034] For each second rendering perspective, spatial clustering is performed based on each second weight and the appearance embedding corresponding to each second target perspective to obtain an appearance embedding corresponding to the second rendering perspective, where the appearance embedding represents a feature of the corresponding perspective image related to ambient lighting and / or brightness;

[0035] A rendered image is obtained based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each of the second sampling points, and a scene reconstruction model.

[0036] In one embodiment, the method further comprises:

[0037] Obtain a number of sub-blocks obtained by dividing the scene to be reconstructed, and determine the image to be processed corresponding to each sub-block;

[0038] The obtaining of a rendered image based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each of the second sampling points, and the scene reconstruction model includes:

[0039] Obtaining a scene reconstruction model of each of the sub-blocks;

[0040] Based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point, and the scene reconstruction model of the corresponding sub-block, respectively process to obtain the volume density and color of a plurality of second sampling points corresponding to each sub-block;

[0041] A rendered image is obtained based on the volume density and color of a number of second sampling points corresponding to each sub-block.

[0042] In one embodiment, obtaining a rendered image based on the volume density and color of a plurality of second sampling points corresponding to each sub-block includes:

[0043] Determine the target volume density and target color of the second sampling point based on the positional relationship between the second sampling point and the sub-block;

[0044] A rendered image is obtained based on the target volume density and the target color of each of the second sampling points.

[0045] In one of the embodiments, determining the target volume density and target color of the second sampling point based on the positional relationship between the second sampling point and the sub-block includes:

[0046] In the case where the second sampling point is located within the sub-block, determining the target volume density and target color of the second sampling point based on the corresponding sub-block;

[0047] In the case where the second sampling point is located outside each of the sub-blocks, determining a second distance between the second sampling point and each of the sub-blocks, and selecting a volume density and a color of the second sampling point in the corresponding sub-block based on the second distance as a target volume density and a target color of the second sampling point located outside the sub-block;

[0048] A rendered image is obtained based on the volume density and color of each of the second sampling points.

[0049] In one embodiment, after obtaining a plurality of sub-blocks obtained by dividing the scene to be reconstructed, the method further includes:

[0050] Expanding the area of ​​each of the sub-blocks;

[0051] The determining the target volume density and target color of the second sampling point based on the positional relationship between the second sampling point and the sub-block includes:

[0052] In a case where the second sampling point is located in an overlapping area of ​​the sub-blocks, determining a sub-block corresponding to the overlapping area;

[0053] Acquire a third distance between the second sampling point located in the overlapping area and each of the determined sub-blocks;

[0054] The volume density and color of the second sampling points in the sub-block determined based on the third distance are merged to obtain the target volume density and target color of the second sampling points located in the overlapping area.

[0055] In a third aspect, the present application further provides a scene reconstruction model training device, the device comprising:

[0056] A first eigenvalue determination module, used to determine each first sampling point corresponding to each calibration image, and determine a first eigenvalue corresponding to each first sampling point, wherein each calibration image is obtained by performing image acquisition on a scene to be reconstructed;

[0057] A first weight determination module, used to determine the posture of a camera corresponding to each of the multiple perspectives of the scene, and determine first target perspectives corresponding to first rendering perspectives and first weights corresponding to the first target perspectives based on the postures of the cameras corresponding to the perspectives;

[0058] a first appearance embedding determination module, configured to perform first spatial clustering for each of the first rendering perspectives based on the first weights and the appearance embeddings corresponding to the first target perspectives to obtain an appearance embedding corresponding to the first rendering perspective, wherein the appearance embedding represents a feature of the corresponding perspective image related to ambient lighting and / or brightness;

[0059] A training module is used to perform model training based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalues ​​corresponding to each of the first sampling points to obtain a scene reconstruction model.

[0060] In a fourth aspect, the present application further provides a scene reconstruction device, the device comprising:

[0061] A second eigenvalue determination module, used to determine each second sampling point corresponding to the image to be processed, and determine a second eigenvalue corresponding to each second sampling point, wherein the image to be processed is obtained by performing image acquisition on the scene to be reconstructed;

[0062] A second weight determination module, used to determine the posture of a camera corresponding to each of the multiple perspectives of the scene to be reconstructed, and determine each second target perspective corresponding to each second rendering perspective based on the posture of the camera corresponding to each of the perspectives, and a second weight corresponding to each of the second target perspectives;

[0063] a second appearance embedding determination module, configured to perform first spatial clustering for each second rendering perspective based on each second weight and the appearance embedding corresponding to each second target perspective to obtain an appearance embedding corresponding to the second rendering perspective, wherein the appearance embedding represents a feature of the corresponding perspective image related to ambient lighting and / or brightness;

[0064] A reconstruction module is used to obtain a rendered image based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each of the second sampling points, and a scene reconstruction model.

[0065] In a fifth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above-mentioned embodiments when executing the computer program.

[0066] In a sixth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method in any one of the above-mentioned embodiments.

[0067] In a seventh aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method in any one of the above-mentioned embodiments when executed by a processor.

[0068] The scene reconstruction model training method, scene reconstruction method, device, computer equipment, computer-readable storage medium and computer program product determine each first sampling point corresponding to each calibration image, and determine the first eigenvalue corresponding to each first sampling point. Each calibration image is obtained by performing image acquisition on the scene to be reconstructed, and then the position and posture of the camera corresponding to each of the multiple perspectives of the scene are determined, and based on the position and posture of the camera corresponding to each of the perspectives, each first target perspective corresponding to each first rendering perspective and the first weight corresponding to each first target perspective are determined. For each of the first rendering perspectives, based on the first weights and the appearance embedding corresponding to each of the first target perspectives, spatial clustering is performed to obtain the appearance embedding corresponding to the first rendering perspective and the first eigenvalue corresponding to each of the first sampling points. The appearance embedding represents the features of the corresponding perspective image related to the ambient illumination and / or brightness. In this way, the model is guided to learn the features related to the local illumination and / or brightness through spatial clustering, taking into account the local consistency of the appearance embedding in the geometric space, and improving the robustness to illumination and / or brightness changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 A schematic diagram of a flow chart of a scene reconstruction model training method in one embodiment;

[0071] Figure 2 It is an appearance embedding comparison diagram in one embodiment;

[0072] Figure 3 A schematic diagram of model training in one embodiment;

[0073] Figure 4 is a flow chart of a scene reconstruction method in one embodiment;

[0074] Figure 5 It is a structural block diagram of a scene reconstruction model training device in one embodiment;

[0075] Figure 6 is a structural block diagram of a scene reconstruction device in one embodiment;

[0076] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0078] In one embodiment, Figure 1 As shown, a scene reconstruction model training method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0079] S102: Determine each first sampling point corresponding to each calibration image, and determine a first eigenvalue corresponding to each first sampling point, wherein each calibration image is obtained by performing image acquisition on a scene to be reconstructed.

[0080] Each calibration image is obtained by collecting images of a scene to be reconstructed. The calibration image is an image used for model training. The color of each pixel point is calibrated in the calibration image, and optionally only the color of each sampled pixel point is calibrated.

[0081] The first sampling point is determined based on the calibration image. For each calibration image, the calibration image can be sampled to obtain various sampling pixel points, and then the sampling pixel points are used as the origin to extend along the direction of the light to obtain rays, and sampling is performed on each ray to obtain a number of first sampling points.

[0082] The first eigenvalue corresponding to the first sampling point can be obtained based on the position of each first sampling point. In some optional embodiments, determining the first eigenvalue corresponding to each first sampling point includes: based on a hash table, querying the first eigenvalue represented by a hash grid corresponding to each first sampling point, wherein each hash grid corresponds to a sub-block, and each hash grid stores the information of the calibration image of the corresponding sub-block in the corresponding hash table. Specifically, the hash table gives a mapping relationship between the position of each first sampling point and the first eigenvalue, that is, taking the position of the first sampling point as an index, the corresponding value is the first eigenvalue.

[0083] In the above embodiment, the first sampling point is stored in an easily accessible data structure such as a hash table through a hash grid, and the query cost that requires hundreds of floating point operations is reduced to one query operation, thereby reducing the training time.

[0084] S104: Determine the posture of the camera corresponding to each of the multiple perspectives of the scene, and determine first target perspectives corresponding to first rendering perspectives and first weights corresponding to first target perspectives based on the postures of the cameras corresponding to the perspectives.

[0085] S106: For each first rendering perspective, spatial clustering is performed based on each first weight and the appearance embedding corresponding to each first target perspective to obtain the appearance embedding corresponding to the first rendering perspective, where the appearance embedding represents features of the corresponding perspective image related to ambient lighting and / or brightness.

[0086] In large outdoor scenes, cameras often produce distortions such as overexposure and shadows due to the nonlinear lighting in the environment. Appearance Embedding represents the features of the corresponding perspective image that are related to the ambient lighting and / or brightness. Appearance Embedding is often used exclusively as the input of the color network, providing the model with the ability to flexibly adjust the scene radiation brightness in a specific image. However, the relevant technology does not consider the local consistency of appearance embedding in geometric space, resulting in sudden changes in lighting when using appearance embedding in practice. In this embodiment, appearance embedding is regarded as compensation for the nonlinear lighting fitted by the color network, which is related to the nonlinear light source position of the scene. During training, the appearance embedding exhibits clustering characteristics in space, which makes the transition smoother when rendering new perspective images in this embodiment. The specific limitations are as follows.

[0087] In the embodiment of the present application, when using appearance embedding, the appearance embedding of the perspective near the first rendering perspective position (i.e., the first target perspective) is weighted, and then the weighted appearance embedding is input into the neural network to generate the corresponding color under the first rendering perspective.

[0088] Specifically, each viewing angle corresponds to a camera. Assume that the set of all cameras in the scene is P = {P1, ..., P N}, P i =[R i |m i ] is the camera pose, where R i is the posture of camera i, m i is the position of camera i.

[0089] Each perspective is used as a first rendering perspective, and each first target perspective corresponding to the first rendering perspective is determined. Optionally, each first target perspective corresponding to the first rendering perspective can be determined based on the position and posture of the camera corresponding to each perspective, for example, k perspectives closest to the first rendering perspective are selected as each first target perspective. And the weight corresponding to each first target perspective is obtained. Specifically, the weight corresponding to each first target perspective can be determined based on the distance between each first target perspective and the first rendering perspective. In other embodiments, the weight corresponding to each first target perspective can also be determined based on other parameters. For example, a set of sorting weights can be pre-set, and then the first target perspectives are sorted based on the distance between each first target perspective and the first rendering perspective. In this way, the corresponding weights are assigned to the sorted first target perspectives based on the sorting weights, and so on. The method for obtaining the weight corresponding to the first target perspective is not specifically limited here.

[0090] In one of the optional embodiments, first target perspectives corresponding to first rendering perspectives and first weights corresponding to first target perspectives are determined based on the posture of the camera corresponding to each perspective, including: obtaining a first distance between the camera corresponding to each first rendering perspective and cameras corresponding to other perspectives based on the posture of the camera corresponding to each perspective; based on the first distance, selecting k other perspectives as first target perspectives for each first rendering perspective, where k is a positive integer greater than 0; and determining a first weight corresponding to each first target perspective based on the first distance.

[0091] Since the viewing angle includes position and attitude (ie, normal) information, the first distance can be determined by the following formula:

[0092]

[0093] Among them, λ RT is a hyperparameter, d(P i , P a ) is the distance between camera i and camera a, For the matrix Trace of.

[0094] Based on the first distance, selecting k other perspectives as the first target perspective for each first rendering perspective may include: firstly sorting the perspectives based on the first distance, and then selecting the first k other perspectives as the first target perspective, which may be specifically determined by the following formula:

[0095] S k (j) = sort({d(P i , P a )} a＝1，......N )[1:k] where sort indicates sorting the first distance, N indicates the number of viewing angles, S k(j) is the first distance between the selected first target perspective k and the first rendering perspective j.

[0096] The first weight is determined based on the first distance, for example, the inverse of the first distance, that is, the first weight corresponding to each first target viewing angle k is

[0097] Therefore, for each first rendering perspective, spatial clustering is performed based on the first weights and the appearance embeddings corresponding to the first target perspectives to obtain the appearance embedding corresponding to the first rendering perspective, which may include weighting the first weights and the appearance embeddings corresponding to the first target perspectives to obtain the appearance embedding corresponding to the first rendering perspective, which may be specifically expressed by the following formula:

[0098]

[0099] Among them, l j represents the appearance embedding corresponding to the jth first rendering perspective, represents the appearance embedding corresponding to the i-th first target view, where 0 <i≤k。

[0100] In the above embodiment, the appearance embedding presents the clustering characteristics in space during training, which makes the present application have a smoother transition when rendering new perspective images. Figure 2 shown.

[0101] in Figure 2 This is a comparison between the appearance embedding based on spatial clustering and the original appearance embedding proposed in the embodiment of the present application. The leftmost column of images represents rendering using only the most recent appearance embedding, the middle column of images represents rendering using the appearance embedding with spatial clustering, and the rightmost column of images represents the true value. The first row of images is a visualization of the appearance embedding, and the second row of images is the rendered scene image / true value. It can be seen from the first row and first column of images that only using the most recent appearance embedding for training will result in a larger variance in the appearance embedding distribution, resulting in an image with a large difference in lighting from the true value during the rendering stage. The first row and second column of images show that after using the appearance embedding with spatial clustering, the variance of the appearance embedding distribution can be made smaller, smoother when switching between different perspectives, and closer to the true value.

[0102] S108: Performing model training based on the appearance embedding corresponding to each first rendering perspective and the first eigenvalue corresponding to each first sampling point to obtain a scene reconstruction model.

[0103] After obtaining the appearance embeddings corresponding to each first rendering perspective, the scene reconstruction model can be trained. The model is trained based on the appearance embeddings corresponding to each first rendering perspective and the first eigenvalues ​​corresponding to each first sampling point to obtain the scene reconstruction model. For example, the appearance embeddings corresponding to each first rendering perspective and the first eigenvalues ​​corresponding to each first sampling point are used as the input of the model, and then the output of the model is obtained. The color of each corresponding pixel point in the calibration image is calculated based on the output of the model, and the scene reconstruction model is trained through supervision through the calibrated colors in the calibration image.

[0104] The scene reconstruction model training method determines each first sampling point corresponding to each calibration image, and determines the first eigenvalue corresponding to each first sampling point. Each calibration image is obtained by performing image acquisition on the scene to be reconstructed, and then the position and posture of the camera corresponding to each of the multiple perspectives of the scene are determined, and the first target perspectives corresponding to each first rendering perspective are determined based on the position and posture of the camera corresponding to each perspective, and the first weights corresponding to each first target perspective are determined. For each first rendering perspective, based on the first weights and the appearance embedding corresponding to each first target perspective, spatial clustering is performed to obtain the appearance embedding corresponding to the first rendering perspective and the first eigenvalue corresponding to each first sampling point. The appearance embedding represents the features of the corresponding perspective image related to ambient illumination and / or brightness. In this way, the model is guided to learn features related to local illumination and / or brightness through spatial clustering, taking into account the local consistency of the appearance embedding in the geometric space, thereby improving the robustness to changes in illumination and / or brightness.

[0105] In one of the optional embodiments, before determining each first sampling point corresponding to each calibration image, the method includes: obtaining parameters of each camera and a first sparse point cloud corresponding to the scene, and determining a first space occupied by the scene based on the parameters of each camera and the first sparse point cloud corresponding to the scene; dividing the first space into a plurality of sub-blocks, and determining each calibration image corresponding to each sub-block; determining each first sampling point corresponding to each calibration image, including: for each sub-block, determining each first sampling point corresponding to each calibration image; performing model training based on the appearance embedding corresponding to each first rendering perspective and the first eigenvalue corresponding to each first sampling point to obtain a scene reconstruction model, including: for each sub-block, performing neural network model training based on the appearance embedding corresponding to each first rendering perspective and the first eigenvalue corresponding to each first sampling point to obtain a scene reconstruction model corresponding to each sub-block.

[0106] Among them, for large scenes, since the space involved is large, the first space involved in the large scene is divided into several sub-blocks in the present application, and for each sub-block, a scene reconstruction model is trained separately, thereby reducing the complexity of the model, and several sub-blocks can be trained in parallel, thereby shortening the training time and improving training efficiency.

[0107] The first space occupied by the corresponding scene can be obtained based on the parameters of each camera and the first sparse point cloud corresponding to the scene. For example, the first sparse point clouds are projected to the target coordinate system, such as the world coordinate system, based on the parameters of each camera, and then the projected first sparse point clouds are spliced ​​to obtain the point cloud corresponding to the scene. In this way, the scope of the scene can be determined based on the point cloud corresponding to the scene, that is, the first space occupied by the scene is obtained. The first sparse point clouds can be obtained by back-projecting the images collected by each camera combined with the depth information, which is not specifically limited here.

[0108] The rule for dividing the first space may be predetermined. For example, for a first space of size E×F, the number of sub-blocks to be divided may be determined, for example, e×f. Based on the size of the first space and the number of sub-blocks, the size of each sub-block may be calculated, that is, In this way, the first space is divided based on the size of each sub-block. In this embodiment, a rule for uniform space division is given. In other embodiments, an uneven division rule can also be adopted, that is, the size of each sub-block obtained by division can be different. For example, the size of each sub-block can be determined based on the amount of information in the sub-block. No specific limitation is made here.

[0109] In this application, uniform division is used as an example to illustrate that the first space is divided into a number of rectangular sub-blocks:

[0110] (O i , S i )∈R 3×2

[0111] Among them, O i represents the center of mass of the cuboid sub-block, S i Indicates the side length of the cuboid sub-block.

[0112] Subsequently, each calibration image is divided based on the spatial range corresponding to each sub-block, that is, the calibration image is divided into multiple groups according to the spatial range corresponding to the sub-block, and each group corresponds to one sub-block.

[0113] In this way, the calibration images are divided into different training sets, and each training set is trained to obtain a scene reconstruction model corresponding to a sub-block.

[0114] During the training process of each sub-block, for each calibration image, the first sampling points corresponding to the calibration image are collected, and then the camera posture corresponding to each perspective is determined, and then the step of determining the first target perspectives corresponding to the first rendering perspectives based on the camera posture corresponding to each perspective is continued. Finally, the model is trained based on the appearance embedding corresponding to each first rendering perspective and the first eigenvalue corresponding to each first sampling point to obtain a scene reconstruction model. In this way, scene reconstruction models corresponding to multiple sub-blocks can be trained.

[0115] It should be noted that the scene reconstruction model corresponding to each of the above sub-blocks can be trained in parallel, thereby improving the training efficiency.

[0116] In the above embodiments, as the scene scale increases, the capacity of the neural network is difficult to carry the scene details and content that grow exponentially with the scale. Although deepening or widening the neural network helps to alleviate the capacity limitation, it will also cause a significant increase in training costs. In this application, block reconstruction is used to effectively reduce training costs and adapt to scale growth.

[0117] In one of the optional embodiments, after determining each first sampling point based on each calibration image, it also includes: determining an initial bounded area based on a first space occupied by the scene; when the first sampling point is not within the initial bounded area, nonlinearly mapping the first sampling point to a difference area between a target bounded area and the initial bounded area, the initial bounded area being a subset of the target bounded area.

[0118] In large outdoor scenes, there may be situations where cameras observe distant spaces. For example, some cameras may observe tall buildings or the sky in the distance. When the NeRF algorithm samples these spaces in the scene, the first sampling point sampled will be too large in scale to be conducive to the neural network and hash grid learning scene. Therefore, it is necessary to perform nonlinear mapping on these first sampling points to shrink the first sampling points to a bounded area.

[0119] Specifically, the initial bounded area is determined based on the first space occupied by the scene, which can be specifically combined with Figure 3 As shown, Figure 3 The small and medium cuboids are the initial bounded areas determined based on the first space occupied by the scene. When the first sampling point is within the initial bounded area, the position of the first sampling point is directly used. When the first sampling point is not within the initial bounded area, the first sampling point is nonlinearly mapped to the difference area between the target bounded area and the initial bounded area. The initial bounded area is a subset of the target bounded area, and the target bounded area is Figure 3 The area corresponding to the large rectangular block.

[0120] The nonlinear mapping may be of any scale, for example, the scales of the mappings along different directions of the sub-block may be different, so that any first sampling point outside the rectangular sub-block may be mapped to the difference set region, which can better adapt to the rectangular shaped scene.

[0121] In one of the optional embodiments, please continue to combine Figure 3 As shown, model training is performed based on the appearance embedding corresponding to each first rendering perspective and the first eigenvalue corresponding to each first sampling point to obtain a scene reconstruction model, including: inputting the first eigenvalue corresponding to each first sampling point into the first neural network to obtain the model body density and model features of each first sampling point; obtaining the model color corresponding to each first sampling point based on the model features of each first sampling point, the direction of each first sampling point and the appearance embedding corresponding to each first rendering perspective; synthesizing the model body density and model color of each first sampling point to obtain the model color of the corresponding pixel point on the calibration image plane; and performing model training based on the real color of the pixel point in the calibration image and the model color to obtain the scene reconstruction model.

[0122] in, Figure 3 The scene is divided into 4 sub-blocks. In other embodiments, the number of sub-blocks can be other values, which are not specifically limited here.

[0123] For each sub-block, background shrinkage is first performed, that is, the first sampling points that are not in the initial bounded area are mapped to the difference set area; then the first eigenvalue of each first sampling point in the target bounded area is represented by a hash grid, and the first eigenvalue of each first sampling point is input into the first neural network to obtain the model volume density and model features of each first sampling point, and the model features, the directions of the first sampling points and the appearance embeddings of each perspective are vector spliced, and then the spliced ​​vectors are input into the second neural network to obtain the model color, wherein the appearance embeddings of each perspective can be processed in the manner described above to obtain the appearance embeddings corresponding to each first rendering perspective, so that there is a smoother transition when rendering the new perspective image.

[0124] The specific representation of the model can be:

[0125] σ=MLP n (x)

[0126]

[0127] where n = arg i |xO i |<S i ,

[0128]

[0129] Where n = arg i |xO i |<S i is the number of the sub-block where the first sampling point is located, σ is the model body density, that is, based on the first neural network model corresponding to the nth sub-block, MLP n (x) indicates that the first neural network model processes the first eigenvalue of the first sampling point, x is the first sampling point, d is the direction of the first sampling point. c is the model color, which is obtained based on the second neural network model corresponding to the nth sub-block. It indicates that the second neural network model processes the first eigenvalue of the first sampling point, the direction of the first sampling point, and the appearance embedding of each viewing angle.

[0130] Each neural network model can include a two-layer MLP network, which can reduce the training time by one to two orders of magnitude.

[0131] Subsequently, the model volume density and the model color of each first sampling point are synthesized to obtain the model color of the corresponding pixel point on the calibration image plane. Specifically, the following formula can be used:

[0132]

[0133] Among them, σ i is the first sampling point x i The volume density, d i To observe the first sampling point x i direction, c i is the color of the first sampling point, α j =1-exp(σ i δ i ) is the first sampling point x i The opacity, δ i is the distance between adjacent first sampling points, t i is the first sampling point x i Distance to the camera, T i is the transmittance, which can be regarded as the energy of the pixel gradually dissipating as it moves along the light. i is the intermediate variable, is the model color of pixel r, is the depth of pixel r.

[0134] The calculated model color is compared with the real color of the pixel in the calibration image, and the model is trained based on the comparison result, so that the scene reconstruction model corresponding to the sub-block can be trained.

[0135] In one embodiment, Figure 4As shown, a scene reconstruction method is provided. This embodiment uses the method applied to a terminal with electronic data processing capabilities as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0136] S402: Determine each second sampling point corresponding to the image to be processed, and determine a second eigenvalue corresponding to each second sampling point, wherein the image to be processed is obtained by performing image acquisition on the scene to be reconstructed.

[0137] The method for obtaining the second sampling point may refer to the first sampling point mentioned above, and the method for obtaining the second eigenvalue may refer to the first eigenvalue mentioned above, which will not be described in detail here.

[0138] S404: Determine the posture of the camera corresponding to each of the multiple perspectives of the scene to be reconstructed, and determine each second target perspective corresponding to each second rendering perspective and a second weight corresponding to each second target perspective based on the posture of the camera corresponding to each perspective.

[0139] Among them, the limitations of the second rendering perspective, the second target perspective and the second weight can also refer to the first rendering perspective, the first target perspective and the first weight mentioned above, which will not be repeated here.

[0140] S406: For each second rendering perspective, spatial clustering is performed based on the second weights and the appearance embeddings corresponding to the second target perspectives to obtain the appearance embeddings corresponding to the second rendering perspective, where the appearance embeddings represent features of the corresponding perspective image related to ambient lighting and / or brightness.

[0141] The limitation on the appearance embedding corresponding to the second rendering perspective may also refer to the limitation on the appearance embedding corresponding to the first rendering perspective mentioned above, which will not be described in detail here.

[0142] S408: Obtain a rendered image based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point, and the scene reconstruction model.

[0143] Among them, in this application, based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point and the scene reconstruction model, the volume density and color corresponding to each second sampling point are obtained, and then the rendered image corresponding to the second rendering perspective is obtained based on the volume density and color synthesis corresponding to each second sampling point. The formula for obtaining the color and depth of the corresponding pixel based on the volume density and color synthesis of the second sampling points can be referred to the formula for obtaining the model color and depth of the corresponding pixel point on the calibrated image plane based on the model volume density and model color synthesis mentioned above, which will not be repeated here.

[0144] In one of the optional embodiments, the method also includes: obtaining a number of sub-blocks obtained by dividing the scene to be reconstructed, and determining the image to be processed corresponding to each sub-block; obtaining a rendered image based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point, and the scene reconstruction model, including: obtaining the scene reconstruction model of each sub-block; based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point, and the scene reconstruction model of the corresponding sub-block, respectively processing to obtain the volume density and color of the number of second sampling points corresponding to each sub-block; obtaining the rendered image based on the volume density and color of the number of second sampling points corresponding to each sub-block.

[0145] Among them, in the present application, a scene reconstruction model is trained for each sub-block separately, so the image to be processed corresponding to each sub-block can be processed separately, for example, the corresponding image to be processed is processed by the scene reconstruction model of the corresponding sub-block, wherein the processing of the scene reconstruction models of each sub-block can be parallel, thereby improving the processing efficiency.

[0146] Can still be combined Figure 3 As shown, for each sub-block corresponding to the image to be processed, the second sampling point is first obtained, and then the background is shrunk, and the second eigenvalue of each second sampling point is obtained through the hash grid, and the second eigenvalue is input into the trained first neural network to obtain the volume density and model features of the second sampling point, and then the model features, the direction of the second sampling point and the appearance embedding of each perspective are vector spliced, and finally the spliced ​​vector is input into the trained second neural network to obtain the color of the second sampling point. The second neural network can perform spatial clustering based on the appearance embedding of each perspective to obtain the appearance embedding of the second rendering perspective, thereby making the image smoother.

[0147] In one of the optional embodiments, a rendered image is obtained based on the volume density and color of a number of second sampling points corresponding to each sub-block, including: determining a target volume density and a target color of the second sampling points based on a positional relationship between the second sampling points and the sub-blocks; and obtaining a rendered image based on the target volume density and the target color of each second sampling point.

[0148] Since the volume density and color of the second sampling point of each sub-block are calculated separately, the results of each sub-block need to be merged later to obtain the target volume density and target color of each second sampling point.

[0149] Optionally, based on the positional relationship between the second sampling point and the sub-block, determining the target volume density and target color of the second sampling point includes: when the second sampling point is located in the sub-block, determining the target volume density and target color of the second sampling point based on the corresponding sub-block; when the second sampling point is located outside the sub-block, determining a second distance between the second sampling point and each sub-block, and selecting the volume density and color of the second sampling point in the corresponding sub-block based on the second distance as the target volume density and target color of the second sampling point located outside the sub-block; and obtaining a rendered image based on the volume density and color of each second sampling point.

[0150] When the second sampling point falls within a single sub-block, the volume density and color of the second sampling point in the sub-block are used as the target volume density and target color of the second sampling point.

[0151] In the case where the second sampling point is outside each sub-block, the volume density and color of the second sampling point output by the sub-block closest to the second sampling point are selected as the target volume density and target color of the second sampling point. Specifically, the second distance between the second sampling point and each sub-block can be obtained, for example, the distance from the second sampling point to the centroid of each sub-block is used as the second distance, and then the sub-block that meets the second distance requirement, for example, the sub-block with the smallest second distance is selected as the target sub-block, and the volume density and color of the second sampling point in the target sub-block are obtained as the target volume density and target color of the second sampling point located outside the sub-block.

[0152] In one of the optional embodiments, after obtaining a plurality of sub-blocks obtained by dividing the scene to be reconstructed, the method further includes: expanding the area of ​​each sub-block; determining the target volume density and target color of the second sampling point based on the positional relationship between the second sampling point and the sub-block, including: when the second sampling point is located in an overlapping area of ​​the sub-blocks, determining the sub-block corresponding to the overlapping area; obtaining a third distance between the second sampling point located in the overlapping area and each determined sub-block; and fusing the volume density and color of the second sampling point in the determined sub-block based on the third distance to obtain the target volume density and target color of the second sampling point located in the overlapping area.

[0153] In which, in the block division stage, the area of ​​each sub-block is expanded, so that there is overlap between the sub-blocks. In the fusion stage, the sub-block where each second sampling point is located is queried, and the target volume density and target color of the second sampling point are obtained through the corresponding sub-block, so that when the second sampling point falls on the overlapping area of ​​the sub-block, the sub-block corresponding to the overlapping area can be determined; the third distance between the second sampling point located in the overlapping area and each determined sub-block is obtained, such as the third distance between the second sampling point and the center of mass of the determined sub-block, and then the second weight corresponding to each sub-block is determined based on the third distance, for example, the inverse of the third distance can be used as the second weight of the sub-block, and finally weighted based on the second weight and the volume density and color of the corresponding second sampling point in each sub-block to obtain the target volume density and target color of the second sampling point located in the overlapping area.

[0154] In the above embodiment, the appearance embedding based on spatial clustering fully considers the local consistency of the appearance embedding in the geometric space, thereby making the rendered image smoother, and the sub-block processing can improve the processing efficiency. Finally, the reconstruction of the large scene is synthesized by fusion of sub-blocks, which not only improves the efficiency but also ensures the accuracy.

[0155] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0156] Based on the same inventive concept, the embodiment of the present application also provides a scene reconstruction model training device and a scene reconstruction device for implementing the scene reconstruction model training method and scene reconstruction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more scene reconstruction model training devices and scene reconstruction device embodiments provided below can refer to the limitations of the scene reconstruction model training method and scene reconstruction method above, and will not be repeated here.

[0157] In an exemplary embodiment, Figure 5 As shown, a scene reconstruction model training device is provided, comprising: a first feature value determination module 501, a first weight determination module 502, a first appearance embedding determination module 503 and a training module 504, wherein:

[0158] A first eigenvalue determination module 501 is used to determine each first sampling point corresponding to each calibration image, and determine a first eigenvalue corresponding to each first sampling point, wherein each calibration image is obtained by performing image acquisition on a scene to be reconstructed;

[0159] A first weight determination module 502 is used to determine the position and posture of a camera corresponding to each of the multiple perspectives of the scene, and determine first target perspectives corresponding to first rendering perspectives and first weights corresponding to the first target perspectives based on the position and posture of the camera corresponding to the perspectives;

[0160] A first appearance embedding determination module 503 is configured to perform first spatial clustering for each first rendering perspective based on each first weight and the appearance embedding corresponding to each first target perspective to obtain an appearance embedding corresponding to the first rendering perspective, wherein the appearance embedding represents a feature of the corresponding perspective image related to ambient illumination and / or brightness;

[0161] The training module 504 is used to perform model training based on the appearance embedding corresponding to each first rendering perspective and the first eigenvalue corresponding to each first sampling point to obtain a scene reconstruction model.

[0162] In one of the optional embodiments, the above-mentioned first weight determination module 502 is specifically used to obtain a first distance between a camera corresponding to each first rendering perspective and cameras corresponding to other perspectives based on the posture of the camera corresponding to each perspective; based on the first distance, select k other perspectives as first target perspectives for each first rendering perspective, where k is a positive integer greater than 0; and determine a first weight corresponding to each first target perspective based on the first distance.

[0163] In one of the optional embodiments, the apparatus further includes: a block division module, configured to obtain parameters of each camera and a first sparse point cloud corresponding to the scene, and determine a first space occupied by the scene based on the parameters of each camera and the first sparse point cloud corresponding to the scene; divide the first space into a plurality of sub-blocks, and determine calibration images corresponding to each sub-block;

[0164] The first eigenvalue determination module 501 is specifically used to determine, for each sub-block, each first sampling point corresponding to each calibration image;

[0165] The training module 504 is specifically used to perform neural network model training for each sub-block based on the appearance embedding corresponding to each first rendering perspective and the first eigenvalue corresponding to each first sampling point to obtain a scene reconstruction model corresponding to each sub-block.

[0166] In one of the optional embodiments, the above-mentioned first eigenvalue determination module 501 is specifically used to query the first eigenvalue represented by the hash grid corresponding to each first sampling point based on the hash table, wherein each hash grid corresponds to a sub-block, and each hash grid stores the information of the calibration image of the corresponding sub-block in the corresponding hash table.

[0167] In one of the optional embodiments, the above-mentioned device also includes: a background shrinkage module, used to determine an initial bounded area based on a first space occupied by the scene; when the first sampling point is not within the initial bounded area, the first sampling point is nonlinearly mapped to a difference area between the target bounded area and the initial bounded area, and the initial bounded area is a subset of the target bounded area.

[0168] In one of the optional embodiments, the above-mentioned training module 504 is specifically used to input the first eigenvalue corresponding to each first sampling point into the first neural network to obtain the model body density and model features of each first sampling point; based on the model features of each first sampling point, the direction of each first sampling point and the appearance embedding corresponding to each first rendering perspective, obtain the model color corresponding to each first sampling point; synthesize the model body density and model color of each first sampling point to obtain the model color of the corresponding pixel point on the calibration image plane; based on the real color and model color of the pixel point in the calibration image, perform model training to obtain a scene reconstruction model.

[0169] In an exemplary embodiment, Figure 6 As shown, a scene reconstruction device is provided, comprising: a second feature value determination module 601, a second weight determination module 602, a second appearance embedding determination module 603 and a reconstruction module 604, wherein:

[0170] A second eigenvalue determination module 601 is used to determine each second sampling point corresponding to the image to be processed, and determine the second eigenvalue corresponding to each second sampling point, wherein the image to be processed is obtained by performing image acquisition on the scene to be reconstructed;

[0171] A second weight determination module 602 is used to determine the position and posture of a camera corresponding to each of the multiple perspectives of the scene to be reconstructed, and determine each second target perspective corresponding to each second rendering perspective based on the position and posture of the camera corresponding to each perspective, and a second weight corresponding to each second target perspective;

[0172] A second appearance embedding determination module 603 is configured to perform first spatial clustering for each second rendering perspective based on each second weight and the appearance embedding corresponding to each second target perspective to obtain an appearance embedding corresponding to the second rendering perspective, wherein the appearance embedding represents a feature of the corresponding perspective image related to ambient illumination and / or brightness;

[0173] The reconstruction module 604 is used to obtain a rendered image based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point, and the scene reconstruction model.

[0174] In one of the optional embodiments, the apparatus further comprises: a to-be-processed image determination module, configured to obtain a plurality of sub-blocks obtained by dividing the scene to be reconstructed, and determine an to-be-processed image corresponding to each sub-block;

[0175] The reconstruction module 604 is specifically used to obtain a scene reconstruction model for each sub-block; based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point, and the corresponding scene reconstruction model of the sub-block, respectively process to obtain the volume density and color of several second sampling points corresponding to each sub-block; based on the volume density and color of several second sampling points corresponding to each sub-block, obtain a rendered image.

[0176] In one of the optional embodiments, the reconstruction module 604 is specifically used to determine the target volume density and target color of the second sampling point based on the positional relationship between the second sampling point and the sub-block; and obtain a rendered image based on the target volume density and target color of each second sampling point.

[0177] In one of the optional embodiments, the reconstruction module 604 is specifically used to determine the target volume density and target color of the second sampling point based on the corresponding sub-block when the second sampling point is located within the sub-block; determine the second distance between the second sampling point and each sub-block when the second sampling point is located outside each sub-block, and select the volume density and color of the second sampling point in the corresponding sub-block based on the second distance as the target volume density and target color of the second sampling point located outside the sub-block; and obtain a rendered image based on the volume density and color of each second sampling point.

[0178] In one of the optional embodiments, the above-mentioned device also includes: an area expansion module, which is used to expand the area of ​​each sub-block; the above-mentioned reconstruction module 604 is specifically used to determine the sub-block corresponding to the overlapping area when the second sampling point is located in the overlapping area of ​​the sub-block; obtain the third distance between the second sampling point located in the overlapping area and each determined sub-block; based on the third distance, the volume density and color of the second sampling point in the determined sub-block are merged to obtain the target volume density and target color of the second sampling point located in the overlapping area.

[0179] Each module in the scene reconstruction model training method and scene reconstruction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0180] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a scene reconstruction model training method and a scene reconstruction method are implemented.

[0181] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0182] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0183] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0184] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0185] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A scene reconstruction model training method, characterized in that: The method comprises: Determine each first sampling point corresponding to each calibration image, and determine a first eigenvalue corresponding to each first sampling point, wherein each calibration image is obtained by performing image acquisition on a scene to be reconstructed; Determine a position and posture of a camera corresponding to each of a plurality of perspectives of a scene, and determine first target perspectives corresponding to first rendering perspectives and first weights corresponding to first target perspectives based on the position and posture of the camera corresponding to each of the perspectives; For each of the first rendering perspectives, spatial clustering is performed based on the first weights and the appearance embeddings corresponding to the first target perspectives to obtain an appearance embedding corresponding to the first rendering perspective, where the appearance embedding represents features of the corresponding perspective image related to ambient lighting and / or brightness; Model training is performed based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalues ​​corresponding to each of the first sampling points to obtain a scene reconstruction model.

2. The method according to claim 1, characterized in that The determining, based on the position and posture of the camera corresponding to each of the perspectives, each of the first target perspectives corresponding to each of the first rendering perspectives, and each of the first weights corresponding to each of the first target perspectives, includes: Based on the position and posture of the camera corresponding to each of the perspectives, obtaining a first distance between the camera corresponding to each of the first rendering perspectives and cameras corresponding to other perspectives; Based on the first distance, selecting k other perspectives as first target perspectives for each of the first rendering perspectives, where k is a positive integer greater than 0; A first weight corresponding to each first target viewing angle is determined based on the first distance.

3. The method according to claim 1 or 2, characterized in that: Before determining each first sampling point corresponding to each calibration image, the method includes: Acquire parameters of each camera and a first sparse point cloud corresponding to the scene, and determine a first space occupied by the scene based on the parameters of each camera and the first sparse point cloud corresponding to the scene; Dividing the first space into a plurality of sub-blocks, and determining calibration images corresponding to each sub-block; The determining of each first sampling point corresponding to each calibration image includes: For each sub-block, determining each first sampling point corresponding to each calibration image; The performing model training based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalue corresponding to each of the first sampling points to obtain a scene reconstruction model includes: For each sub-block, based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalues ​​corresponding to each of the first sampling points, a neural network model training is performed to obtain a scene reconstruction model corresponding to each sub-block.

4. The method according to claim 3, characterized in that: The determining the first eigenvalue corresponding to each of the first sampling points includes: Based on the hash table, the first eigenvalue represented by the hash grid corresponding to each of the first sampling points is queried, wherein each of the hash grids corresponds to one of the sub-blocks, and each hash grid stores the information of the calibration image of the corresponding sub-block in the corresponding hash table.

5. The method according to claim 3, characterized in that: After determining each first sampling point based on each calibration image, the method further includes: Determining an initial bounded area based on a first space occupied by the scene; In the case that the first sampling point is not within the initial bounded region, the first sampling point is nonlinearly mapped to a difference region between a target bounded region and the initial bounded region, and the initial bounded region is a subset of the target bounded region.

6. The method according to claim 1, characterized in that The performing model training based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalue corresponding to each of the first sampling points to obtain a scene reconstruction model includes: Inputting the first characteristic value corresponding to each of the first sampling points into the first neural network to obtain the model volume density and model characteristics of each of the first sampling points; Obtaining a model color corresponding to each of the first sampling points based on the model features of each of the first sampling points, the directions of each of the first sampling points, and the appearance embedding corresponding to each of the first rendering perspectives; The model volume density and the model color of each of the first sampling points are synthesized to obtain the model color of the corresponding pixel point on the calibration image plane; Based on the real colors of the pixels in the calibration image and the model colors, model training is performed to obtain a scene reconstruction model.

7. A scene reconstruction method, characterized in that: The method comprises: Determine each second sampling point corresponding to the image to be processed, and determine a second eigenvalue corresponding to each second sampling point, wherein the image to be processed is obtained by performing image acquisition on the scene to be reconstructed; Determine the posture of a camera corresponding to each of the multiple perspectives of the scene to be reconstructed, and determine second target perspectives corresponding to each second rendering perspective based on the posture of the camera corresponding to each of the perspectives, and second weights corresponding to each of the second target perspectives; For each second rendering perspective, spatial clustering is performed based on each second weight and the appearance embedding corresponding to each second target perspective to obtain an appearance embedding corresponding to the second rendering perspective, where the appearance embedding represents a feature of the corresponding perspective image related to ambient lighting and / or brightness; A rendered image is obtained based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each of the second sampling points, and a scene reconstruction model.

8. The method according to claim 7, characterized in that The method further comprises: Obtain a number of sub-blocks obtained by dividing the scene to be reconstructed, and determine the image to be processed corresponding to each sub-block; The obtaining of a rendered image based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each of the second sampling points, and the scene reconstruction model includes: Obtaining a scene reconstruction model of each of the sub-blocks; Based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each second sampling point, and the scene reconstruction model of the corresponding sub-block, respectively process to obtain the volume density and color of a plurality of second sampling points corresponding to each sub-block; A rendered image is obtained based on the volume density and color of a number of second sampling points corresponding to each sub-block.

9. The method according to claim 8, characterized in that The step of obtaining a rendered image based on the volume density and color of a plurality of second sampling points corresponding to each sub-block includes: Determine the target volume density and target color of the second sampling point based on the positional relationship between the second sampling point and the sub-block; A rendered image is obtained based on the target volume density and the target color of each of the second sampling points.

10. The method according to claim 9, characterized in that The determining, based on the positional relationship between the second sampling point and the sub-block, the target volume density and the target color of the second sampling point includes: In the case where the second sampling point is located within the sub-block, determining the target volume density and target color of the second sampling point based on the corresponding sub-block; In the case where the second sampling point is located outside each of the sub-blocks, determining a second distance between the second sampling point and each of the sub-blocks, and selecting a volume density and a color of the second sampling point in the corresponding sub-block based on the second distance as a target volume density and a target color of the second sampling point located outside the sub-block; A rendered image is obtained based on the volume density and color of each of the second sampling points.

11. The method according to claim 9, characterized in that After obtaining a plurality of sub-blocks obtained by dividing the scene to be reconstructed, the method further includes: Expanding the area of ​​each of the sub-blocks; The determining, based on the positional relationship between the second sampling point and the sub-block, the target volume density and the target color of the second sampling point includes: When the second sampling point is located in an overlapping area of ​​the sub-blocks, determining a sub-block corresponding to the overlapping area; Acquire a third distance between the second sampling point located in the overlapping area and each of the determined sub-blocks; The volume density and color of the second sampling points in the sub-block determined based on the third distance are merged to obtain the target volume density and target color of the second sampling points located in the overlapping area.

12. A scene reconstruction model training device, characterized in that: The device comprises: A first eigenvalue determination module, used to determine each first sampling point corresponding to each calibration image, and determine a first eigenvalue corresponding to each first sampling point, wherein each calibration image is obtained by performing image acquisition on a scene to be reconstructed; A first weight determination module, used to determine the posture of a camera corresponding to each of the multiple perspectives of the scene, and determine first target perspectives corresponding to first rendering perspectives and first weights corresponding to the first target perspectives based on the postures of the cameras corresponding to the perspectives; a first appearance embedding determination module, configured to perform first spatial clustering for each of the first rendering perspectives based on the first weights and the appearance embeddings corresponding to the first target perspectives to obtain an appearance embedding corresponding to the first rendering perspective, wherein the appearance embedding represents a feature of the corresponding perspective image related to ambient lighting and / or brightness; A training module is used to perform model training based on the appearance embedding corresponding to each of the first rendering perspectives and the first eigenvalues ​​corresponding to each of the first sampling points to obtain a scene reconstruction model.

13. A scene reconstruction device, characterized in that: The device comprises: A second eigenvalue determination module, used to determine each second sampling point corresponding to the image to be processed, and determine a second eigenvalue corresponding to each second sampling point, wherein the image to be processed is obtained by performing image acquisition on the scene to be reconstructed; A second weight determination module, used to determine the posture of a camera corresponding to each of the multiple perspectives of the scene to be reconstructed, and determine each second target perspective corresponding to each second rendering perspective based on the posture of the camera corresponding to each of the perspectives, and a second weight corresponding to each of the second target perspectives; a second appearance embedding determination module, configured to perform first spatial clustering for each second rendering perspective based on each second weight and the appearance embedding corresponding to each second target perspective to obtain an appearance embedding corresponding to the second rendering perspective, wherein the appearance embedding represents a feature of the corresponding perspective image related to ambient lighting and / or brightness; A reconstruction module is used to obtain a rendered image based on the appearance embedding corresponding to the second rendering perspective, the second eigenvalues ​​corresponding to each of the second sampling points, and a scene reconstruction model.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images