Three-dimensional reconstruction model construction method and three-dimensional reconstruction method

Abstract

The embodiment of the application provides a three-dimensional reconstruction model construction method and a three-dimensional reconstruction method, and relates to the technical field of image processing. Gradient significant sampling point sets and gradient non-significant sampling point sets in a two-dimensional sample image are obtained by gradient sampling on the two-dimensional sample image. The gradient significant sampling point sets include a plurality of pixel points in the two-dimensional sample image, the gradient amplitude of each pixel point is greater than or equal to the gradient amplitude average value and the difference between the gradient amplitude and the gradient amplitude average value is greater than a first preset threshold. The gradient non-significant sampling point sets include a plurality of pixel points in the two-dimensional sample image, the gradient amplitude of each pixel point is greater than the gradient amplitude average value and the difference between the gradient amplitude and the gradient amplitude average value is less than the first preset threshold or less than or equal to the gradient amplitude average value. A three-dimensional reconstruction model is trained based on the gradient significant sampling point sets and the gradient non-significant sampling point sets. The stability in the model training process is improved, the edge and wrinkle information of an object is fully learned, and the accuracy of the reconstructed three-dimensional model is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically, to a method for constructing a three-dimensional reconstruction model and a three-dimensional reconstruction method. Background Technology

[0002] 3D reconstruction is a method of restoring and reconstructing a 3D model from a 2D image sequence in a virtual world; it is essentially the inverse operation of a camera. In this process, to achieve precise reconstruction of the 3D model, the color features from the 2D image sequence must also be rendered onto the 3D model.

[0003] In existing technologies, a deep neural network capable of color rendering of a 3D model is obtained by randomly sampling on a 2D image and training a deep neural network with the color information obtained from each sampling point.

[0004] However, the reconstruction quality of deep neural networks trained by this random sampling method is low. The reconstructed 3D model tends to be too smooth, losing the edge and wrinkle information described in the 2D image. Summary of the Invention

[0005] The purpose of this application includes, for example, providing a method for constructing a three-dimensional reconstruction model and a three-dimensional reconstruction method, which can train a three-dimensional reconstruction model based on gradient information in a set of gradient salient sampling points and a set of gradient insignificant sampling points, improve the stability of the model training process, fully learn the edge information and wrinkle information of objects in two-dimensional sample images, and improve the accuracy of the reconstructed three-dimensional model.

[0006] The embodiments of this application can be implemented as follows:

[0007] In a first aspect, embodiments of this application provide a method for constructing a three-dimensional reconstruction model, the method comprising:

[0008] Gradient sampling is performed on a two-dimensional sample image to obtain a set of salient gradient sampling points and a set of insignificant gradient sampling points in the two-dimensional sample image. The set of salient gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is greater than or equal to a first preset threshold. The set of insignificant gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is less than the first preset threshold, or less than or equal to the average gradient magnitude. The average gradient magnitude is the average gradient magnitude of each pixel in the two-dimensional sample image.

[0009] A 3D reconstruction model is trained based on the set of salient gradient sampling points and the set of insignificant gradient sampling points.

[0010] Secondly, embodiments of this application provide a three-dimensional reconstruction method, the method comprising:

[0011] The three-dimensional uniform sampling point set within the local space surrounding the object is input into the three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model, wherein the three-dimensional reconstruction model is constructed based on the three-dimensional reconstruction model construction method described in any one of the first aspects.

[0012] Thirdly, embodiments of this application provide a three-dimensional reconstruction model construction apparatus, comprising:

[0013] The model generation module is used to perform gradient sampling on a two-dimensional sample image to obtain a set of salient gradient sampling points and a set of insignificant gradient sampling points in the two-dimensional sample image. The set of salient gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is greater than or equal to a first preset threshold. The set of insignificant gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is less than or equal to the first preset threshold, or less than the average gradient magnitude. The average gradient magnitude is the average gradient magnitude of each pixel in the two-dimensional sample image.

[0014] The model training module is used to train a 3D reconstruction model based on the set of gradient salient sampling points and the set of gradient insignificant sampling points.

[0015] Fourthly, embodiments of this application also provide a three-dimensional reconstruction apparatus, comprising:

[0016] The model generation module is used to input a set of three-dimensional uniform sampling points in the local space surrounding the object into the three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model. The three-dimensional reconstruction model is constructed based on the three-dimensional reconstruction model construction method described in any one of the first aspects.

[0017] Fifthly, embodiments of this application provide a processing device, the processing device comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, and when the processing device is running, the processor communicates with the storage medium via the bus, the processor executing the machine-readable instructions to perform the steps of the three-dimensional reconstruction model construction method as described in any one of the first aspects or the three-dimensional reconstruction method as described in the second aspect.

[0018] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the three-dimensional reconstruction model construction method as described in any one of the first aspects or the three-dimensional reconstruction method as described in the second aspect.

[0019] The beneficial effects of the embodiments of this application include:

[0020] The 3D reconstruction model construction method and 3D reconstruction method provided in this application can be used to train a 3D reconstruction model based on the gradient salient sampling point set and the gradient non-salient sampling point set obtained after gradient sampling of 2D sample images. This incorporates the gradient information contained in the gradient salient and gradient non-salient sampling point sets during training, improving the stability of the training process, fully learning the edge and wrinkle information of objects in the 2D sample images, and improving the accuracy of the reconstructed 3D model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the steps of a three-dimensional reconstruction model construction method provided in an embodiment of this application.

[0023] Figure 2 A schematic diagram illustrating the steps of determining sampling points in a three-dimensional reconstruction model construction method provided in this application embodiment;

[0024] Figure 3 This is a flowchart illustrating another sampling point determination method for a three-dimensional reconstruction model construction method provided in this application embodiment;

[0025] Figure 4 A flowchart illustrating the steps of determining the mask matrix in a three-dimensional reconstruction model construction method provided in this application embodiment;

[0026] Figure 5 This application provides a schematic diagram of the steps for selecting sampling points in a method for constructing a three-dimensional reconstruction model.

[0027] Figure 6 This is a schematic diagram illustrating another step in the screening of sampling points for a three-dimensional reconstruction model construction method provided in this application embodiment;

[0028] Figure 7 A schematic diagram illustrating the steps of training a model in a method for constructing a three-dimensional reconstruction model provided in this application embodiment;

[0029] Figure 8 This is a schematic diagram of another step in the training model of a three-dimensional reconstruction model construction method provided in an embodiment of this application;

[0030] Figure 9 A schematic diagram illustrating the steps of a three-dimensional reconstruction model construction method provided in this application embodiment;

[0031] Figure 10 This application provides a schematic diagram of the steps for determining rendering colors in a three-dimensional reconstruction model construction method according to an embodiment of the present application.

[0032] Figure 11 This is a flowchart illustrating the steps of a three-dimensional reconstruction method provided in an embodiment of this application;

[0033] Figure 12 This is a schematic diagram of the structure of a three-dimensional reconstruction model construction device provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application;

[0035] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0036] Icons: 110 - 3D reconstruction model building device; 1101 - Point set acquisition module; 1102 - Model training module; 120 - 3D reconstruction device; 1201 - Model generation module; 2001 - Processor; 2002 - Storage medium. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0038] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0039] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0040] 3D reconstruction refers to the process of reconstructing a 3D model from images of a single view or multiple views. In existing technologies, after the deep neural network is constructed, random sampling is required on the input 2D image to obtain multiple random sampling points. Then, color features are extracted from each random sampling point. Based on the extracted color information, a deep neural network capable of color rendering of the 3D model is trained.

[0041] In existing technologies, training with multiple randomly sampled points can generate colored 3D models. However, when the gradient information contained in each random sampled point is distributed in extreme ways, such as when all random sampled points are located in the background or solid color regions, it may lead to instability and difficulty in convergence during the training process of the deep neural network. Furthermore, since the sampled points are randomly generated, the edges and wrinkles in the 2D image are not considered, resulting in an overly smooth overall 3D model that loses the edge and wrinkle information described in the 2D image.

[0042] Based on this, the applicant, through research, proposed a method for constructing a 3D reconstruction model and a 3D reconstruction method. By performing gradient sampling on 2D sample images, a set of salient and insignificant gradient sampling points containing gradient information is obtained. The 3D reconstruction model is then trained based on these two point sets. This improves the stability of the training process, fully learns the edge and wrinkle information of objects in the 2D sample images, and enhances the accuracy of the reconstructed 3D model.

[0043] 3D reconstruction can generate a 3D geometric model of a given 3D object from a sequence of 2D images acquired from multiple viewpoints and the corresponding camera poses. For further refined reconstruction, the color information of the 3D geometric model's surface can be restored based on the color differences of the object under different lighting conditions, resulting in a colored 3D geometric model.

[0044] To accurately obtain edge and wrinkle information describing objects in two-dimensional image sequences, this application provides a method for constructing a three-dimensional reconstruction model and a three-dimensional reconstruction method. During the training of the three-dimensional reconstruction model, a set of salient gradient sampling points and a set of non-salient gradient sampling points are used as training datasets. This fully utilizes the edge and wrinkle information of objects contained in the two-dimensional image sequences to obtain a three-dimensional reconstruction model capable of reconstructing and rendering three-dimensional geometric models.

[0045] The following explanation, using several specific application examples, illustrates the three-dimensional reconstruction model construction method and the three-dimensional reconstruction method provided in the embodiments of this application.

[0046] Figure 1 The diagram shown is a flowchart illustrating the steps of a three-dimensional reconstruction model construction method provided in an embodiment of this application. The execution subject of this method can be a computer device with computing and processing capabilities. Figure 1 As shown, the method includes the following steps:

[0047] S101, perform gradient sampling on the two-dimensional sample image to obtain the set of gradient significant sampling points and the set of gradient non-significant sampling points in the two-dimensional sample image.

[0048] The gradient saliency sampling point set includes multiple pixels in a two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is greater than or equal to a first preset threshold.

[0049] The gradient non-significant sampling point set includes multiple pixels in a two-dimensional sample image, where the average gradient magnitude of each pixel is greater than the gradient magnitude and the difference between the average gradient magnitude and the average gradient magnitude is less than a first preset threshold, or less than or equal to the average gradient magnitude.

[0050] The average gradient magnitude is the average gradient magnitude of all pixels in the two-dimensional sample image. It should be noted that the saliency gradient sampling point set may include multiple pixels with large gradient magnitudes in the two-dimensional sample image, while the insignificant gradient sampling point set may include multiple pixels with small gradient magnitudes in the two-dimensional sample image. Each pixel is still located in the two-dimensional sample image, and its membership to the point set is identified by its gradient magnitude.

[0051] Gradient magnitude can be used to describe the gradient information of a two-dimensional sample image. When there are edges or wrinkles in a two-dimensional sample image, the gray value changes more and the gradient magnitude is also larger. For smoother parts of a two-dimensional sample image, the gray value changes less and the gradient magnitude is correspondingly smaller.

[0052] Therefore, in this embodiment, gradient sampling can be used to select multiple pixels with large gradient magnitudes from the two-dimensional sample image as a set of salient gradient sampling points, and multiple pixels with small gradient magnitudes as a set of insignificant gradient sampling points. In this way, multiple pixels in the set of salient gradient sampling points are distributed near the edges and wrinkles in the two-dimensional sample image, while the set of insignificant gradient sampling points is distributed in other relatively smooth areas, ensuring that the trained three-dimensional reconstruction model can learn more features about edges and wrinkles in the two-dimensional sample image.

[0053] Optionally, to clearly define the gradient difference between the set of salient gradient sampling points and the set of insignificant gradient sampling points, and considering the processor's processing capabilities, after determining the gradient magnitude and the mean gradient magnitude of the two-dimensional sample image, a subset of pixels in the two-dimensional sample image whose gradient magnitude is greater than the mean gradient magnitude and whose difference from the mean gradient magnitude is greater than or equal to a first preset threshold can be designated as the set of salient gradient sampling points. Conversely, a subset of pixels that do not meet the above conditions can be designated as the set of insignificant gradient sampling points. The first preset threshold can be set as needed, and this application does not limit its application.

[0054] For example, if the average gradient magnitude is 2 and the first preset threshold is 1, if the average gradient magnitude of a pixel is 4 (i.e., greater than the gradient magnitude and the difference between the average and the gradient magnitude is greater than the first preset threshold), then the pixel is a pixel in the set of gradient significant sampling points. If the average gradient magnitude of a pixel is 2.5 (i.e., greater than the gradient magnitude and the difference between the average and the gradient magnitude is less than the first preset threshold), then the pixel is a pixel in the set of gradient insignificant sampling points.

[0055] S102, a 3D reconstruction model is trained based on the gradient significant sampling point set and the gradient non-significant sampling point set.

[0056] It is understandable that each pixel in the gradient saliency sampling point set contains gradient information about edges and wrinkles in the two-dimensional sample image, while each pixel in the gradient insignificance sampling point set contains gradient information that can describe the smoother parts of the two-dimensional image.

[0057] Therefore, the training data for training the 3D reconstruction model contains a stable set of gradient salient sampling points, ensuring that the 3D reconstruction model learns enough gradient information about edges and wrinkles in the 2D sample images.

[0058] In this embodiment, a 3D reconstruction model is trained based on a set of gradient salient sampling points and a set of gradient insignificant sampling points. Compared with random sampling, this ensures that the gradient information of edges and wrinkles in the 2D sample images can be considered during the training process of the 3D reconstruction model, thereby improving the reconstruction accuracy of the 3D reconstruction model.

[0059] Optionally, such as Figure 2 As shown, in step S101 above, gradient sampling is performed on the two-dimensional sample image to obtain the set of gradient salient sampling points and the set of gradient non-salient sampling points in the two-dimensional sample image, which can be achieved by the following steps S201 to S203.

[0060] S201, determine the gradient magnitude of each pixel in the two-dimensional sample image.

[0061] Alternatively, the gradient magnitude of the two-dimensional sample image can be calculated in the following manner:

[0062] First, the two-dimensional sample image can be converted into an initial grayscale image. To reduce the edge blurring of the initial grayscale image, the Sobel operator can be used to further process the initial grayscale image. The grayscale values ​​of the four neighboring regions of each pixel in the initial grayscale image are weighted and differ to obtain the sample grayscale image.

[0063] Then, determine the pixels in the sample grayscale image. Horizontal gradient value and vertical gradient value And further determine the pixel points according to the following formula. gradient magnitude :

[0064]

[0065] Among them, pixels Let be any pixel in the sample grayscale image. The gradient magnitude of each pixel in the sample grayscale image can be determined using the method described above.

[0066] S202, Based on the gradient magnitude of each pixel in the two-dimensional sample image, determine the mean gradient magnitude and standard deviation of the gradient magnitude of the two-dimensional sample image.

[0067] gradient magnitude It can be determined by the following formula:

[0068]

[0069] Gradient magnitude standard deviation It can be determined by the following formula:

[0070]

[0071] It is understandable that the gradient magnitude and the standard deviation of the gradient magnitude correspond to the two-dimensional sample image, and the aforementioned gradient magnitude corresponds to each pixel in the two-dimensional sample image.

[0072] S203. Based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, gradient sampling is performed on the two-dimensional sample image to obtain a set of significant gradient sampling points and a set of insignificant gradient sampling points.

[0073] Optionally, the first preset threshold can be set as the standard deviation of the gradient magnitude. By comparing the difference between the gradient magnitude of each pixel and the mean gradient magnitude of the corresponding two-dimensional sample image, and the relationship with the standard deviation of the gradient magnitude, some or all pixels in the two-dimensional sample image with larger gradient magnitudes (i.e., containing information about edge and wrinkle gradients) can be selected as the set of gradient salient sampling points. Simultaneously, some or all pixels in the two-dimensional sample image with smaller gradient magnitudes (containing smooth portions) can be selected as the set of gradient insignificant sampling points.

[0074] In this embodiment, the gradient saliency sampling point set and the gradient non-saliency sampling point set are determined by the gradient magnitude, the gradient magnitude mean, and the gradient magnitude standard deviation, ensuring the difference between the gradient information contained in the gradient saliency sampling points and the gradient non-saliency sampling points.

[0075] Optionally, such as Figure 3 As shown, in step S203 above, gradient sampling is performed on the two-dimensional sample image based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, to obtain a set of significant gradient sampling points and a set of insignificant gradient sampling points, which can be achieved by the following steps S301 to S302.

[0076] S301, based on the gradient magnitude, mean gradient magnitude, and standard deviation of each pixel, determine the set of salient gradient points, the set of non-salient gradient points, and the mask matrix corresponding to the two-dimensional sample image.

[0077] The mask matrix can be the same size as the two-dimensional sample image, and can be represented by 0 or 1 to indicate whether each pixel is a gradient salient point.

[0078] Optionally, based on the gradient magnitude, mean gradient magnitude, and standard deviation of each pixel, it can be determined whether each pixel in the two-dimensional sample image satisfies the gradient saliency condition, thereby dividing all pixels in the two-dimensional sample image into a gradient saliency set and a non-saliency set. The pixels in the gradient saliency set cover most of the edge and wrinkled regions in the two-dimensional sample image, and their gradient magnitudes are relatively large. The pixels in the non-saliency set cover the smooth regions in the two-dimensional sample image, and their gradient magnitudes are relatively small.

[0079] S302, based on the set of salient gradient points, the set of insignificant gradient points, and the mask matrix, determine the set of salient gradient sampling points and the set of insignificant gradient sampling points.

[0080] Optionally, if the processor has sufficient load capacity, the number of pixels in the gradient saliency sampling point set can be equal to the number of pixels in the gradient saliency point set, and the number of pixels in the gradient non-saliency sampling point set can also be equal to the number of pixels in the non-saliency point set.

[0081] Conversely, if the number of pixels in the gradient salient sampling point set is less than the number of pixels in the gradient salient point set, or the number of pixels in the gradient non-salient sampling point set is less than the number of pixels in the non-salient point set, then the above mask matrix can be used to select some pixels from the gradient salient point set as the gradient salient sampling point set, and select some pixels from the above non-salient point set as the gradient non-salient sampling point set.

[0082] It is understandable that, in order to ensure the quantity of gradient saliency sampling points, the gradient saliency sampling point set and the gradient non-saliency sampling point set can be obtained by sampling the gradient saliency point set and the non-saliency point set in proportion.

[0083] In this embodiment, gradient saliency sampling point set and gradient non-saliency sampling point set are obtained by filtering from the gradient saliency point set and the gradient non-saliency point set, which fully considers the processor's processing capability and improves the efficiency of the method.

[0084] Optionally, such as Figure 4 As shown, in step S301 above, the set of salient gradient points, the set of non-salient gradient points, and the mask matrix corresponding to the two-dimensional sample image are determined based on the gradient magnitude, the mean gradient magnitude, and the standard deviation of the gradient magnitude of each pixel. This can be achieved by the following steps S401 to S404.

[0085] S401, determine the difference between the gradient magnitude of each pixel and the average gradient magnitude.

[0086] After determining the gradient magnitude and average gradient magnitude of each pixel in steps S201 to S202 of the above embodiment, the gradient magnitude of each pixel can be subtracted from the average gradient magnitude of the two-dimensional sample image where the pixel is located to determine the difference between the gradient magnitude of each pixel and the average gradient magnitude.

[0087] S402, if the gradient magnitude of the first pixel is greater than the mean gradient magnitude, and the difference between the first pixel and the mean gradient magnitude is greater than or equal to the standard deviation of the gradient magnitude, then the first pixel is taken as a pixel in the gradient saliency set, and the gradient saliency mask of the first pixel is marked as saliency.

[0088] The first pixel is any pixel among the pixels in the two-dimensional sample image.

[0089] In a two-dimensional sample image, if the difference between the gradient magnitude of the first pixel and the mean gradient magnitude is greater than or equal to one standard deviation of the gradient magnitude, then the first pixel is determined to be a gradient salient point and is assigned to the gradient salient point set.

[0090] Furthermore, in the above embodiments, the number of rows and columns of the mask matrix are equal to the number of rows and columns of pixels in the two-dimensional sample image, respectively. Therefore, by setting the value of a certain row and a certain column of the mask matrix, it is possible to mark whether the pixels in the same row and the same column of the two-dimensional sample image are gradient salient points.

[0091] Therefore, after determining that the first pixel is a gradient saliency point, the gradient saliency mask value corresponding to the position of the first pixel in the mask matrix can be set to 1, that is, marked as saliency.

[0092] S403, if the gradient magnitude of the second pixel is greater than the mean gradient magnitude and the difference between the gradient magnitude and the mean gradient magnitude is less than the standard deviation of the gradient magnitude, or is less than or equal to the mean gradient magnitude, then the second pixel is taken as a pixel in the set of gradient non-significant points, and the gradient saliency mask of the second pixel is marked as non-significant.

[0093] The second pixel is any pixel in the two-dimensional sample image.

[0094] For a second pixel that does not meet the above conditions for a significant gradient point, i.e., the gradient magnitude is greater than the mean gradient magnitude but the difference is less than 1 standard deviation, or the gradient magnitude is less than or equal to the mean gradient magnitude, the second pixel can be determined as a non-significant gradient point.

[0095] Furthermore, after determining that the second pixel is a gradient non-significant point, the gradient saliency mask value corresponding to the position of the second pixel in the matrix can be set to 0, that is, marked as non-significant.

[0096] It is understandable that for each pixel in a two-dimensional sample image, the above steps S402-S403 can be used sequentially to make judgments until all pixels in the two-dimensional sample image are divided into either a set of gradient saliency points or a set of gradient insignificance points.

[0097] S404 generates a mask matrix based on the gradient saliency mask markers of each pixel in the two-dimensional sample image.

[0098] After performing the gradient saliency point determination process on each pixel in the two-dimensional sample image, each pixel is marked at its corresponding position in the aforementioned matrix. The marked matrix is ​​then used as a mask matrix.

[0099] In this embodiment, each pixel in the two-dimensional sample image is divided into a set of salient gradient points and a set of insignificant gradient points based on the difference in gradient magnitude. The difference between salient and insignificant gradient points is clearly defined to ensure that enough gradient information can be retained during the training of the three-dimensional reconstruction model, thereby improving the accuracy of the three-dimensional reconstruction model.

[0100] Optionally, such as Figure 5 As shown, in step S302 above, determining the gradient salient sampling point set and the gradient non-salient sampling point set based on the gradient salient point set, the non-salient point set, and the mask matrix can be achieved by the following steps S501 to S502.

[0101] S501, determine the proportion of gradient salient points based on the ratio of the number of pixels in the gradient salient point set to the number of pixels in the two-dimensional sample image.

[0102] Optionally, the percentage of gradient saliency points can be set. The value is the ratio of gradient salient points to all pixels in the 2D sample image. It can be understood that the total number of pixels in the 2D sample image is the sum of the number of pixels in the gradient salient point set and the number of pixels in the gradient non-salient point set.

[0103] S502, according to the preset sampling quantity, mask matrix and gradient saliency ratio, the pixels in the gradient saliency set and the pixels in the non-saliency set are filtered to obtain the gradient saliency sampling point set and the gradient non-saliency sampling point set.

[0104] Preset sampling quantity The number is determined based on the processor's processing power; for example, it can be 512.

[0105] Based on the above sampling quantity and the markers indicating whether a point is a gradient saliency as indicated by the mask matrix, the pixels in the gradient saliency set and the non-saliency set obtained in the above steps can be further randomly selected to determine the gradient saliency sampling point set and the gradient non-saliency sampling point set, respectively.

[0106] Optionally, if the number of samples If the number of pixels is equal to that in a two-dimensional sample image, then the set of gradient salient points can be directly used as the set of gradient salient sampling points, and the set of insignificant points can be used as the set of gradient insignificant sampling points, based on the markings in the mask matrix.

[0107] If the number of samples If the number of pixels is greater than the number of pixels in a two-dimensional sample image, then the proportion of significant gradient points can be used as a reference. By filtering the gradient salient point set and the gradient insignificant point set in the same proportion, we obtained the gradient salient sampling point set and the gradient insignificant sampling point set.

[0108] In this embodiment, according to the preset sampling quantity, mask matrix, and gradient saliency ratio, a set of gradient saliency sampling points and a set of gradient insignificance sampling points are obtained. This ensures that a certain proportion of pixels in the data used to train the 3D reconstruction model contain gradient information about wrinkles and edges from the 2D sample images, so that the 3D reconstruction model can learn more features about edges and wrinkles.

[0109] Optionally, such as Figure 6 As shown, in step S502 above, the pixels in the gradient salient point set and the non-salient point set are filtered according to the preset sampling quantity, mask matrix and gradient salient point ratio to obtain the gradient salient sampling point set and the gradient non-salient sampling point set, which can be achieved by the following steps S601 to S603.

[0110] S601, determine the number of gradient significant points and the number of gradient non-significant points based on the preset number of samples and the proportion of gradient significant points.

[0111] Among them, the sum of the number of gradient salient points sampled and the number of gradient non-salient points sampled is equal to the preset number of samples, and the ratio of the number of gradient salient points sampled to the preset number of samples is equal to the proportion of gradient salient points.

[0112] From the above embodiments, the number of samples is Set the number of gradient salient points to be sampled. The number of gradient non-significant points sampled is Then we have: .

[0113] Furthermore, to avoid filtering out too many significant gradient points during the screening process, it is possible to make That is, maintaining the number of gradient significant points sampled relative to the preset number of samples. The ratio is equal to the ratio of gradient saliency points to all pixels in the two-dimensional sample image, so as to maintain the number of gradient saliency points.

[0114] S602, according to the number of gradient salient points sampled and the mask matrix, the pixels in the gradient salient point set are filtered to obtain the gradient salient sampling point set.

[0115] In the above embodiments, the mask matrix is ​​used to mark whether each pixel is a gradient saliency point. When filtering the gradient saliency point set, the location of each pixel in the two-dimensional sample image belonging to the gradient saliency point set can be determined according to the marking of the mask matrix, that is, the matrix value marked as saliency by the gradient saliency mask. Thus, each pixel in the gradient saliency point set is sampled randomly or at a preset interval, and multiple pixels marked as saliency by the gradient saliency mask are obtained, which constitute the gradient saliency sampling point set.

[0116] S603, according to the number of gradient non-salient points sampled and the mask matrix, the pixels in the non-salient point set are filtered to obtain the gradient non-salient sampling point set.

[0117] Similar to the above method for filtering gradient saliency sampling point sets, when filtering gradient non-saliency sampling point sets, the location of each pixel in the two-dimensional sample image belonging to the non-saliency point set can be determined according to the label of the mask matrix, that is, the matrix value marked as non-saliency by the gradient saliency mask. Then, each pixel in the non-saliency point set is sampled randomly or at a preset interval, resulting in multiple pixels marked as non-saliency by the gradient saliency mask, which constitute the gradient non-saliency sampling point set.

[0118] In this embodiment, by filtering the set of gradient salient points and the set of gradient insignificant points based on the proportion of gradient salient points, the set of gradient salient sampling points and the set of gradient insignificant sampling points are determined, ensuring the number of pixels containing gradient information, thereby ensuring the accuracy of the 3D reconstruction model.

[0119] Optionally, such as Figure 7 As shown, in step S102 above, a three-dimensional reconstruction model is trained based on the gradient significant sampling point set and the gradient non-significant sampling point set, which can be achieved by the following steps S701 to S702.

[0120] S701, based on the set of salient gradient sampling points, the set of insignificant gradient sampling points, the two-dimensional sample image, and the camera pose of the two-dimensional sample image, determine the three-dimensional ray direction corresponding to each salient gradient sampling point and each insignificant gradient sampling point, as well as the three-dimensional sampling points on the corresponding ray.

[0121] Camera pose can be represented by the camera's rotation and translation parameters, which describe the angle at which the camera captures a two-dimensional sample image.

[0122] The 3D ray direction of a certain pixel can be understood as the direction of the camera's line of sight to that pixel in the current camera pose, or the direction of the light emitted by the camera to that pixel.

[0123] Optionally, based on the camera pose and the surrounding pixels of each gradient salient and gradient insignificant sampling point in the two-dimensional sample image, the relative positional relationship between each sampling point and the camera can be determined, further determining the three-dimensional ray direction corresponding to each gradient salient and gradient insignificant sampling point, and the three-dimensional sampling point on the corresponding ray. The specific algorithm for determining the three-dimensional ray direction is not limited herein.

[0124] S702, input the set of significant gradient sampling points, the set of insignificant gradient sampling points, each significant gradient sampling point, the three-dimensional ray direction corresponding to each insignificant gradient sampling point, and the three-dimensional sampling points on the corresponding ray into the initial three-dimensional reconstruction model, train the initial three-dimensional reconstruction model, and obtain the three-dimensional reconstruction model.

[0125] The three-dimensional ray direction corresponding to each sampling point determined in the above steps can describe the viewing direction information of that pixel. It can be used to input the viewing direction information, together with the gradient significant sampling point set and the gradient non-significant sampling point set, into the initial model of three-dimensional reconstruction to obtain the spatial position information, opacity information and color attribute information of each sampling point.

[0126] Then, based on the information obtained above, the network parameters of the initial 3D reconstruction model are corrected until the difference between the information output by the initial 3D reconstruction model and the actual information is less than a preset threshold, and the initial 3D reconstruction model is used as the 3D reconstruction model.

[0127] It should be noted that a preset number of iterations can be set for the initial 3D reconstruction model. Training is performed based on the aforementioned gradient salient sampling point set and gradient insignificant sampling point set. When the preset number of iterations is reached... At this point, you can switch to a completely random sampling method to obtain sampling points and continue training. Alternatively, you can use a gradient-significant sampling method throughout the entire training period; this application does not impose any specific limitations on this.

[0128] In this embodiment, the initial model for 3D reconstruction is trained based on the set of significant gradient sampling points, the set of insignificant gradient sampling points, the direction of the 3D ray, and the 3D sampling points on the corresponding ray to obtain the 3D reconstruction model. This improves the 3D reconstruction model's ability to learn wrinkles and edges in 2D sample images and enhances the accuracy of the 3D reconstruction model.

[0129] Optionally, the initial model for 3D reconstruction includes: an implicit surface initial neural network and a neural rendering initial network.

[0130] like Figure 8 As shown, in step S702 above, the set of gradient significant sampling points, the set of gradient non-significant sampling points, each gradient significant sampling point, the three-dimensional ray direction corresponding to each gradient non-significant sampling point, and the three-dimensional sampling points on the corresponding ray are input into the three-dimensional reconstruction initial model. The three-dimensional reconstruction initial model is trained to obtain the three-dimensional reconstruction model, which can be achieved by the following steps S801 to S804.

[0131] S801 inputs the three-dimensional sampling points on the corresponding rays of the gradient significant sampling point set and the gradient insignificant sampling point set into the implicit surface initial neural network to obtain the symbolic distance function value, normal vector and spatial feature encoding corresponding to each sampling point.

[0132] The Signed Distance Function (SDF) is an implicit representation of the surface of a 3D object. It is defined by determining the distance from a point on a finite region in space to the object's surface and simultaneously defining the sign of the distance: positive if the point is inside the object, negative if it is outside, and 0 if it is on the surface. Therefore, the SDF can be used to describe the continuous surface information of a reconstructed 3D training model.

[0133] Normal vectors and spatial feature encoding can be used to describe the height and depth information of the surface reconstructed from each sampling point.

[0134] Based on the symbolic distance function value, normal vector, and spatial feature encoding output by the aforementioned implicit surface initial neural network, a smooth implicit surface can be reconstructed, which is the three-dimensional surface plot of the three-dimensional training model described below.

[0135] S802, the three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set, the sign distance function value of each three-dimensional sampling point, the normal vector, the spatial feature encoding, and the three-dimensional ray direction corresponding to each gradient salient sampling point and each gradient insignificant sampling point are respectively input into the neural rendering initial network to obtain the sampling color value corresponding to each three-dimensional sampling point.

[0136] The neural rendering initial network can be used for color rendering of each pixel. Based on the three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set, the signed distance function value, normal vector, spatial feature encoding of each sampling point, and the three-dimensional ray direction of each sampling point, the visual attributes of each sampling point, such as color information, are restored.

[0137] S803 determines the 3D training model and the rendering color value of each 2D pixel in the 3D training model based on the sampled color value of each 3D sampling point and the corresponding signed distance function value.

[0138] Furthermore, since the above steps only determine the sampled color values ​​of each sampling point, in order to obtain the color values ​​of each sampling point on the 3D training model, the sampled color values ​​of each sampling point can be further rendered based on the sampled color values ​​of each sampling point output by the initial neural rendering network and the signed distance function values ​​corresponding to each sampling point output by the aforementioned implicit surface initial neural network, thereby determining the rendering color values ​​of the corresponding pixels on the implicit surface established by each sampling point.

[0139] S804 determines the loss information of the initial 3D reconstruction model based on the 3D training model and the rendering color value of each pixel in the 3D training model, and iteratively corrects the initial 3D reconstruction model according to the loss information until the corrected initial 3D reconstruction model meets the preset conditions, and uses the initial 3D reconstruction model that meets the preset conditions as the 3D reconstruction model.

[0140] Optionally, the loss information can be a loss function determined based on the Mean Absolute Error (MAE) between the rendered color value of each pixel and the true color of each pixel. This loss function can be used to describe the similarity between the generated rendered color value and the true color. It is understood that the smaller the loss function, the better the reconstruction effect.

[0141] During the training of the initial model for 3D reconstruction, since the rendering color value of each pixel is determined by the sampling color value of each sampling point and the symbolic distance function value, the network parameters of the above-mentioned implicit surface initial neural network and neural rendering initial network can be iteratively updated by gradient descent according to the loss function. Optionally, when the loss function value is less than a preset threshold, the training can be considered to be completed, and a 3D reconstruction model composed of the trained implicit surface initial neural network and neural rendering initial network is obtained.

[0142] In this embodiment, the initial neural network for implicit surface and the initial neural rendering network are corrected using loss information, thereby determining the 3D reconstruction model and improving the accuracy of 3D reconstruction.

[0143] See Figure 9 The process of constructing a three-dimensional reconstruction model in this application will be described using the above embodiments:

[0144] First, based on the input two-dimensional sample image, determine the gradient magnitude of each pixel in the two-dimensional sample image, and then determine the mean gradient magnitude and the standard deviation of the gradient magnitude based on the gradient magnitude of each pixel.

[0145] Furthermore, based on the mean and standard deviation of the gradient magnitude, each pixel in the two-dimensional sample image is divided into a set of salient gradient points and a set of non-salient gradient points. On this basis, the mask matrix and the proportion of salient gradient points in the two-dimensional sample image are determined.

[0146] Then, based on the mask matrix and the proportion of salient gradient points, the salient and insignificant gradient point sets are further filtered to determine the salient gradient sampling point set and the insignificant gradient sampling point set.

[0147] Based on the aforementioned set of salient and insignificant gradient sampling points, the 3D ray direction corresponding to each sampling point and the corresponding 3D sampling point on the ray are determined. These 3D ray directions and sampling points are then input into the implicit surface initialization neural network to obtain the signed distance function value, normal vector, and spatial feature encoding. Further, the 3D sampling points, signed distance function value, normal vector, spatial feature encoding, and the corresponding 3D ray direction are input into the neural rendering initialization network to obtain the sampled color value of each sampling point.

[0148] Finally, based on the symbolic distance function value and the sampled color value of each sampling point, the rendering color value of each pixel in the reconstructed 3D training model is determined. Based on the above rendering color values, the loss information is determined, and the network parameters of the above implicit surface initial neural network and neural rendering initial network are corrected to obtain the 3D reconstruction model.

[0149] Optionally, such as Figure 10 As shown, in step S803 above, the three-dimensional training model and the rendering color value of each two-dimensional pixel in the three-dimensional training model are determined based on the sampled color value of each three-dimensional sampling point and the corresponding symbolic distance function value. This can be achieved by the following steps S901 to S903.

[0150] S901 converts the symbolic distance function value of each three-dimensional sampling point into a volume density value.

[0151] For the continuous surface of the 3D training model, given hyperparameters Sampling points SDF function value at the location Convert to volume density It can be determined by the following formula:

[0152]

[0153] S902 determines the rendering weight of each three-dimensional sampling point based on the volume density value of each sampling point.

[0154] Volume density values ​​of three-dimensional sampling points Convert to transparency The value can be determined by the following formula:

[0155]

[0156] Furthermore, based on the transparency, the rendering weights corresponding to each 3D sampling point on a ray can be determined using the following formula. :

[0157]

[0158] S903 determines the rendering color value of each two-dimensional pixel based on the product of the sampled color value of each three-dimensional sampling point and the corresponding rendering weight.

[0159] Finally, based on the sampled color values ​​of each three-dimensional sampling point and the corresponding rendering weights, the rendering color values ​​of each two-dimensional pixel are determined using stereo rendering.

[0160] Optionally, render color values It can be calculated using the following formula:

[0161]

[0162] in, These are the sampled color values ​​for each sampling point.

[0163] In this embodiment, after converting the SDF value into a volume density value, the rendering color value of each two-dimensional pixel in the three-dimensional training model is determined. The density field and radiation field are coupled, providing more accurate colors and improving the accuracy of surface rendering in generating the three-dimensional training model.

[0164] After the above-mentioned 3D reconstruction model is completed, it can be used to reconstruct the local 3D space surrounding the photographed target to obtain a 3D object model with color information.

[0165] Optionally, embodiments of this application also provide a three-dimensional reconstruction method, the method comprising:

[0166] The three-dimensional uniform sampling point set within the local space surrounding the object is input into the three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model. The three-dimensional reconstruction model is constructed based on the three-dimensional reconstruction model construction method provided in the aforementioned embodiments.

[0167] The three-dimensional uniform sampling point set in the local space surrounding the object is calculated based on the poses of each camera taken around the target. First, the three-dimensional boundary information surrounding the object is calculated based on the poses of each camera. Then, the voxel resolution is set, and three-dimensional points are uniformly sampled in the xyz direction within the three-dimensional boundary.

[0168] Before inputting the three-dimensional uniform sampling point set within the local space surrounding the object into the three-dimensional reconstruction model, the reconstructed three-dimensional object model can be determined by the method provided in the above embodiments. This three-dimensional object model is a complete three-dimensional object model corresponding to the pixel set to be reconstructed, and carries the color information represented by the pixel set to be reconstructed.

[0169] In this embodiment, a three-dimensional object model with color information, wrinkle information and edge information is generated based on the set of pixels to be reconstructed, thereby improving the accuracy of the three-dimensional object model reconstruction.

[0170] The 3D reconstruction model includes: implicit surface neural networks and neural rendering networks.

[0171] The implicit surface neural network can be trained from the aforementioned implicit surface initial neural network, and the neural rendering network can be trained from the aforementioned neural rendering initial network.

[0172] like Figure 11 As shown, in the above steps, the three-dimensional uniform sampling point set in the local space surrounding the object is input into the three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model, which can be achieved by the following steps S1001 to S1002.

[0173] S1001 inputs the set of three-dimensional uniform sampling points in the local space surrounding the object into the implicit surface neural network to obtain the surface information of the reconstructed three-dimensional object model, including surface vertices, vertex normals and triangular meshes.

[0174] After inputting the set of three-dimensional uniform sampling points in the local space surrounding the object into the implicit surface neural network, the implicit surface neural network outputs the signed distance function value of each three-dimensional sampling point.

[0175] As described in the above embodiments, the SDF value is used to represent the distance of the pixel to be reconstructed from the implicit surface. Therefore, the 3D sampling points with an SDF value of 0 can be extracted using the Marching cubes algorithm and used as surface vertices. These vertices, along with the normal vector and spatial feature encoding output by the implicit surface neural network, serve as the surface information of the 3D object model. Furthermore, the implicit surface can be determined based on this surface information.

[0176] S1002, the above surface information and the three-dimensional normals of the surface vertices are input into the neural rendering network in the opposite direction for color rendering to obtain the color of each vertex of the three-dimensional object model.

[0177] Referring to the training process in the above embodiments, the above surface information, namely the surface vertices, normal vectors, and spatial features with an SDF value of 0, and the reverse normal vector of the normal vector as the ray direction, are input into the input neural rendering network for color rendering to obtain a three-dimensional object model.

[0178] In this embodiment, the 3D reconstruction model, based on the set of pixels to be reconstructed, can generate a 3D object model with color information and clear details on the basis of surface information.

[0179] See Figure 12 This application embodiment also provides a three-dimensional reconstruction model construction device 110, including:

[0180] The point set acquisition module 1101 is used to perform gradient sampling on a two-dimensional sample image to obtain a gradient salient sampling point set and a gradient non-salient sampling point set in the two-dimensional sample image. The gradient salient sampling point set includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is greater than or equal to a first preset threshold. The gradient non-salient sampling point set includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is less than the first preset threshold, or less than or equal to the average gradient magnitude. The average gradient magnitude is the average gradient magnitude of each pixel in the two-dimensional sample image.

[0181] The model training module 1102 is used to train a 3D reconstruction model based on a set of gradient significant sampling points and a set of gradient non-significant sampling points.

[0182] The point set acquisition module 1101 is further used to: determine the gradient magnitude of each pixel in the two-dimensional sample image; determine the mean gradient magnitude and standard deviation of the two-dimensional sample image based on the gradient magnitude of each pixel in the two-dimensional sample image; and perform gradient sampling on the two-dimensional sample image based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude and standard deviation of the gradient magnitude to obtain a set of gradient significant sampling points and a set of gradient non-significant sampling points.

[0183] The point set acquisition module 1101 is further used to determine the gradient salient point set, the non-salient point set, and the mask matrix corresponding to the two-dimensional sample image based on the gradient magnitude, the mean gradient magnitude, and the standard deviation of the gradient magnitude of each pixel; and to determine the gradient salient sampling point set and the gradient non-salient sampling point set based on the gradient salient point set, the non-salient point set, and the mask matrix.

[0184] The point set acquisition module 1101 is further used to determine the proportion of gradient saliency points based on the ratio of the number of pixels in the gradient saliency point set to the number of pixels in the two-dimensional sample image; and to filter the pixels in the gradient saliency point set and the non-saliency point set according to the preset sampling quantity, mask matrix and gradient saliency point proportion to obtain the gradient saliency sampling point set and the gradient non-saliency sampling point set.

[0185] The point set acquisition module 1101 is further configured to: determine the difference between the gradient magnitude and the mean gradient magnitude of each pixel; if the gradient magnitude of the first pixel is greater than the mean gradient magnitude and the difference between it and the mean gradient magnitude is greater than or equal to the standard deviation of the gradient magnitude, then the first pixel is designated as a pixel in the gradient saliency point set, and the gradient saliency mask of the first pixel is marked as saliency, and the first pixel is any pixel in the two-dimensional sample image; if the gradient magnitude of the second pixel is greater than the mean gradient magnitude and the difference between it and the mean gradient magnitude is less than the standard deviation of the gradient magnitude, or less than or equal to the mean gradient magnitude, then the second pixel is designated as a pixel in the gradient insignificant point set, and the gradient saliency mask of the second pixel is marked as insignificant, and the second pixel is any pixel in the two-dimensional sample image; and generate a mask matrix based on the gradient saliency mask markings of each pixel in the two-dimensional sample image.

[0186] The point set acquisition module 1101 is further configured to: determine the number of gradient salient points and the number of gradient non-salient points based on a preset sampling number and the proportion of gradient salient points; the sum of the number of gradient salient points and the number of gradient non-salient points is equal to the preset sampling number; and the ratio of the number of gradient salient points to the preset sampling number is equal to the proportion of gradient salient points. The module then filters the pixels in the gradient salient point set according to the number of gradient salient points and the mask matrix to obtain a gradient salient sampling point set. Finally, the module filters the pixels in the non-salient point set according to the number of gradient non-salient points and the mask matrix to obtain a gradient non-salient sampling point set.

[0187] The model training module 1102 is further used to determine the three-dimensional ray direction corresponding to each gradient salient sampling point and each gradient insignificant sampling point, as well as the three-dimensional sampling points on the corresponding ray, based on the gradient salient sampling point set, the gradient insignificant sampling point set, the two-dimensional sample image, and the camera pose of the two-dimensional sample image; input the three-dimensional ray direction corresponding to each gradient salient sampling point and each gradient insignificant sampling point, as well as the three-dimensional sampling points on the corresponding ray, into the three-dimensional reconstruction initial model, train the three-dimensional reconstruction initial model, and obtain the three-dimensional reconstruction model.

[0188] The model training module 1102 is further configured to: input the three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set into the implicit surface initial neural network to obtain the signed distance function value, normal vector, and spatial feature encoding corresponding to each sampling point; input the three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set, the signed distance function value, normal vector, spatial feature encoding of each three-dimensional sampling point, and the three-dimensional ray direction corresponding to each gradient salient sampling point and each gradient insignificant sampling point into the neural rendering initial network to obtain the sampling color value corresponding to each three-dimensional sampling point; determine the three-dimensional training model and the rendering color value of each two-dimensional pixel in the three-dimensional training model based on the sampling color value and the corresponding signed distance function value of each three-dimensional sampling point; determine the loss information of the three-dimensional reconstruction initial model based on the three-dimensional training model and the rendering color value of each pixel in the three-dimensional training model, and iteratively correct the three-dimensional reconstruction initial model according to the loss information until the corrected three-dimensional reconstruction initial model meets the preset conditions, and use the three-dimensional reconstruction initial model that meets the preset conditions as the three-dimensional reconstruction model.

[0189] The model training module 1102 is further used to convert the signed distance function value of each three-dimensional sampling point into a volume density value; determine the rendering weight of each three-dimensional sampling point based on the volume density value of each three-dimensional sampling point; and determine the rendering color value of each corresponding two-dimensional pixel based on the product of the sampling color value of each three-dimensional sampling point and the corresponding rendering weight.

[0190] See Figure 13 This application embodiment also provides a three-dimensional reconstruction device 120, including:

[0191] The model generation module 1201 is used to input a set of three-dimensional uniform sampling points in the local space surrounding the object into the three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model. The three-dimensional reconstruction model is constructed based on any of the three-dimensional reconstruction model construction methods in the foregoing embodiments.

[0192] The model generation module 1201 is further configured to input a set of three-dimensional uniform sampling points in the local space surrounding the object into an implicit surface neural network to obtain the surface information of the reconstructed three-dimensional object model, including surface vertices, vertex normals, and triangular meshes; and input the surface information and the three-dimensional normals of the surface vertices into a neural rendering network in the opposite direction for color rendering to obtain the color of each vertex of the three-dimensional object model.

[0193] Figure 14This illustration shows a schematic diagram of an electronic device provided in an embodiment of this application, including: a processor 2001, a storage medium 2002, and a bus 2003. The storage medium 2002 stores machine-readable instructions executable by the processor 2001. When the electronic device runs a three-dimensional reconstruction model construction method or a three-dimensional reconstruction method as described in the embodiment, the processor 2001 communicates with the storage medium 2002 via the bus 2003. The processor 2001 executes the machine-readable instructions. The preamble of the three-dimensional reconstruction model construction method item of the processor 2001 performs the following steps:

[0194] Gradient sampling is performed on a two-dimensional sample image to obtain a set of salient gradient sampling points and a set of insignificant gradient sampling points in the two-dimensional sample image. The set of salient gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is greater than or equal to a first preset threshold. The set of insignificant gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is less than the first preset threshold, or less than or equal to the average gradient magnitude. The average gradient magnitude is the average gradient magnitude of each pixel in the two-dimensional sample image.

[0195] A 3D reconstruction model is trained based on a set of gradient-significant sampling points and a set of gradient-insignificant sampling points.

[0196] When processor 2001 performs gradient sampling on a two-dimensional sample image to obtain a set of salient and insignificant gradient sampling points in the two-dimensional sample image, it specifically performs the following:

[0197] Determine the gradient magnitude of each pixel in the two-dimensional sample image;

[0198] Based on the gradient magnitude of each pixel in the two-dimensional sample image, determine the mean and standard deviation of the gradient magnitude of the two-dimensional sample image.

[0199] Based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, gradient sampling is performed on the two-dimensional sample image to obtain a set of gradient significant sampling points and a set of gradient non-significant sampling points.

[0200] When processor 2001 performs gradient sampling on a two-dimensional sample image based on the gradient magnitude of each pixel, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, to obtain a set of significant and insignificant gradient sampling points, it specifically performs the following:

[0201] Based on the gradient magnitude, mean gradient magnitude, and standard deviation of each pixel, determine the set of salient gradient points, the set of non-salient gradient points, and the mask matrix corresponding to the two-dimensional sample image.

[0202] Based on the set of salient points, the set of insignificant points, and the mask matrix, determine the set of salient sampling points and the set of insignificant sampling points.

[0203] When processor 2001 executes the process of determining the set of salient gradient sampling points and the set of insignificant gradient sampling points based on the set of salient gradient points, the set of insignificant gradient points, and the mask matrix, it specifically performs the following tasks:

[0204] The proportion of gradient salient points is determined by the ratio of the number of pixels in the gradient salient point set to the number of pixels in the two-dimensional sample image.

[0205] Based on the preset sampling quantity, mask matrix, and gradient saliency ratio, the pixels in the gradient saliency set and the non-saliency set are filtered to obtain the gradient saliency sampling point set and the gradient non-saliency sampling point set.

[0206] When processor 2001 performs gradient sampling on a two-dimensional sample image based on the gradient magnitude of each pixel, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, to obtain a set of salient gradient sampling points, a set of insignificant gradient sampling points, and a mask matrix corresponding to the two-dimensional sample image, it specifically performs the following:

[0207] Determine the difference between the gradient magnitude of each pixel and the mean gradient magnitude;

[0208] If the gradient magnitude of the first pixel is greater than the mean gradient magnitude, and the difference between the first pixel and the mean gradient magnitude is greater than or equal to the standard deviation of the gradient magnitude, then the first pixel is taken as a pixel in the set of gradient saliency points, and the gradient saliency mask of the first pixel is marked as saliency. The first pixel is any pixel in the two-dimensional sample image.

[0209] If the gradient magnitude of the second pixel is greater than the mean gradient magnitude and the difference between the second pixel and the mean gradient magnitude is less than the standard deviation of the gradient magnitude, or is less than or equal to the mean gradient magnitude, then the second pixel is taken as a pixel in the set of gradient non-significant points, and the gradient saliency mask of the second pixel is marked as non-significant. The second pixel is any pixel in the two-dimensional sample image.

[0210] A mask matrix is ​​generated based on the gradient saliency mask markings of each pixel in the two-dimensional sample image.

[0211] When processor 2001 executes the process of filtering the gradient salient point set and the gradient insignificant point set according to the preset sampling quantity, mask matrix, and gradient salient point ratio to obtain the gradient salient sampling point set and the gradient insignificant sampling point set, it specifically performs the following:

[0212] Based on the preset sampling quantity and the proportion of gradient saliency points, determine the sampling quantity of gradient saliency points and the sampling quantity of gradient non-saliency points. The sum of the sampling quantity of gradient saliency points and the sampling quantity of gradient non-saliency points is equal to the preset sampling quantity, and the ratio of the sampling quantity of gradient saliency points to the preset sampling quantity is equal to the proportion of gradient saliency points.

[0213] Based on the number of gradient salient points sampled and the mask matrix, the pixels in the gradient salient point set are filtered to obtain the gradient salient sampling point set;

[0214] Based on the number of gradient non-salient points sampled and the mask matrix, the pixels in the non-salient point set are filtered to obtain the gradient non-salient sampling point set.

[0215] When the processor 2001 executes training to obtain a 3D reconstruction model based on gradient-significant sampling point sets and gradient-insignificant sampling point sets, it is specifically used for:

[0216] Based on the set of salient gradient sampling points, the set of insignificant gradient sampling points, the two-dimensional sample image, and the camera pose of the two-dimensional sample image, determine the three-dimensional ray direction corresponding to each salient gradient sampling point and each insignificant gradient sampling point, as well as the three-dimensional sampling points on the corresponding ray.

[0217] The set of salient gradient sampling points, the set of insignificant gradient sampling points, each salient gradient sampling point, the corresponding 3D ray direction and the 3D sampling points on the corresponding ray are input into the initial 3D reconstruction model. The initial 3D reconstruction model is then trained to obtain the 3D reconstruction model.

[0218] In one feasible implementation, the initial model for 3D reconstruction includes: an implicit surface initial neural network and a neural rendering initial network.

[0219] When processor 2001 executes the process of inputting the set of salient gradient sampling points, the set of insignificant gradient sampling points, each salient gradient sampling point, the corresponding 3D ray direction, and the 3D sampling points on the ray into the initial 3D reconstruction model, and training the initial 3D reconstruction model to obtain the 3D reconstruction model, it is specifically used for:

[0220] Input the three-dimensional sampling points on the corresponding rays of the gradient significant sampling point set and the gradient insignificant sampling point set into the implicit surface initial neural network to obtain the symbolic distance function value, normal vector and spatial feature encoding corresponding to each sampling point;

[0221] The three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set, the signed distance function value, normal vector, spatial feature encoding of each three-dimensional sampling point, and the three-dimensional ray direction corresponding to each gradient salient sampling point and each gradient insignificant sampling point are respectively input into the neural rendering initial network to obtain the sampling color value corresponding to each three-dimensional sampling point.

[0222] Based on the sampled color values ​​of each 3D sampling point and the corresponding signed distance function value, determine the 3D training model and the rendering color values ​​of each corresponding 2D pixel in the 3D training model;

[0223] The loss information of the initial 3D reconstruction model is determined based on the 3D training model and the rendering color value of each pixel in the 3D training model. The initial 3D reconstruction model is iteratively corrected according to the loss information until the corrected initial 3D reconstruction model meets the preset conditions. The initial 3D reconstruction model that meets the preset conditions is used as the 3D reconstruction model.

[0224] When processor 2001 executes the process of determining the 3D training model and the rendering color value of each 2D pixel in the 3D training model based on the sampled color values ​​of each 3D sampling point and the corresponding signed distance function value, it specifically performs the following tasks:

[0225] The signed distance function values ​​of each 3D sampling point are converted into volume density values ​​respectively;

[0226] The rendering weight of each 3D sampling point is determined based on the volume density value of each 3D sampling point.

[0227] The rendering color value of each two-dimensional pixel is determined by multiplying the sampled color value of each three-dimensional sampling point with the corresponding rendering weight.

[0228] The preamble of the Processor 2001 3D Reconstruction Method item performs the following steps:

[0229] The three-dimensional uniform sampling point set within the local space surrounding the object is input into the three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model. The three-dimensional reconstruction model is constructed based on any of the three-dimensional reconstruction model construction methods in the aforementioned embodiments.

[0230] In one feasible implementation, the 3D reconstruction model includes: an implicit surface neural network and a neural rendering network.

[0231] When processor 2001 executes the process of inputting a set of uniformly sampled three-dimensional points within the local space surrounding an object into a three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model, it specifically performs the following tasks:

[0232] The three-dimensional uniform sampling point set within the local space surrounding the object is input into the implicit surface neural network to obtain the surface information of the reconstructed three-dimensional object model, including surface vertices, vertex normals, and triangular meshes.

[0233] The surface information and the opposite direction of the vertex normal vector are input into the neural rendering network as ray directions for color rendering, thus obtaining the color of each vertex of the 3D object model.

[0234] By incorporating the gradient information contained in both salient and non-salient gradient sampling point sets during the training of the 3D reconstruction model, the stability of the training process is improved. This allows for the full learning of edge and wrinkle information of objects in 2D sample images, thereby enhancing the accuracy of the reconstructed 3D model.

[0235] This application embodiment also provides a computer-readable storage medium storing a computer program, which is executed by a processor, and the processor performs the following steps:

[0236] Gradient sampling is performed on a two-dimensional sample image to obtain a set of salient gradient sampling points and a set of insignificant gradient sampling points in the two-dimensional sample image. The set of salient gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than or equal to the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is greater than a first preset threshold. The set of insignificant gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is less than or equal to the first preset threshold, or less than the average gradient magnitude. The average gradient magnitude is the average gradient magnitude of each pixel in the two-dimensional sample image.

[0237] A 3D reconstruction model is trained based on a set of gradient-significant sampling points and a set of gradient-insignificant sampling points.

[0238] When the processor performs gradient sampling on a two-dimensional sample image to obtain a set of salient and insignificant gradient sampling points in the two-dimensional sample image, it specifically performs the following:

[0239] Determine the gradient magnitude of each pixel in the two-dimensional sample image;

[0240] Based on the gradient magnitude of each pixel in the two-dimensional sample image, determine the mean and standard deviation of the gradient magnitude of the two-dimensional sample image.

[0241] Based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, gradient sampling is performed on the two-dimensional sample image to obtain a set of gradient significant sampling points and a set of gradient non-significant sampling points.

[0242] When the processor performs gradient sampling on a two-dimensional sample image based on the gradient magnitude of each pixel, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, to obtain a set of significant and insignificant gradient sampling points, it specifically performs the following:

[0243] Based on the gradient magnitude, mean gradient magnitude, and standard deviation of each pixel, determine the set of salient gradient points, the set of non-salient gradient points, and the mask matrix corresponding to the two-dimensional sample image.

[0244] Based on the set of salient points, the set of insignificant points, and the mask matrix, determine the set of salient sampling points and the set of insignificant sampling points.

[0245] When the processor determines the set of salient and insignificant gradient sampling points based on the set of salient and insignificant gradient points and the mask matrix, it specifically performs the following tasks:

[0246] The proportion of gradient salient points is determined by the ratio of the number of pixels in the gradient salient point set to the number of pixels in the two-dimensional sample image.

[0247] Based on the preset sampling quantity, mask matrix, and gradient saliency ratio, the pixels in the gradient saliency set and the non-saliency set are filtered to obtain the gradient saliency sampling point set and the gradient non-saliency sampling point set.

[0248] When the processor performs gradient sampling on the two-dimensional sample image based on the gradient magnitude of each pixel, the mean gradient magnitude, and the standard deviation of the gradient magnitude, to obtain a set of salient gradient sampling points, a set of insignificant gradient sampling points, and a mask matrix corresponding to the two-dimensional sample image, it specifically performs the following:

[0249] Determine the difference between the gradient magnitude of each pixel and the mean gradient magnitude;

[0250] If the gradient magnitude of the first pixel is greater than the mean gradient magnitude, and the difference between the first pixel and the mean gradient magnitude is greater than or equal to the standard deviation of the gradient magnitude, then the first pixel is taken as a pixel in the set of gradient saliency points, and the gradient saliency mask of the first pixel is marked as saliency. The first pixel is any pixel in the two-dimensional sample image.

[0251] If the gradient magnitude of the second pixel is greater than the mean gradient magnitude and the difference between the second pixel and the mean gradient magnitude is less than the standard deviation of the gradient magnitude, or is less than or equal to the mean gradient magnitude, then the second pixel is taken as a pixel in the set of gradient non-significant points, and the gradient saliency mask of the second pixel is marked as non-significant. The second pixel is any pixel in the two-dimensional sample image.

[0252] A mask matrix is ​​generated based on the gradient saliency mask markings of each pixel in the two-dimensional sample image.

[0253] When the processor performs the task of filtering the set of salient and insignificant gradient points according to a preset sampling number, mask matrix, and percentage of salient gradient points, to obtain the set of salient and insignificant gradient sampling points, it specifically performs the following:

[0254] Based on the preset sampling quantity and the proportion of gradient saliency points, determine the sampling quantity of gradient saliency points and the sampling quantity of gradient non-saliency points. The sum of the sampling quantity of gradient saliency points and the sampling quantity of gradient non-saliency points is equal to the preset sampling quantity, and the ratio of the sampling quantity of gradient saliency points to the preset sampling quantity is equal to the proportion of gradient saliency points.

[0255] Based on the number of gradient salient points sampled and the mask matrix, the pixels in the gradient salient point set are filtered to obtain the gradient salient sampling point set;

[0256] Based on the number of gradient non-salient points sampled and the mask matrix, the pixels in the non-salient point set are filtered to obtain the gradient non-salient sampling point set.

[0257] When the processor executes training to obtain a 3D reconstruction model based on gradient-significant and gradient-insignificant sampling point sets, it is specifically used for:

[0258] Based on the set of salient gradient sampling points, the set of insignificant gradient sampling points, the two-dimensional sample image, and the camera pose of the two-dimensional sample image, determine the three-dimensional ray direction corresponding to each salient gradient sampling point and each insignificant gradient sampling point, as well as the three-dimensional sampling points on the corresponding ray.

[0259] The set of salient gradient sampling points, the set of insignificant gradient sampling points, each salient gradient sampling point, the corresponding 3D ray direction and the 3D sampling points on the corresponding ray are input into the initial 3D reconstruction model. The initial 3D reconstruction model is then trained to obtain the 3D reconstruction model.

[0260] In one feasible implementation, the initial model for 3D reconstruction includes: an implicit surface initial neural network and a neural rendering initial network.

[0261] When the processor executes the process of inputting the 3D ray directions corresponding to each significant gradient sampling point and each insignificant gradient sampling point, as well as the 3D sampling points on the rays, into the initial 3D reconstruction model, and training the initial 3D reconstruction model to obtain the 3D reconstruction model, it is specifically used for:

[0262] Input the three-dimensional sampling points on the corresponding rays of the gradient significant sampling point set and the gradient insignificant sampling point set into the implicit surface initial neural network to obtain the symbolic distance function value, normal vector and spatial feature encoding corresponding to each sampling point;

[0263] The three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set, the signed distance function value, normal vector, spatial feature encoding of each three-dimensional sampling point, and the three-dimensional ray direction corresponding to each gradient salient sampling point and each gradient insignificant sampling point are respectively input into the neural rendering initial network to obtain the sampling color value corresponding to each three-dimensional sampling point.

[0264] Based on the sampled color values ​​of each 3D sampling point and the corresponding signed distance function value, determine the 3D training model and the rendering color values ​​of each corresponding 2D pixel in the 3D training model;

[0265] The loss information of the initial 3D reconstruction model is determined based on the 3D training model and the rendering color value of each pixel in the 3D training model. The initial 3D reconstruction model is iteratively corrected according to the loss information until the corrected initial 3D reconstruction model meets the preset conditions. The initial 3D reconstruction model that meets the preset conditions is used as the 3D reconstruction model.

[0266] When the processor determines the 3D training model and the rendering color value of each 2D pixel in the 3D training model based on the sampled color values ​​of each 3D sampling point and the corresponding signed distance function value, it is specifically used for:

[0267] The signed distance function values ​​of each 3D sampling point are converted into volume density values ​​respectively;

[0268] The rendering weight of each 3D sampling point is determined based on the volume density value of each 3D sampling point.

[0269] The rendering color value of each two-dimensional pixel is determined by multiplying the sampled color value of each three-dimensional sampling point with the corresponding rendering weight.

[0270] The preamble of the processor's 3D reconstruction method item performs the following steps:

[0271] The three-dimensional uniform sampling point set within the local space surrounding the object is input into the three-dimensional reconstruction model to obtain the reconstructed three-dimensional object model. The three-dimensional reconstruction model is constructed based on any of the three-dimensional reconstruction model construction methods in the aforementioned embodiments.

[0272] In one feasible implementation, the 3D reconstruction model includes: an implicit surface neural network and a neural rendering network.

[0273] When the processor executes the process of inputting a set of uniformly sampled 3D points within the local space surrounding the object into the 3D reconstruction model to obtain the reconstructed 3D object model, it is specifically used for:

[0274] The three-dimensional uniform sampling point set in the local space of the object to be surrounded is input into the implicit surface neural network to obtain the surface information of the reconstructed three-dimensional object model, including surface vertices, vertex normals and triangular meshes.

[0275] The surface information and the opposite direction of the vertex normal vector are input into the neural rendering network as ray directions for color rendering, thus obtaining the color of each vertex of the 3D object model.

[0276] By incorporating the gradient information contained in both salient and non-salient gradient sampling point sets during the training of the 3D reconstruction model, the stability of the training process is improved. This allows for the full learning of edge and wrinkle information of objects in 2D sample images, thereby enhancing the accuracy of the reconstructed 3D model.

[0277] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0278] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0279] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0280] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0281] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0282] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0283] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a three-dimensional reconstruction model, characterized in that, The method includes: Gradient sampling is performed on a two-dimensional sample image to obtain a set of salient gradient sampling points and a set of insignificant gradient sampling points in the two-dimensional sample image. The set of salient gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is greater than or equal to a first preset threshold. The set of insignificant gradient sampling points includes multiple pixels in the two-dimensional sample image, where the gradient magnitude of each pixel is greater than the average gradient magnitude and the difference between the gradient magnitude and the average gradient magnitude is less than the first preset threshold, or less than or equal to the average gradient magnitude. The average gradient magnitude is the average gradient magnitude of each pixel in the two-dimensional sample image. A 3D reconstruction model is trained based on the set of salient gradient sampling points and the set of insignificant gradient sampling points.

2. The three-dimensional reconstruction model construction method according to claim 1, characterized in that, The step of performing gradient sampling on a two-dimensional sample image to obtain a set of salient gradient sampling points and a set of insignificant gradient sampling points in the two-dimensional sample image includes: Determine the gradient magnitude of each pixel in the two-dimensional sample image; Based on the gradient magnitude of each pixel in the two-dimensional sample image, determine the mean gradient magnitude and standard deviation of the gradient magnitude of the two-dimensional sample image. Based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, gradient sampling is performed on the two-dimensional sample image to obtain the set of significant gradient sampling points and the set of insignificant gradient sampling points.

3. The method for constructing a three-dimensional reconstruction model according to claim 2, characterized in that, The step of performing gradient sampling on the two-dimensional sample image based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, to obtain the set of significant gradient sampling points and the set of insignificant gradient sampling points, includes: Based on the gradient magnitude of each pixel, the mean gradient magnitude, and the standard deviation of the gradient magnitude, determine the set of salient gradient points, the set of non-salient gradient points, and the mask matrix corresponding to the two-dimensional sample image. The set of salient gradient points and the set of insignificant gradient points are determined based on the set of salient gradient points, the set of insignificant gradient points, and the mask matrix.

4. The method for constructing a three-dimensional reconstruction model according to claim 3, characterized in that, The step of determining the set of salient gradient sampling points and the set of insignificant gradient sampling points based on the set of salient gradient points, the set of insignificant gradient points, and the mask matrix includes: The proportion of gradient salient points is determined based on the ratio of the number of pixels in the gradient salient point set to the number of pixels in the two-dimensional sample image. According to the preset sampling quantity, the mask matrix, and the proportion of gradient salient points, the pixels in the gradient salient point set and the pixels in the non-salient point set are filtered to obtain the gradient salient sampling point set and the gradient non-salient sampling point set.

5. The method for constructing a three-dimensional reconstruction model according to claim 3, characterized in that, The step of performing gradient sampling on the two-dimensional sample image based on the gradient magnitude of each pixel in the two-dimensional sample image, the mean gradient magnitude of the two-dimensional sample image, and the standard deviation of the gradient magnitude, to obtain the set of salient gradient sampling points, the set of insignificant gradient sampling points, and the mask matrix corresponding to the two-dimensional sample image includes: Determine the difference between the gradient magnitude of each pixel and the average gradient magnitude; If the gradient magnitude of the first pixel is greater than the mean gradient magnitude, and the difference between the first pixel and the mean gradient magnitude is greater than or equal to the standard deviation of the gradient magnitude, then the first pixel is taken as a pixel in the set of gradient saliency points, and the gradient saliency mask of the first pixel is marked as saliency. The first pixel is any pixel in the two-dimensional sample image. If the gradient magnitude of the second pixel is greater than the mean gradient magnitude and the difference between the second pixel and the mean gradient magnitude is less than the standard deviation of the gradient magnitude, or is less than or equal to the mean gradient magnitude, then the second pixel is taken as a pixel in the set of gradient non-significant points, and the gradient saliency mask of the second pixel is marked as non-significant. The second pixel is any pixel in the two-dimensional sample image. The mask matrix is ​​generated based on the gradient saliency mask markers of each pixel in the two-dimensional sample image.

6. The method for constructing a three-dimensional reconstruction model according to claim 4, characterized in that, The step of filtering the set of salient gradient points and the set of insignificant gradient points according to a preset sampling number, the mask matrix, and the proportion of salient gradient points to obtain the set of salient gradient sampling points and the set of insignificant gradient sampling points includes: Based on the preset sampling quantity and the percentage of gradient salient points, the sampling quantity of gradient salient points and the sampling quantity of gradient non-salient points are determined. The sum of the sampling quantity of gradient salient points and the sampling quantity of gradient non-salient points is equal to the preset sampling quantity, and the ratio of the sampling quantity of gradient salient points to the preset sampling quantity is equal to the percentage of gradient salient points. According to the number of gradient salient points sampled and the mask matrix, the pixels in the gradient salient point set are filtered to obtain the gradient salient sampling point set; The pixels in the set of insignificant points are filtered according to the number of gradient insignificant points sampled and the mask matrix to obtain the set of gradient insignificant sampling points.

7. The method for constructing a three-dimensional reconstruction model according to any one of claims 1-6, characterized in that, The process of training a 3D reconstruction model based on the set of salient gradient sampling points and the set of insignificant gradient sampling points includes: Based on the set of salient gradient sampling points, the set of insignificant gradient sampling points, the two-dimensional sample image, and the camera pose of the two-dimensional sample image, determine the three-dimensional ray direction corresponding to each salient gradient sampling point and each insignificant gradient sampling point, as well as the three-dimensional sampling points on the corresponding ray. The three-dimensional ray directions corresponding to each gradient significant sampling point and each gradient insignificant sampling point, as well as the three-dimensional sampling points on the corresponding rays, are input into the three-dimensional reconstruction initial model. The three-dimensional reconstruction initial model is then trained to obtain the three-dimensional reconstruction model.

8. The method for constructing a three-dimensional reconstruction model according to claim 7, characterized in that, The initial model for 3D reconstruction includes: an implicit surface initial neural network and a neural rendering initial network; The process involves inputting the three-dimensional ray directions corresponding to each significant gradient sampling point and each insignificant gradient sampling point, as well as the three-dimensional sampling points on the rays, into the initial three-dimensional reconstruction model, and training the initial three-dimensional reconstruction model to obtain the three-dimensional reconstruction model, including: The three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set are input into the implicit surface initial neural network to obtain the symbolic distance function value, normal vector and spatial feature code corresponding to each three-dimensional sampling point; The three-dimensional sampling points on the corresponding rays of the gradient salient sampling point set and the gradient insignificant sampling point set, the signed distance function value, normal vector, spatial feature encoding of each three-dimensional sampling point, and the three-dimensional ray direction corresponding to each gradient salient sampling point and each gradient insignificant sampling point are respectively input into the neural rendering initial network to obtain the sampling color value corresponding to each three-dimensional sampling point. Based on the sampled color values ​​of each three-dimensional sampling point and the corresponding signed distance function value, the three-dimensional training model and the rendering color values ​​of each corresponding two-dimensional pixel in the three-dimensional training model are determined. The loss information of the initial 3D reconstruction model is determined based on the 3D training model and the rendering color value of each pixel in the 3D training model. The initial 3D reconstruction model is iteratively corrected according to the loss information until the corrected initial 3D reconstruction model meets the preset conditions. The initial 3D reconstruction model that meets the preset conditions is used as the 3D reconstruction model.

9. The method for constructing a three-dimensional reconstruction model according to claim 8, characterized in that, The step of determining the 3D training model and the rendering color value of each 2D pixel in the 3D training model based on the sampled color value of each 3D sampling point and the corresponding signed distance function value includes: The symbolic distance function values ​​of each of the three-dimensional sampling points are converted into volume density values ​​respectively; The rendering weight of each three-dimensional sampling point is determined based on the volume density value of each sampling point. The rendering color value of each two-dimensional pixel is determined by multiplying the sampled color value of each three-dimensional sampling point with the corresponding rendering weight.

10. A three-dimensional reconstruction method, characterized in that, The method includes: A set of three-dimensional uniform sampling points within the local space surrounding the object is input into a three-dimensional reconstruction model to obtain a reconstructed three-dimensional object model. The three-dimensional reconstruction model is constructed based on the three-dimensional reconstruction model construction method according to any one of claims 1-9.

11. The three-dimensional reconstruction method according to claim 10, characterized in that, The 3D reconstruction model includes: an implicit surface neural network and a neural rendering network; The step of inputting a set of uniformly sampled three-dimensional points within the local space surrounding the object into a three-dimensional reconstruction model to obtain a reconstructed three-dimensional object model includes: The three-dimensional uniform sampling points are input into the implicit surface neural network to obtain the surface information of the reconstructed three-dimensional object model, including surface vertices, vertex normals, and triangular meshes. The surface information and the opposite direction of the vertex normal vector are input into the neural rendering network as ray directions for color rendering to obtain the color of each vertex of the 3D object model.

12. A processing apparatus, characterized in that, The processing device includes a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the processing device is running, the processor communicates with the storage medium via the bus. The processor executes the machine-readable instructions to perform the steps of the three-dimensional reconstruction model construction method as described in any one of claims 1-9 or the three-dimensional reconstruction method as described in any one of claims 10-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the three-dimensional reconstruction model construction method as described in any one of claims 1-9 or the three-dimensional reconstruction method as described in any one of claims 10-11.

Images