Image-based 3D reconstruction methods, equipment, media and products

Abstract

This application discloses an image-based 3D reconstruction method, device, medium, and product, relating to the field of 3D reconstruction technology. The method comprises: obtaining one reference image from m original images and N-1 source images; sending the reference image and N-1 source images to a pre-trained segmentation model and a pre-trained feature pyramid network to generate a reference feature images of different scales and a*(N-1) source feature images of different scales; then sending the image to a pre-trained cost volume-depth map model to generate a final depth image; each of the m original images is used as a reference image in turn, and the above steps are performed to generate m final depth images; and the m final depth images are sent to a point cloud fusion algorithm to output a 3D point cloud model of the target object. This application improves the perception of weakly textured areas, thereby improving the accuracy of 3D reconstruction.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to an image-based three-dimensional reconstruction method, device, medium and product. Background Art

[0002] With advancements in image processing technology, image-based 3D reconstruction is increasingly being used in fields such as photogrammetry, computer vision, and computer graphics. The general process of 3D reconstruction involves obtaining a reference image and source image with known camera parameters; constructing a cost volume for each image; generating multiple depth maps based on the cost volume; and finally, fusing these depth maps using a point cloud fusion algorithm to obtain a 3D view. However, reconstructing regions with weak textures has always been a major challenge for 3D reconstruction solutions.

[0003] Traditional 3D reconstruction algorithms rely on manual feature extraction and matching. While these methods achieve good reconstruction results in areas with rich textures, they often struggle with weak textures or form holes. Convolutional neural networks were later used for feature extraction, but their smaller receptive fields limit their understanding of global features, resulting in suboptimal reconstruction results in areas with weak textures. While 3D reconstruction is generally progressing, these methods often suffer from low accuracy in areas with weak textures, leading to lower overall 3D reconstruction accuracy. Summary of the Invention

[0004] The purpose of this application is to provide an image-based three-dimensional method, device, medium and product to solve the problems described in the background technology.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an image-based three-dimensional reconstruction method, comprising:

[0007] Acquire m original images of a target object, wherein the m original images are images taken from different angles of the target object, and the target object is the object to be three-dimensionally reconstructed;

[0008] Obtain one image from the m original images as a reference image, and then obtain N-1 images as source images;

[0009] Sending the reference image and N-1 source images to a multi-scale feature extraction model, wherein the multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, where a is the number of layers of the pre-trained feature pyramid network;

[0010] Sending the a reference feature images of different scales and a*(N-1) source feature images of different scales to a pre-trained cost volume-depth map model, so that the cost volume-depth map model outputs a final depth image;

[0011] For each of the m original images, take turns as a reference image, and execute the steps of "obtaining one image from the m original images as a reference image, and then obtaining N-1 images as source images" to "making the cost volume-depth map model output a final depth image" to obtain m final depth images;

[0012] The m final depth images are sent to a point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object.

[0013] Optionally, weight parameters of the visual encoder of the pre-trained segmentation-everything model are in a frozen state.

[0014] Optionally, sending the reference image and N-1 source images to a multi-scale feature extraction model, the multi-scale feature extraction model including a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, including:

[0015] Sending the one reference image and N-1 source images to the pre-trained segmentation model so that the segmentation model outputs a first feature image corresponding to the one reference image and each of the N-1 source images;

[0016] The reference image, N-1 source images, and the first feature image are sent to the pre-trained feature pyramid network, so that the pre-trained feature pyramid network outputs the a reference feature images of different scales and a*(N-1) source feature images of different scales.

[0017] Optionally, the pre-trained cost volume-depth map model includes a cost volume construction model, a regularization model, and a depth prediction model. Then, the a reference feature images of different scales and a*(N-1) source feature images of different scales are sent to the pre-trained cost volume-depth map model so that the cost volume-depth map model outputs a final depth image, including:

[0018] Obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales, and sending the images to the cost volume construction model so that the cost volume construction model outputs a first cost volume;

[0019] Regularizing the first cost volume to obtain a first probability volume;

[0020] Sending the first probability volume to a depth prediction model so that the depth prediction model outputs a first depth image;

[0021] Upsampling the first depth image to transform the first depth image into a secondary depth image, where the scale of the secondary depth image is consistent with the scale of the secondary coarse-scale feature image;

[0022] Based on the secondary depth image, the reference feature image at the secondary coarse scale, and the N-1 source feature images, re-performing the steps of “obtaining a reference feature image at the coarsest scale and N-1 source feature images from the a reference feature images at different scales and the a*(N-1) source feature images at different scales” to “ensuring that the scale of the secondary depth image is consistent with the scale of the feature image at the secondary coarse scale” to obtain a second depth image;

[0023] Repeat the steps of "obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales" to "obtaining a second depth image" until the final depth image is obtained based on the reference image of the finest scale and the N-1 source feature images.

[0024] Optionally, the pre-trained depth prediction model is a pre-trained binary search-based depth prediction model, and the first probability body is sent to the depth prediction model so that the depth prediction model outputs a first depth image, including: sending the first probability body to the pre-trained binary search-based depth prediction model so that the pre-trained binary search-based depth prediction model outputs the first depth image.

[0025] Optionally, the depth prediction model based on binary search is trained by the following method:

[0026] Collect image datasets of different scenes to obtain training sets;

[0027] Sending the first cost volume of each reference image in the training set to an initial binary search-based depth prediction model, so that the initial binary search-based depth prediction model outputs a predicted depth image;

[0028] Sending the depth image predicted and the real depth image of the training data set at each stage to the adaptive weight adjustment loss function, so that the adaptive weight adjustment loss function calculates the loss value of each stage, and reflects and propagates based on the loss value to adjust the weight parameters of the initial binary search-based depth prediction model;

[0029] For the predicted depth map corresponding to each image in the training set, the above-mentioned "collecting image data sets of different scenes" to "and adjusting the weight parameters of the initial binary search-based depth prediction model based on reflection propagation" are executed one by one until the loss value no longer decreases. The training is stopped, and the weight parameters at this time are determined as the optimal weight parameter model of the binary search-based depth prediction model to obtain the binary search-based depth prediction model.

[0030] Optionally, the point cloud fusion algorithm is a visibility-based depth map fusion algorithm, and sending the m ultimate depth images to the point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object includes: sending the m depth images to the visibility-based depth map fusion algorithm so that the visibility-based depth map fusion algorithm outputs a three-dimensional point cloud model.

[0031] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0032] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0033] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0034] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0035] An embodiment of the present application provides an image-based three-dimensional reconstruction method, which includes: obtaining a reference image and N-1 source images for three-dimensional reconstruction; using a segmentation-all model (SAM) and a pre-trained feature pyramid network (FPN) to extract features from the reference image and the source image to obtain a reference feature image and a source feature image; then obtaining N ultimate depth images based on the reference feature image and the source feature image; and finally, fusing the N ultimate depth images to obtain a three-dimensional point cloud model. The segmentation-all model (SAM) and the pre-trained feature pyramid network (FPN) can greatly improve the network's perception of weak texture areas, enabling the network to better perform three-dimensional reconstruction for weak textures. However, in the prior art, convolutional neural networks are used for feature extraction, and their small receptive field limits the understanding of global features. Therefore, the reconstruction effect in weak texture areas is still not ideal, resulting in low three-dimensional reconstruction accuracy. Therefore, compared to the prior art, the present application improves the three-dimensional reconstruction accuracy of weak textures, thereby also improving the accuracy of the three-dimensional reconstruction of the entire target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 This is a diagram of an application environment of an image-based three-dimensional reconstruction method in one embodiment of the present application;

[0038] Figure 2 A schematic diagram of a flow chart of an image-based 3D reconstruction method provided in one embodiment of the present application;

[0039] Figure 3 A model block diagram used in an image-based 3D reconstruction method provided in one embodiment of the present application;

[0040] Figure 4 A model block diagram used in an image-based 3D reconstruction method provided in one embodiment of the present application;

[0041] Figure 5 A flowchart of a method for generating a feature image provided in one embodiment of the present application;

[0042] Figure 6 A schematic diagram of an image processing process using a SAM model provided in another embodiment of the present application;

[0043] Figure 7 A schematic diagram of a process for generating a final depth image according to an embodiment of the present application;

[0044] Figure 8 A schematic diagram of a binary depth search process provided in one embodiment of the present application;

[0045] Figure 9 A schematic diagram of a process for generating a depth prediction model based on binary search provided in one embodiment of the present application;

[0046] Figure 10 A different embodiment of the present application provides The impact of the value on the weight function change trend;

[0047] Figure 11 A schematic diagram of a flow chart of an image-based 3D reconstruction method provided in one embodiment of the present application;

[0048] Figure 12 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] The image-based 3D reconstruction method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the application environment includes a terminal and a server. The terminal communicates with the server through a network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the image to be processed to the server. After the server receives the image to be processed, it can store it first, and then retrieve it from the storage location when it needs to be processed. Alternatively, the processing task can be performed while storing, and the server performs three-dimensional reconstruction based on the reference image and the source image. The server can feed back the obtained three-dimensional point cloud model to the terminal. In addition, in some embodiments, the image-based three-dimensional reconstruction method can also be implemented by the server or the terminal alone, such as the terminal can directly perform three-dimensional reconstruction on the image to be processed, or the server can obtain the image to be processed from the data storage system and perform image-based three-dimensional reconstruction on the image to be processed.

[0052] Terminals include, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices include smart watches, smart bracelets, and head-mounted devices. Servers can be implemented as standalone servers or server clusters consisting of multiple servers, or even cloud servers.

[0053] In an exemplary embodiment, Figure 2 As shown, combined with Figure 3 and Figure 4 , provides an image-based three-dimensional reconstruction method, which is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 Taking the server in FIG. 1 as an example, the method includes the following steps 201 to 206:

[0054] Step 200 : Acquire m original images of a target object, wherein the m original images are images captured from different angles of the target object, and the target object is an object to be three-dimensionally reconstructed.

[0055] For example, if a total of 20 images are taken, then m=20.

[0056] Step 201 : obtaining one image from the m original images as a reference image, and then obtaining N-1 images as source images.

[0057] Reference image: In algorithms, a reference image is typically chosen as a benchmark against which other images (i.e., source images) are compared for feature matching, depth prediction, and other processing. The reference image acts as an "anchor" in the algorithmic process, with which information from other images is compared and integrated.

[0058] Source images: Source images are images that overlap with the reference image. These images provide additional perspective information, enhancing the algorithm's ability to reconstruct the object. The number of source images and the degree of overlap affect the algorithm's performance and reconstruction accuracy.

[0059] In the present application, during the 3D reconstruction process, one of the m original images is used as a reference image, and then based on the reference image, N-1 images are selected from the remaining m-1 images as source images.

[0060] The value of m can be determined manually, for example, it can be set to 20.

[0061] Furthermore, the value of N is predetermined and can be manually determined. For example, assuming N=6, there is one reference image and five source images. Generally, a larger value of N results in a more complete reconstructed point cloud model, but also results in higher computer memory consumption. Therefore, N can be selected based on experience. The example of 5 in this application is only for the convenience of subsequent description.

[0062] In step 202 , the reference image and N-1 source images are sent to a multi-scale feature extraction model. The multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales.

[0063] Among them, the multi-scale feature extraction model can also be called a global-local feature extraction model.

[0064] Where a is the number of layers of the pre-trained feature pyramid network

[0065] For each image input to the global-local feature extraction model (including the reference image and each source image), a feature image of varying scales is output. That is, after a reference image passes through the global-local feature extraction model, a reference feature image of varying scales is output. After N-1 source images pass through the global-local feature extraction model, each source image also outputs a source feature image of varying scales, for a total of a*(N-1) source feature images.

[0066] Furthermore, it can be understood as a reference feature image at different scales, including a different scales, each scale including one reference feature image; a*(N-1) source feature images at different scales, including a different scales, but each scale including N-1 source feature images of the same scale. Therefore, for each scale, it includes N images, that is, one reference feature image and N-1 source feature images.

[0067] Furthermore, the size of a is determined by the number of convolutional layers included in the pre-trained feature pyramid network. Figure 4 As shown in the figure, for each image input to the global-local feature extraction model, 4 feature images will be output, that is, a is 4 at this time.

[0068] The detailed process of the Segment Anything Model (SAM) is described below.

[0069] The specific process of Feature Pyramid Networks (FPN) is described below.

[0070] In step 203 , the a reference feature images of different scales and a*(N-1) source feature images of different scales are sent to a pre-trained cost volume-depth map model, so that the cost volume-depth map model outputs a final depth image.

[0071] The pre-trained cost volume-depth map model includes a cost volume construction model, a regularization model, and a depth prediction model. The usage process of the cost volume construction model, regularization model, and depth prediction model is as follows:

[0072] For example, for each scale, two optional depth values and N images (including a reference feature map and N-1 source feature maps) are sequentially processed through the cost volume construction model, regularization model, and depth prediction model to obtain an intermediate depth image. This intermediate depth image is then upsampled to the same scale as the next scale. The upsampled intermediate depth image and the N images at the next scale are then sequentially processed through the cost volume construction model, regularization model, and depth prediction model to obtain another intermediate depth image. These operations are repeated for each scale, resulting in a final depth map. This means that each reference image corresponds to a final depth map. This is a brief introduction; please refer to the relevant description below for a detailed introduction.

[0073] From step 201 to step 203 , it can be seen that after selecting one image from the m images as a reference image and N−1 images as source images, a terminal depth image can be finally obtained.

[0074] In step 204 , each of the m original images is used as a reference image in turn, and the above steps 201 to 203 are performed to obtain m final depth images.

[0075] When each original image is used as a reference image, a final depth image can be obtained. Therefore, a total of m final depth images can be obtained.

[0076] Step 205: Send the m final depth images to a point cloud fusion algorithm, so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object.

[0077] An embodiment of the present application provides an image-based three-dimensional reconstruction method, which includes: obtaining a reference image and N-1 source images for three-dimensional reconstruction; using a segmentation-all model (SAM) and a pre-trained feature pyramid network (FPN) to extract features from the reference image and the source image to obtain a reference feature image and a source feature image; then obtaining N ultimate depth images based on the reference feature image and the source feature image; and finally, fusing the N ultimate depth images to obtain a three-dimensional point cloud model. The segmentation-all model (SAM) and the pre-trained feature pyramid network (FPN) can greatly improve the network's perception of weak texture areas, enabling the network to better perform three-dimensional reconstruction for weak textures. However, in the prior art, convolutional neural networks are used for feature extraction, and their small receptive field limits the understanding of global features. Therefore, the reconstruction effect in weak texture areas is still not ideal, resulting in low three-dimensional reconstruction accuracy. Therefore, compared to the prior art, the present application improves the three-dimensional reconstruction accuracy of weak textures, thereby also improving the accuracy of the three-dimensional reconstruction of the entire target object.

[0078] Alternatively, see Figure 4 In an exemplary embodiment of the present application, the connection relationship between the pre-trained segmentation model and the pre-trained feature pyramid network is as follows: the input of the pre-trained segmentation model is the reference image and the N-1 source images, the input of the pre-trained feature pyramid network is the reference image and the N-1 source images, the output of the pre-trained segmentation model is added and fused with the coarsest scale features of the pre-trained feature pyramid network, and the output of the pre-trained feature pyramid network is used as the output of the global-local multi-scale feature extraction model; see Figure 5 , the step 202 includes step 2021 and step 2022:

[0079] Step 2021: Send the one reference image and N-1 source images to the pre-trained segmentation model, so that the segmentation model outputs a first feature image corresponding to each image in the one reference image and the N-1 source images.

[0080] The first feature image is a set of segmentation masks.

[0081] The global-local multi-scale feature extraction model includes a two-branch structure. The first branch is the pre-trained SAM, and the second branch is the feature pyramid (FPN). Through these two branch structures, features are extracted for each image in the multi-view image (that is, 1 reference image and N-1 source images). For each image, a feature map is obtained. In this application, a is 4 as an example, that is, each image corresponds to four feature images of different scales.

[0082] The input of both branches is 1 reference image I0 and N-1 source images I i In this application, for the convenience of description, 1 reference image I0 and N-1 source images are Denoted as the input image, the input image is respectively subjected to the pre-trained SAM branch and the feature pyramid (FPN) branch, wherein the output result of the pre-trained SAM branch is denoted as the output result of the first branch, and is added and fused with the coarsest-scale features in the FPN branch. The final output result is the output result of multiple convolutional layers of FPN (this application takes four convolutional layers as an example), and its height (H), width (W) and channel (C) dimensions are: (H / 8, W / 8, 8C), (H / 4, W / 4, 4C), (H / 2, W / 2, 2C) and (H, W, C), respectively.

[0083] (1) Pre-trained SAM branch

[0084] The input of the pre-trained SAM branch is 1 reference image and N-1 source images. SAM includes a key component, the visual encoder, which is a Vision Transformer (ViT) pre-trained based on MAE. It divides the image into multiple patches and uses the self-attention mechanism to calculate the global correlation between different patches, thereby better capturing the semantic correlation between distant contexts in the image. The matrix for calculating the attention output can be expressed as formula (1):

[0085] (1)

[0086] in, Represents the matrix for calculating attention output;

[0087] Q, K, and V represent the query matrix, key matrix, and value matrix, respectively;

[0088] Softmax(·) is a normalization function;

[0089] d is a scaling factor used to prevent the gradient of the softmax(·) function from vanishing during calculation.

[0090] The pre-trained SAM branch can greatly enhance the global semantic understanding of weak texture areas, thereby improving the accuracy and confidence of depth prediction in the basic stage and providing a more accurate basic value for the binary search in the subsequent stage.

[0091] SAM generates a set of (one or more) segmentation masks for each input image, which represent the boundaries of different objects or regions in the image.

[0092] Optionally, weight parameters of the visual encoder of the pre-trained segmentation-everything model are in a frozen state.

[0093] After freezing the weight parameters of the pre-trained SAM visual encoder, the weight parameters are no longer updated with backpropagation during training to improve the stability of the model.

[0094] Figure 6 The detailed process of pre-training SAM branch is shown. Taking the input of an image into SAM as an example for polishing, SAM's visual encoder requires the input image size to be 1024*1024, so first the image to be input (including 1 reference image and any one of N-1 source images) is scaled and padded to convert the image into a size acceptable to the pre-trained SAM, and the required image is converted into a data type acceptable to the pre-trained SAM through regularization and data type conversion; then the image with acceptable size and data type is sent to the pre-trained SAM visual encoder with frozen weights, so that its output H, W, C are [64, 64, 256] respectively and with padding (padding) feature images, so it is necessary to remove padding and size conversion operations on the feature image to adjust the feature image to the same size as the coarsest scale feature map F of FPN. input Size The same image, denoted as F SAM Finally, a trainable encoder-decoder convolutional layer is used to reduce F SAM The channel dimension of the image is similar to the coarsest feature image of FPN The channel dimension of is the same as that of , so that it can be fused with the FPN branch. The encoding-decoding convolution layer can be expressed as formulas (2), (3) and (4):

[0095] (2)

[0096] (3)

[0097] (4)

[0098] Among them, F input It is the coarsest scale feature map of FPN;

[0099] F SAM is the feature image with padding;

[0100] for The channel dimension;

[0101] for The channel dimension;

[0102] For the encoder;

[0103] For the decoder;

[0104] represents the convolutional layer;

[0105] and Represents two deconvolution layers.

[0106] Step 2022: Send the reference image, N-1 source images, and the first feature image to the pre-trained feature pyramid network, so that the pre-trained feature pyramid network outputs the a reference feature images of different scales and a*(N-1) source feature images of different scales.

[0107] For the convenience of description, a reference feature images of different scales and a*(N-1) source feature images of different scales are recorded as second feature images.

[0108] (2) Feature Pyramid Branch

[0109] The input of the FPN branch is 1 reference image and N-1 source images. Each input image is encoded and decoded separately through the convolution method, and the feature fusion between different convolutional layers is achieved through the jump connection layer. Finally, a four-layer convolutional coding operation is performed to output four feature maps of different sizes with height (H), width (W) and channel (C) dimensions: (H / 8, W / 8, 8C), (H / 4, W / 4, 4C), (H / 2, W / 2, 2C) and (H, W, C).

[0110] For each input image, 4 feature maps of different sizes are generated, that is, there are 4 reference feature maps and 4(N-1) source feature maps .

[0111] Alternatively, see Figure 7 , the pre-trained cost volume-depth map model includes a pre-trained cost volume construction model, a regularization model, and a depth prediction model, then step 203 includes the following steps 2031 to 2036:

[0112] Step 2031: Obtain a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales, and send them to the cost volume construction model, so that the cost volume construction model outputs a first cost volume.

[0113] The general process of constructing the cost volume model includes: for each scale, each source feature image generates a similarity volume with the reference feature image, so there are N-1 similarities in total; then the N-1 similarities are aggregated into a cost volume.

[0114] Furthermore, similar bodies can be obtained through homography transformation. The following is a brief introduction to homography transformation:

[0115] First, each source feature image is Warp to the reference feature via homography In this way, N-1 similar bodies are constructed one by one.

[0116] For pixel-level homography transformation, refer to formula (5): For a certain pixel on the reference feature image and the source feature image, assume that p is a certain pixel on the reference feature image, and p' is a pixel on the source feature image according to the depth hypothesis. The pixel obtained after distortion, the pixel p' can be calculated by formula (5):

[0117] (5)

[0118] Where p is a pixel on the reference feature image;

[0119] p' is the source feature image based on the depth hypothesis The pixels obtained after homography transformation;

[0120] is the depth hypothesis, j represents a depth hypothesis layer, and D is the total number of depth hypothesis samples;

[0121] is the camera rotation matrix from the source feature image to the reference feature image, i represents the sequence number of the source feature image;

[0122] is the camera translation matrix from the source feature image to the reference feature image;

[0123] is the camera intrinsic parameter of the reference image;

[0124] is the camera intrinsic parameter of the source image.

[0125] Furthermore, in order to speed up the operation efficiency of computer equipment, the N-1 similar bodies can be decomposed along the channel dimension through group association calculation to obtain Slice the similar body, and then The similar bodies are sliced and synthesized into N-1 similar bodies.

[0126] For each source feature image And the corresponding reference feature image The similar body is calculated by formula (6). Assume that the number of channels of the source feature image and the reference feature image are both , for each reference feature map , divided into Group (a picture is divided into groups), each group has channels; for each source feature map , according to the channel dimension, it is also divided into groups, each group also has channels. Assume is the source feature map The g-th group of , each source feature map can be expressed by formula (6) With the reference feature map Similar bodies after group association calculation :

[0127] (6)

[0128] Among them, j represents a certain depth hypothesis layer;

[0129] is the number of channels of the source feature image and the reference feature image;

[0130] is the number of groups divided according to the channel dimension;

[0131] Represents the reference feature image The g-th group;

[0132] is the source feature image The g-th group;

[0133] Represents the calculation vector The inner product of Group association of a group;

[0134] Group correlation successfully inherits the advantages of traditional correlation cost matching and provides rich similarity measurement features for 3D aggregation networks.

[0135] For each source feature image , are calculated according to the above formula (6) and the reference feature image at the same scale A similar body of , thus N-1 source feature images Can be constructed with a reference feature map A similar body.

[0136] After obtaining N-1 similar bodies, they are aggregated to form a cost body. The cost body formation process is as follows:

[0137] After constructing each source feature image With the reference feature image Similar Afterwards, a set of 3D CNNs are used to predict a set of pixel-by-pixel weight matrices , and then these weight matrices are weightedly fused with the similar bodies through formula (7), so that N-1 similar bodies are aggregated into the final cost body ;

[0138] (7)

[0139] in, It is the cost body after the aggregation of N-1 similar bodies;

[0140] To use a set of 3D CNNs to predict a set of pixel-by-pixel weight matrices;

[0141] The similarity volume constructed for the i-th source feature image and the reference feature image;

[0142] Step 2032: Regularize the first cost volume to obtain a first probability volume.

[0143] Finally, a 3D U-Net network is used to aggregate the cost volume Regularize the cost The channel dimension is reduced to 1, and then the reference feature image is output The corresponding regularized cost volume , and use the Softmax(·) function to normalize along the depth D dimension to construct a pixel-by-pixel probability volume , the matching cost of the pixel at each depth is converted into a probability value for the final depth prediction.

[0144] Step 2033: Send the first probability volume to a depth prediction model, so that the depth prediction model outputs a first depth image.

[0145] Step 2034 : Upsample the first depth image to transform it into a secondary depth image. The scale of the secondary depth image is consistent with the scale of the secondary coarse-scale feature image.

[0146] Step 2035 : Based on the secondary depth image, the reference feature image at the secondary coarse scale, and N−1 source feature images, re-execute steps 2031 to 2034 to obtain a second depth image.

[0147] Step 2036 : Repeat steps 2031 to 2035 until the final depth image is obtained based on a reference image of the finest scale and N-1 source feature images.

[0148] Optionally, in another exemplary embodiment of the present application, the pre-trained depth prediction model is a pre-trained binary search-based depth prediction model, then step 2033 includes: sending the first probability body to the pre-trained binary search-based depth prediction model, so that the pre-trained binary search-based depth prediction model outputs the first depth image.

[0149] The following is an introduction to binary search:

[0150] The depth prediction of binary search refers to refining the depth search range by binary search stage by stage. Specifically, for the k-th search stage, the depth range is divided into two equal boxes, namely ,in Represents a box. Since each box represents a range of depth values, and a certain depth value is required in the homography transformation of the feature map, the center point of the box is used to represent the depth sampling value of each box. In this way, the cost volume can be constructed for the two boxes and the depth prediction can be performed. The depth sampling values of the two boxes at the kth stage are It can be expressed as the following formula (8):

[0151] (8)

[0152] in, is the depth sampling value of the two boxes at the kth stage;

[0153] Represents the boundaries of the box;

[0154] The three sides of the two boxes from left to right are After obtaining the probability volume P according to the two depth sampling values, The D dimension is argmax(·) operated to obtain the depth prediction label of this stage. The depth prediction label represents the predicted depth value of the next stage is located in the corresponding box. Therefore, in the k+1 stage, by continuing the binary search in the box where the depth prediction label of the k stage is located, Continue to divide into two new boxes of equal width , and then repeat the above operation until the depth prediction label of the last stage is generated and used as the final depth prediction value.

[0155] The following is an introduction to the fault tolerance box and gradient mask optimization:

[0156] like Figure 8 As shown in Figure 1, the Error Tolerance Box (ETB) mechanism refers to adding an equal number of equal-width error-tolerant boxes at both ends of the original two boxes in each search stage to enhance the network's tolerance to classification errors. Specifically, after the box is determined by the depth prediction label in the kth stage, the box is divided into two for the k+1th stage binary search. Assuming D is the final number of boxes (D is a small even number), Additional boxes are filled on both sides of the original two boxes. Therefore, the box at stage k+1 can be expressed as , the number of boundaries of the box can be expressed as Still using the center sample of the box as the depth hypothesis value, we can still obtain D depth sampling values through formula (8) In order to unify the number of boxes in each stage, the initial depth range is also divided into D boxes in the first stage, so the width of each box in the initial stage is , in the kth stage the box width is Adding error-tolerant boxes increases the depth search range, increasing the probability that the true depth value falls within the depth search range. This allows the network to optimize for incorrectly predicted depth labels within a certain range and reduce error accumulation. Using D=4, adding an error-tolerant box on each side, enhances the network's tolerance to prediction errors with virtually no increase in video memory consumption.

[0157] The gradient mask optimization mechanism is a mask mechanism that determines whether the true depth value exceeds the range of the depth box. The network is supervised by the real depth map. In the kth stage, the real depth value is projected into the box to determine which box the real depth value is in. Then, a one-hot label can be constructed based on the center sampling value to calculate the loss. However, according to the previous description, the deviation of the classification process may cause the true depth value to be outside the range of the depth box. Therefore, it is impossible to construct a valid one-hot label, which affects the back propagation optimization of the network. Therefore, consider introducing a mask mechanism to determine whether the true depth value of each stage is within the boundary range of D depth boxes. If it exceeds the boundary range of the depth box, it is considered to be an invalid pixel. After filtering the mask map, only valid pixels are retained for back propagation to update the gradient parameters. The masked pixels terminate the back propagation of this stage and subsequent stages. Suppose the true depth value of a pixel is , the boundary of the current stage is , then formula (9) can be used to determine whether the pixel is a valid pixel:

[0158] (9)

[0159] in, is the boundary of the current stage;

[0160] is the true depth value of the pixel;

[0161] By introducing these two mechanisms, most pixels can participate in the learning and gradient optimization process, greatly enhancing the performance and stability of the network.

[0162] Alternatively, see Figure 9 In another exemplary embodiment of the present application, the depth prediction model based on binary search is trained through the following steps 2041 to 2047:

[0163] Step 2041: collect image data sets of different scenes to obtain a training set.

[0164] Step 2042 : Send the first cost volume of each reference image in the training set to the initial binary search-based depth prediction model, so that the initial binary search-based depth prediction model outputs a predicted depth image.

[0165] Step 2043: Send the depth image predicted by the training data set at each stage and the actual depth image to the adaptive weight adjustment loss function, so that the adaptive weight adjustment loss function calculates the loss value of each stage, and reflects and propagates based on the loss value to adjust the weight parameters of the initial binary search-based depth prediction model.

[0166] Step 2044: For the predicted depth map corresponding to each image in the training set, execute steps 2041 to 2043 one by one until the loss value no longer decreases, stop training, and determine the weight parameters at this time as the optimal weight parameter model of the depth prediction model based on binary search, thereby obtaining the depth prediction model based on binary search.

[0167] Finally, the accuracy of the depth prediction model based on binary search can be tested using any data in the test set. When the accuracy is less than the expected result, steps 2041 to 2044 are re-executed. When the accuracy is greater than or equal to the expected result, no further training is required.

[0168] In addition, the training set can use a public training dataset or build it yourself. Similarly, the test set can be collected by yourself or use a public one.

[0169] See also Figure 10 , introduce the adaptive weight adjustment loss function:

[0170] In the MVS dataset, it was found that samples with richer textures generally have smaller loss values, while samples with weaker textures generally have larger loss values. Pixels with smaller loss values are called easy samples, while those with larger loss values are called difficult samples.

[0171] The adaptive weight adjustment loss function calculates the loss between the depth maps generated at each stage of the network and the ground-truth depth map, and performs backpropagation to optimize network parameters to achieve optimal depth prediction for samples in weakly textured areas. This loss function has two main operations: adaptive classification of easy / difficult samples and dynamic shifting of sample attention.

[0172] Adaptive classification of easy / hard samples

[0173] Based on the standard cross entropy loss, the sample adaptive classification loss function (Adaptive Weight Adjustment Loss, referred to as AWA-Loss) is expressed as the following formula (10):

[0174] (10)

[0175] in, The weight coefficient of the cross entropy loss for each pixel sample;

[0176] is the valid pixel set determined by formula (9);

[0177] The probability of the label location for each pixel sample;

[0178] is the standard cross entropy loss.

[0179] SAC-Loss can be calculated based on The size of the adaptive function distinguishes between simple and difficult samples and reduces the relative loss value of simple samples, thereby giving relatively greater attention to difficult samples, making the model more focused on optimizing difficult samples.

[0180] The following is a brief introduction to the dynamic transfer of sample attention:

[0181] Since only adaptive classification of samples will cause SAC-Loss to ignore the optimization of simple samples, and the early loss value of training is generally high, it cannot be used as the basis for classification of simple or difficult samples. Therefore, a dynamic transfer strategy of sample attention from easy to difficult is proposed to further optimize the loss function. The final AWA-Loss can be expressed as formula (11):

[0182] (11)

[0183] in, It is a weight reduction function that decreases monotonically with the training process;

[0184] It is a weighting function that increases monotonically with the training process;

[0185] The current epoch.

[0186] and is defined as formulas (12) and (13):

[0187] (12)

[0188] (13)

[0189] in, Indicates the total number of epochs;

[0190] and is the weight function curve.

[0191] Alternatively, see Figure 11 In another exemplary embodiment of the present application, after the above step 204, the method further includes the following step 300:

[0192] Step 300: Filter the m final depth maps to obtain m filtered depth images.

[0193] This application uses two constraints, geometric consistency and photometric consistency, to filter pixels in the depth map that exceed a predetermined threshold.

[0194] Among them, the geometric consistency constraint is used to measure the depth consistency between m depth maps. Projection onto source image On, then through Depth estimation ,Will Reproject back to the reference image. If the reprojection coordinates and reprojection depth If formula (14) and formula (15) are satisfied, the depth estimation d1 of p1 is called dual-view Figure 1 In the experiment, all depths should be at least three views. Figure 1 To.

[0195] (14)

[0196] (15)

[0197] in,

[0198] is the pixel on the reference feature;

[0199] is the reprojection depth;

[0200] is the depth value.

[0201] The photometric consistency constraint filters low confidence pixel depth values through the confidence map. The confidence map is obtained by the probability body in step 203. , the maximum value is obtained along the depth D dimension, and the maximum probability value of each pixel is the confidence value. Depth predictions with confidence lower than 0.8 are filtered out. In order to further suppress reconstruction noise, the visible view of each pixel is determined in the filtering step, and all reprojected depths are The average value of is taken as the final depth estimate of the pixel.

[0202] The filtered m depth images are sent to the point cloud fusion algorithm as new depth images.

[0203] Optionally, in another exemplary embodiment of the present application, the point cloud fusion algorithm is a visibility-based depth map fusion algorithm, and step 206 includes: sending the m depth images to the visibility-based depth map fusion algorithm, so that the visibility-based depth map fusion algorithm outputs a three-dimensional point cloud model.

[0204] Depth occlusions and violations between different viewpoints are minimized through a visibility-based depth map fusion algorithm, thereby integrating depth maps from different views into a unified point cloud representation.

[0205] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image-based three-dimensional reconstruction related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for image-based three-dimensional reconstruction is implemented.

[0206] Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0207] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0208] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0209] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0211] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0212] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0213] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image-based three-dimensional reconstruction method, characterized in that: include: Acquire m original images of a target object, wherein the m original images are images taken from different angles of the target object, and the target object is the object to be three-dimensionally reconstructed; Obtain one image from the m original images as a reference image, and then obtain N-1 images as source images; Sending the reference image and N-1 source images to a multi-scale feature extraction model, wherein the multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, where a is the number of layers of the pre-trained feature pyramid network; Sending the a reference feature images of different scales and a*(N-1) source feature images of different scales to a pre-trained cost volume-depth map model, so that the cost volume-depth map model outputs a final depth image; For each of the m original images, use it as a reference image in turn, and perform the steps of "obtaining one image from the m original images as a reference image, and then obtaining N-1 images as source images" to "enabling the cost volume-depth map model to output a final depth image" to obtain m final depth images; The m final depth images are sent to a point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object.

2. The image-based 3D reconstruction method according to claim 1, wherein: The weight parameters of the visual encoder of the pre-trained segmentation-everything model are in a frozen state.

3. The image-based 3D reconstruction method according to claim 2, wherein: The step of sending the reference image and N-1 source images to a multi-scale feature extraction model, wherein the multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, including: Sending the one reference image and N-1 source images to the pre-trained segmentation model so that the segmentation model outputs a first feature image corresponding to the one reference image and each of the N-1 source images; The reference image, N-1 source images, and the first feature image are sent to the pre-trained feature pyramid network, so that the pre-trained feature pyramid network outputs the a reference feature images of different scales and a*(N-1) source feature images of different scales.

4. The image-based 3D reconstruction method according to claim 1, wherein: The pre-trained cost volume-depth map model includes a pre-trained cost volume construction model, a regularization model, and a depth prediction model, and the a reference feature images of different scales and a*(N-1) source feature images of different scales are sent to the pre-trained cost volume-depth map model so that the cost volume-depth map model outputs a final depth image, including: Obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales, and sending the images to the cost volume construction model so that the cost volume construction model outputs a first cost volume; Regularizing the first cost volume to obtain a first probability volume; Sending the first probability volume to a depth prediction model so that the depth prediction model outputs a first depth image; Upsampling the first depth image to transform the first depth image into a secondary depth image, where the scale of the secondary depth image is consistent with the scale of the secondary coarse-scale feature image; Based on the secondary depth image, the reference feature image at the secondary coarse scale, and the N-1 source feature images, re-performing the steps of "obtaining a reference feature image at the coarsest scale and N-1 source feature images from the a reference feature images at different scales and the a*(N-1) source feature images at different scales" to "ensuring that the scale of the secondary depth image is consistent with the scale of the feature image at the secondary coarse scale" to obtain a second depth image; Repeat the steps of "obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales" to "obtaining a second depth image" until the final depth image is obtained based on the reference image of the finest scale and the N-1 source feature images.

5. The image-based 3D reconstruction method according to claim 4, characterized in that: If the pre-trained depth prediction model is a pre-trained binary search-based depth prediction model, sending the first probability body to the depth prediction model so that the depth prediction model outputs a first depth image includes: sending the first probability body to the pre-trained binary search-based depth prediction model so that the pre-trained binary search-based depth prediction model outputs the first depth image.

6. The image-based 3D reconstruction method according to claim 5, characterized in that: The depth prediction model based on binary search is trained by the following method: Collect image datasets of different scenes to obtain training sets; Sending the first cost volume of each reference image in the training set to an initial binary search-based depth prediction model, so that the initial binary search-based depth prediction model outputs a predicted depth image; Sending the depth image predicted and the real depth image of the training data set at each stage to the adaptive weight adjustment loss function, so that the adaptive weight adjustment loss function calculates the loss value of each stage, and reflects and propagates based on the loss value to adjust the weight parameters of the initial binary search-based depth prediction model; For the predicted depth map corresponding to each image in the training set, the above-mentioned "collecting image data sets of different scenes" to "and adjusting the weight parameters of the initial binary search-based depth prediction model based on reflection propagation" are executed one by one until the loss value no longer decreases. The training is stopped, and the weight parameters at this time are determined as the optimal weight parameter model of the binary search-based depth prediction model to obtain the binary search-based depth prediction model.

7. The image-based 3D reconstruction method according to any one of claims 1 to 6, characterized in that: If the point cloud fusion algorithm is a visibility-based depth map fusion algorithm, then sending the m ultimate depth images to the point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object includes: sending the m depth images to the visibility-based depth map fusion algorithm so that the visibility-based depth map fusion algorithm outputs a three-dimensional point cloud model.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image-based three-dimensional reconstruction method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image-based three-dimensional reconstruction method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image-based three-dimensional reconstruction method according to any one of claims 1 to 7 is implemented.

Images