Image-based three-dimensional reconstruction method and device, medium and product

Abstract

The invention discloses an image-based three-dimensional reconstruction method and device, a medium and a product, and relates to the technical field of three-dimensional reconstruction, and the method comprises the steps: obtaining one of m original images as a reference image, and N-1 original images as source images; the reference image and the N-1 source images are sent to a pre-trained segmentation model and a pre-trained feature pyramid network, so that a reference feature images with different scales and a * (N-1) source feature images with different scales are obtained; sending the image to a pre-trained cost body-depth map model so as to enable the pre-trained cost body-depth map model to output an ultimate depth image; taking each image in the m original images as a reference image in turn, and executing the steps to obtain m ultimate depth images; and sending the m ultimate depth images to a point cloud fusion algorithm to output a three-dimensional point cloud model of the target object. According to the invention, the perception of the weak texture region is improved, so that the precision of three-dimensional reconstruction is improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to an image-based three-dimensional reconstruction method, device, medium and product. Background Art

[0002] With the improvement of image processing technology, image-based 3D reconstruction is increasingly used in photogrammetry, computer vision, computer graphics and other fields. The general process of 3D reconstruction is: obtain reference images and source images with known camera parameters; construct cost volumes for each image; generate multiple depth maps based on the cost volumes; and fuse multiple depth maps through point cloud fusion algorithms to obtain a 3D view. However, reconstruction of weak texture areas has always been a major problem in 3D reconstruction solutions.

[0003] In traditional 3D reconstruction algorithms, feature matching is performed by manually extracting features. Although it can achieve good reconstruction results in areas with rich textures, it is often difficult to reconstruct or form holes in areas with weak textures. Later, convolutional neural networks were used for feature extraction, but their small receptive field limited the understanding of global features, so the reconstruction effect in areas with weak textures was still not ideal. Although 3D reconstruction is progressing overall, these methods have low accuracy for reconstructing areas with weak textures, which also leads to low accuracy of the entire 3D reconstruction. Summary of the invention

[0004] The purpose of this application is to provide an image-based three-dimensional method, device, medium and product to solve the problems described in the background technology.

[0005] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides an image-based three-dimensional reconstruction method, comprising: Acquire m original images of a target object, wherein the m original images are images taken from different angles of the target object, and the target object is an object to be three-dimensionally reconstructed; Obtain one image from the m original images as a reference image, and then obtain N-1 images as source images; Sending the one reference image and N-1 source images to a multi-scale feature extraction model, wherein the multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, where a is the number of layers of the pre-trained feature pyramid network; Sending the a reference feature images of different scales and a*(N-1) source feature images of different scales to a pre-trained cost volume-depth map model, so that the cost volume-depth map model outputs a final depth image; For each of the m original images, take turns as a reference image, and execute the steps of “obtaining one image from the m original images as a reference image, and then obtaining N-1 images as source images” to “so that the cost volume-depth map model outputs a final depth image” to obtain m final depth images; The m ultimate depth images are sent to a point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object.

[0006] Optionally, weight parameters of the visual encoder of the pre-trained segmentation-everything model are in a frozen state.

[0007] Optionally, sending the reference image and N-1 source images to a multi-scale feature extraction model, the multi-scale feature extraction model including a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, including: Sending the one reference image and N-1 source images to the pre-trained segmentation model so that the segmentation model outputs a first feature image corresponding to each of the one reference image and the N-1 source images; The one reference image, N-1 source images and the first feature image are sent to the pre-trained feature pyramid network, so that the pre-trained feature pyramid network outputs the a reference feature images of different scales and a*(N-1) source feature images of different scales.

[0008] Optionally, the pre-trained cost volume-depth map model includes a cost volume construction model, a regularization model and a depth prediction model, and the a reference feature images of different scales and a*(N-1) source feature images of different scales are sent to the pre-trained cost volume-depth map model so that the cost volume-depth map model outputs a final depth image, including: Obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales, and sending them to the cost volume construction model, so that the cost volume construction model outputs a first cost volume; Regularizing the first cost body to obtain a first probability body; Sending the first probability volume to a depth prediction model so that the depth prediction model outputs a first depth image; Upsampling the first depth image to transform the first depth image into a secondary depth image, wherein a scale of the secondary depth image is consistent with a scale of a secondary coarse-scale feature image; Based on the secondary depth image, the one reference feature image of the secondary coarse scale and the N-1 source feature images, re-execute the steps of “obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales” to “the scale of the secondary depth image is consistent with the scale of the feature image of the secondary coarse scale” to obtain a second depth image; Repeat the steps of “obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales” to “obtaining a second depth image” until the ultimate depth image is obtained based on the reference image of the finest scale and the N-1 source feature images.

[0009] Optionally, the pre-trained depth prediction model is a pre-trained binary search-based depth prediction model, and the first probability body is sent to the depth prediction model so that the depth prediction model outputs a first depth image, including: sending the first probability body to the pre-trained binary search-based depth prediction model so that the pre-trained binary search-based depth prediction model outputs the first depth image.

[0010] Optionally, the depth prediction model based on binary search is trained by the following method: Collect image data sets of different scenes to obtain training sets; Sending the first cost volume of each reference image in the training set to the initial binary search-based depth prediction model, so that the initial binary search-based depth prediction model outputs a predicted depth image; Sending the depth image predicted by the training data set at each stage and the real depth image to the adaptive weight adjustment loss function, so that the adaptive weight adjustment loss function calculates the loss value of each stage, and reflects and propagates based on the loss value to adjust the weight parameters of the initial binary search-based depth prediction model; For the predicted depth map corresponding to each image in the training set, execute the above-mentioned "collecting image data sets of different scenes" to "adjusting the weight parameters of the initial depth prediction model based on binary search based on reflection propagation of the loss value" one by one until the loss value no longer decreases, stop training, and determine the weight parameters at this time as the optimal weight parameter model of the depth prediction model based on binary search, so as to obtain the depth prediction model based on binary search.

[0011] Optionally, the point cloud fusion algorithm is a visibility-based depth map fusion algorithm, and sending the m ultimate depth images to the point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object includes: sending the m depth images to the visibility-based depth map fusion algorithm so that the visibility-based depth map fusion algorithm outputs a three-dimensional point cloud model.

[0012] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0013] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0014] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0015] According to the specific embodiments provided in this application, this application discloses the following technical effects: The embodiment of the present application provides an image-based three-dimensional reconstruction method, which includes: obtaining a reference image and N-1 source images for three-dimensional reconstruction; using the segmentation model SAM and the pre-trained feature pyramid network FPN to extract features from the reference image and the source image to obtain a reference feature image and a source feature image; then obtaining N ultimate depth images based on the reference feature image and the source feature image; finally, fusing the N ultimate depth images to obtain a three-dimensional point cloud model. Among them, the segmentation model SAM and the pre-trained feature pyramid network FPN can greatly improve the network's perception of weak texture areas, so that the network can better perform three-dimensional reconstruction for weak textures. For the prior art, a convolutional neural network is used for feature extraction, and its small receptive field limits the understanding of global features. Therefore, the reconstruction effect in the weak texture area is still not ideal, resulting in low three-dimensional reconstruction accuracy. Therefore, compared with the prior art, the present application improves the three-dimensional reconstruction accuracy of weak textures, thereby also improving the accuracy of the three-dimensional reconstruction of the entire target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is a diagram of an application environment of an image-based three-dimensional reconstruction method in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of an image-based three-dimensional reconstruction method provided in one embodiment of the present application; Figure 3 A model block diagram used in an image-based three-dimensional reconstruction method provided in one embodiment of the present application; Figure 4 A model block diagram used in an image-based three-dimensional reconstruction method provided in one embodiment of the present application; Figure 5 A flowchart of a method for generating a feature image provided by an embodiment of the present application; Figure 6 A schematic diagram of a SAM model processing process for an image provided by another embodiment of the present application; Figure 7 A schematic diagram of a process for generating a final depth image provided by an embodiment of the present application; Figure 8 A schematic diagram of a binary depth search process provided by an embodiment of the present application; Fig. 9 A schematic diagram of a process for generating a depth prediction model based on binary search provided in one embodiment of the present application; Fig.10 A different embodiment of the present application provides The influence of the value on the weight function change trend; Fig.11 A schematic diagram of a flow chart of an image-based three-dimensional reconstruction method provided in one embodiment of the present application; Fig.12 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] The image-based three-dimensional reconstruction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the application environment includes a terminal and a server. Among them, the terminal communicates with the server through a network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the image to be processed to the server. After the server receives the image to be processed, it can store it first, and then retrieve it from the storage location when it needs to be processed. Alternatively, the processing task can be performed while storing, and the server performs three-dimensional reconstruction based on the reference image and the source image. The server can feed back the obtained three-dimensional point cloud model to the terminal. In addition, in some embodiments, the image-based three-dimensional reconstruction method can also be implemented by a server or a terminal alone, such as the terminal can directly perform three-dimensional reconstruction on the image to be processed, or the server can obtain the image to be processed from the data storage system and perform image-based three-dimensional reconstruction on the image to be processed.

[0021] The terminals may be, but are not limited to, various desktop computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers, or a cloud server.

[0022] In an exemplary embodiment, Figure 2 As shown, combined with Figure 3 and Figure 4 , provides an image-based 3D reconstruction method, which is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server in the example is used to illustrate, including the following steps 201 to 206: Step 200 , obtaining m original images of a target object, wherein the m original images are images taken from different angles of the target object, and the target object is an object to be three-dimensionally reconstructed.

[0023] For example, if a total of 20 images are taken, then m=20.

[0024] Step 201 , obtaining one image from the m original images as a reference image, and then obtaining N-1 images as source images.

[0025] Reference image: In the algorithm, the reference image is usually selected as a benchmark, and other images (i.e. source images) will be processed with it for feature matching, depth prediction, etc. The reference image plays the role of an "anchor point" in the algorithm process, and the information of other images will be compared and fused with it.

[0026] Source images: Source images are other images that overlap with the reference image. These images provide additional perspective information to enhance the algorithm's ability to reconstruct the target object in three dimensions. The number and degree of overlap of source images will affect the algorithm's performance and reconstruction accuracy.

[0027] In the present application, during the three-dimensional reconstruction process, one image among the m original images is used as a reference image, and then based on the reference image, N-1 images are selected from the remaining m-1 images as source images.

[0028] The value of m can be determined artificially, for example, set to 20.

[0029] Furthermore, the value of N is predetermined and can be determined manually. For example, assuming N=6, there is one reference image and five source images. Generally, the larger the value of N, the higher the integrity of the reconstructed point cloud model, but the corresponding computer video memory consumption is also higher. Therefore, N can be determined based on experience. The example of 5 in this application is only for the convenience of subsequent description.

[0030] Step 202, sending the one reference image and N-1 source images to a multi-scale feature extraction model, wherein the multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales.

[0031] Among them, the multi-scale feature extraction model can also be called a global-local feature extraction model.

[0032] Where a is the number of layers of the pre-trained feature pyramid network For each image input to the global-local feature extraction model (including each image in the reference image and the source image), a feature images of different scales will be output. That is, after a reference image passes through the global-local feature extraction model, a reference feature images of different scales will be output. After N-1 source images pass through the global-local feature extraction model, each source image will also output a source feature image of different scales, and a*(N-1) source feature images will be output in total.

[0033] Furthermore, it can also be understood as a reference feature images of different scales, including a different scales, each scale includes 1 reference feature image; a*(N-1) source feature images of different scales, including a different scales, but each scale includes N-1 source feature images of the same scale. Therefore, for each scale, it includes N images, that is, one reference feature image and N-1 source feature images.

[0034] Furthermore, the size of a is determined by the number of convolutional layers included in the pre-trained feature pyramid network. Figure 4 As shown, for each image input into the global-local feature extraction model, 4 feature images will be output, that is, a is 4 at this time.

[0035] The specific process of the Segment Anything Model (SAM) can be found in the following description.

[0036] The specific process of the feature pyramid network (FPN) can be found in the following description.

[0037] Step 203 : sending the a reference feature images of different scales and a*(N-1) source feature images of different scales to a pre-trained cost volume-depth map model, so that the cost volume-depth map model outputs a final depth image.

[0038] The pre-trained cost volume-depth graph model includes a cost volume construction model, a regularization model, and a depth prediction model. The usage process of the cost volume construction model, the regularization model, and the depth prediction model is as follows: For a different scales, take the scale increasing in sequence as an example. For each scale, two optional depth values ​​and N images (including a reference feature map and N-1 source feature maps), a cost volume construction model, a regularization model, and a depth prediction model are sequentially used to obtain an intermediate depth image; then the intermediate depth image is upsampled to make its scale the same as the next scale; then the intermediate depth image after upsampling and the N images of the next scale are sequentially passed through the cost volume construction model, regularization model, and depth prediction model to obtain another intermediate depth image. For a scale, the above operations are performed sequentially to finally obtain a final depth map, that is, a reference map corresponds to a final depth map. Only a brief introduction is given here, and a detailed introduction can be found in the relevant description below.

[0039] It can be seen from step 201 to step 203 that after selecting one image from the m images as a reference image and N-1 images as source images, a terminal depth image can be finally obtained.

[0040] Step 204: For each of the m original images, use it as a reference image in turn, and perform the above steps 201 to 203 to obtain m final depth images.

[0041] When each original image is used as a reference image, a final depth image can be obtained, so a total of m final depth images can be obtained.

[0042] Step 205: Send the m ultimate depth images to a point cloud fusion algorithm, so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object.

[0043] The embodiment of the present application provides an image-based three-dimensional reconstruction method, which includes: obtaining a reference image and N-1 source images for three-dimensional reconstruction; using the segmentation model SAM and the pre-trained feature pyramid network FPN to extract features from the reference image and the source image to obtain a reference feature image and a source feature image; then obtaining N ultimate depth images based on the reference feature image and the source feature image; finally, fusing the N ultimate depth images to obtain a three-dimensional point cloud model. Among them, the segmentation model SAM and the pre-trained feature pyramid network FPN can greatly improve the network's perception of weak texture areas, so that the network can better perform three-dimensional reconstruction for weak textures. For the prior art, a convolutional neural network is used for feature extraction, and its small receptive field limits the understanding of global features. Therefore, the reconstruction effect in the weak texture area is still not ideal, resulting in low three-dimensional reconstruction accuracy. Therefore, compared with the prior art, the present application improves the three-dimensional reconstruction accuracy of weak textures, thereby also improving the accuracy of the three-dimensional reconstruction of the entire target object.

[0044] Alternatively, see Figure 4 In an exemplary embodiment of the present application, the connection relationship between the pre-trained segmentation model and the pre-trained feature pyramid network is: the input of the pre-trained segmentation model is the one reference image and the N-1 source images, the input of the pre-trained feature pyramid network is the one reference image and the N-1 source images, the output of the pre-trained segmentation model is added and fused with the coarsest scale features of the pre-trained feature pyramid network, and the output of the pre-trained feature pyramid network is used as the output of the global-local multi-scale feature extraction model; see Figure 5 , the step 202 includes step 2021 and step 2022: Step 2021, sending the one reference image and N-1 source images to the pre-trained segmentation model, so that the segmentation model outputs the first feature image corresponding to each image in the one reference image and the N-1 source images.

[0045] The first feature image is a set of segmentation masks.

[0046] The global-local multi-scale feature extraction model includes a dual-branch structure. The first branch is the pre-trained SAM, and the second branch is the feature pyramid (FPN). These two branch structures are used to extract features from each image in the multi-view image (i.e., 1 reference image and N-1 source images). For each image, a feature map is obtained. In this application, a is taken as 4 as an example, that is, each image corresponds to four feature images of different scales.

[0047] The input of both branch structures is a reference image I 0 and N-1 source images I i In this application, for the convenience of description, a reference image I 0 and N-1 source images Denoted as the input image, the input image is respectively subjected to the pre-trained SAM branch and the feature pyramid (FPN) branch, wherein the output result of the pre-trained SAM branch is denoted as the output result of the first branch, and is added and fused with the coarsest scale feature in the FPN branch. The final output result is the output result of multiple convolutional layers of FPN (this application takes four convolutional layers as an example), and its height (H), width (W) and channel (C) dimensions are: (H / 8, W / 8, 8C), (H / 4, W / 4, 4C), (H / 2, W / 2, 2C) and (H, W, C), respectively.

[0048] (1) Pre-trained SAM branch The input of the pre-trained SAM branch is 1 reference image and N-1 source images. SAM includes a key component, the visual encoder, which is a Vision Transformer (ViT) pre-trained based on MAE. It divides the image into multiple patches and calculates the global correlation between different patches through the self-attention mechanism, so as to better capture the semantic association between long-distance contexts in the image. The matrix for calculating the attention output can be expressed as formula (1): (1) in, Represents the matrix for calculating attention output; Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; Softmax(·) is a normalization function; d is a scaling factor used to prevent the gradient of the softmax(·) function from vanishing during calculation.

[0049] The pre-trained SAM branch can greatly enhance the global semantic understanding of weak texture areas, thereby improving the accuracy and confidence of depth prediction in the basic stage and providing a more accurate basic value for the binary search in the subsequent stage.

[0050] SAM generates a set of (one or more) segmentation masks for each input image, where the masks represent the boundaries of different objects or regions in the image.

[0051] Optionally, weight parameters of the visual encoder of the pre-trained segmentation-everything model are in a frozen state.

[0052] After freezing the weight parameters of the pre-trained SAM visual encoder, the weight parameters are no longer updated with backpropagation during training to improve the stability of the model.

[0053] Figure 6 The detailed process of the pre-trained SAM branch is shown. Taking the input of an image into SAM as an example for polishing, the visual encoder of SAM requires the input image size to be 1024*1024, so firstly, the image to be input (including 1 reference image and any one of the N-1 source images) is scaled and padded to convert the image into a size acceptable to the pre-trained SAM, and the required image is converted into a data type acceptable to the pre-trained SAM through regularization and data type conversion; then, the image with acceptable size and data type is sent to the pre-trained SAM visual encoder with frozen weights, so that its output H, W, C are [64, 64, 256] respectively and feature images with padding (padding), so it is necessary to remove padding and resize the feature image to adjust the feature image to the same size as the coarsest scale feature map F of FPN. input Size The same image, denoted as F SAM Finally, a trainable encoder-decoder convolutional layer is used to reduce F SAM The channel dimension of the image is similar to the coarsest feature image of FPN The channel dimension of is the same as that of the FPN branch, so that it can be fused with the FPN branch. The encoding-decoding convolution layer can be expressed as formulas (2), (3) and (4): (2) (3) (4) Among them, F input It is the coarsest scale feature map of FPN; F SAM is the feature image with padding; for The channel dimension of for The channel dimension of For encoder; For the decoder; represents the convolutional layer; and Represents two deconvolutional layers.

[0054] Step 2022: Send the one reference image, N-1 source images and the first feature image to the pre-trained feature pyramid network, so that the pre-trained feature pyramid network outputs the a reference feature images of different scales and a*(N-1) source feature images of different scales.

[0055] For the convenience of description, a reference feature images of different scales and a*(N-1) source feature images of different scales are recorded as second feature images.

[0056] (2) Feature pyramid branch The input of the FPN branch is 1 reference image and N-1 source images. Each input image is encoded and decoded separately through the convolution method, and the feature fusion between different convolutional layers is realized through the jump connection layer. Finally, a four-layer convolutional coding operation is used to output four feature maps of different sizes with height (H), width (W) and channel (C) dimensions: (H / 8, W / 8, 8C), (H / 4, W / 4, 4C), (H / 2, W / 2, 2C) and (H, W, C).

[0057] For each input image, 4 feature maps of different sizes are generated, that is, there are 4 reference feature maps and 4(N-1) source feature maps .

[0058] Alternatively, see Figure 7 , the pre-trained cost volume-depth map model includes a pre-trained cost volume construction model, a regularization model and a depth prediction model, then step 203 includes the following steps 2031 to 2036: Step 2031, obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales, and sending them to the cost volume construction model, so that the cost volume construction model outputs a first cost volume.

[0059] The general process of the cost volume construction model includes: for each scale, each source feature image generates a similarity volume with the reference feature image, so there are N-1 similarities in total; then the N-1 similarities are aggregated into a cost volume.

[0060] Furthermore, similar bodies can be obtained through homography transformation. The following is a brief introduction to homography transformation: First, we pass each source feature image Warp to the reference feature via homography In this way, N-1 similar bodies are constructed one by one.

[0061] For pixel-level homography transformation, refer to formula (5): For a pixel on the reference feature image and the source feature image, assume that p is a pixel on the reference feature image, and p' is a pixel on the source feature image according to the depth assumption. The pixel obtained after distortion, the pixel p' can be calculated by formula (5): (5) Where p is a pixel on the reference feature image; p' is the source feature image based on the depth hypothesis The pixels obtained after homography transformation; is the depth hypothesis, j represents a certain depth hypothesis layer, and D is the total number of depth hypothesis samples; is the camera rotation matrix from the source feature image to the reference feature image, i represents the serial number of the source feature image; is the camera translation matrix from the source feature image to the reference feature image; is the camera intrinsic parameter of the reference image; is the camera intrinsic parameter of the source image.

[0062] Furthermore, in order to improve the operating efficiency of computer equipment, the N-1 similar bodies can be decomposed along the channel dimension through group association calculation to obtain Slice similar bodies and then The similar bodies are sliced ​​into N-1 similar bodies.

[0063] For each source feature image And the corresponding reference feature image The similarity is calculated by formula (6). Assume that the number of channels of the source feature image and the reference feature image are , for each reference feature map , divided into Group (a picture is divided into groups), then each group has channels; for each source feature map , according to the channel dimension, it can also be groups, each group also has channels. Assume is the source feature map The g-th group of , each source feature map can be expressed by formula (6) With reference feature map Similarity after group association calculation : (6) Among them, j represents a certain depth hypothesis layer; is the number of channels of the source feature image and the reference feature image; is the number of groups divided according to the channel dimension; Represents the reference feature image The g-th group; is the source feature image The g-th group; Represents the calculation vector The inner product of Group association of a group; Group correlation successfully inherits the advantages of traditional correlation cost matching and provides rich similarity measurement features for 3D aggregation networks.

[0064] For each source feature image , are calculated according to the above formula (6) and the reference feature image at the same scale A similar body of N-1 source feature images Can be constructed with a reference feature map A similar entity.

[0065] After obtaining N-1 similar bodies, they are aggregated to form a cost body. Further, the cost body formation process is roughly as follows: After constructing each source feature image With the reference feature image Similar Afterwards, a set of 3D CNNs are used to predict a set of pixel-by-pixel weight matrices , and then these weight matrices are weighted and fused with the similar bodies through formula (7), so that N-1 similar bodies are aggregated into the final cost body ; (7) in, It is the cost body after the aggregation of N-1 similar bodies; To use a set of 3D CNNs to predict a set of pixel-by-pixel weight matrices; The similarity volume constructed for the i-th source feature image and the reference feature image; Step 2032: Regularize the first cost volume to obtain a first probability volume.

[0066] Finally, a 3D U-Net network is used to aggregate the cost volume Regularize the cost The channel dimension is reduced to 1, and then the reference feature image is output The corresponding regularized cost volume , and use the Softmax(·) function to normalize along the depth D dimension to construct a pixel-by-pixel probability volume , the matching cost of the pixel at each depth is converted into a probability value for the final depth prediction.

[0067] Step 2033: Send the first probability volume to a depth prediction model so that the depth prediction model outputs a first depth image.

[0068] Step 2034 : up-sample the first depth image to transform the first depth image into a secondary depth image, wherein the scale of the secondary depth image is consistent with the scale of the secondary coarse-scale feature image.

[0069] Step 2035 : Based on the secondary depth image, the reference feature image of the secondary coarse scale and N-1 source feature images, re-execute steps 2031 to 2034 to obtain a second depth image.

[0070] Step 2036, repeating the above steps 2031 to 2035 until the ultimate depth image is obtained based on a reference image of the finest scale and N-1 source feature images.

[0071] Optionally, in another exemplary embodiment of the present application, the pre-trained depth prediction model is a pre-trained binary search-based depth prediction model, then step 2033 includes: sending the first probability body to the pre-trained binary search-based depth prediction model, so that the pre-trained binary search-based depth prediction model outputs the first depth image.

[0072] The following is an introduction to binary search: Depth prediction of binary search refers to refining the depth search range by binary search stage by stage. Specifically, for the kth search stage, the depth range is divided into two equal boxes, namely ,in represents a box. Since each box represents a range of depth values, and a certain depth value is required in the homography transformation of the feature map, the center point of the box is used to represent the depth sampling value of each box. In this way, the cost volume can be constructed for the two boxes and the depth prediction can be performed. Then the depth sampling values ​​of the two boxes at the kth stage are It can be expressed as the following formula (8): (8) in, are the depth sampling values ​​of the two boxes at the kth stage; Represents the boundaries of the box; The three sides of the two boxes from left to right are After obtaining the probability volume P according to the two depth sampling values, follow the probability volume The D dimension is argmax(·) operated to obtain the depth prediction label of this stage. The depth prediction label indicates that the predicted depth value of the next stage is located in the corresponding box. Therefore, in the k+1 stage, by continuing the binary search in the box where the depth prediction label of the k stage is located, Continue to divide into two new boxes of equal width , and then repeat the above operation until the depth prediction label of the last stage is generated and used as the final depth prediction value.

[0073] The following is an introduction to the fault tolerance box and gradient mask optimization: like Figure 8 As shown in Figure 1, the error-tolerant box (ETB) mechanism refers to adding the same number of equal-width error-tolerant boxes at both ends of the original two boxes in each search stage to enhance the network's tolerance to classification errors. Specifically, after the box is determined by the deep prediction label in the kth stage, the box is divided into two for the k+1th stage binary search. Assuming that D is the final number of boxes (D is a small even number), extra boxes are filled on both sides of the original two boxes. Therefore, the box at stage k+1 can be expressed as , the number of boundaries of the box can be expressed as . Still using the center sample of the box as the depth hypothesis value, we can still obtain D depth sampling values ​​through formula (8): In order to unify the number of boxes in each stage, the initial depth range is also divided into D boxes in the first stage, so the width of each box in the initial stage is , in the kth stage, the box width is Adding a tolerance box increases the depth search range, which increases the probability that the true depth value is within the depth search range. Therefore, the network can optimize the depth labels that are incorrectly predicted within a certain range and reduce the accumulation of errors. Using D=4, that is, adding a tolerance box on each side, the network's tolerance to prediction errors can be enhanced without increasing video memory consumption.

[0074] The gradient mask optimization mechanism is a mask mechanism that determines whether the true depth value exceeds the range of the depth box. The network is supervised by the real depth map. In the kth stage, the real depth value is projected into the box to determine which box the real depth value is in. Then, a one-hot label can be constructed based on the center sampling value to calculate the loss. However, according to the previous description, the deviation of the classification process may cause the true depth value to be outside the range of the depth box. Therefore, it is impossible to construct a valid one-hot label, which affects the back propagation optimization of the network. Therefore, consider introducing a mask mechanism to determine whether the true depth value of each stage is within the boundary range of D depth boxes. If it exceeds the boundary range of the depth box, it is considered to be an invalid pixel. After filtering the mask map, only valid pixels are retained for back propagation to update the gradient parameters. The masked pixels terminate the back propagation of this stage and subsequent stages. Suppose the true depth value of a pixel is , the boundary of the current stage is , then formula (9) can be used to determine whether the pixel is a valid pixel: (9) in, is the boundary of the current stage; is the true depth value of the pixel; By introducing these two mechanisms, most of the pixels can participate in the learning and gradient optimization process, which greatly enhances the performance and stability of the network.

[0075] Alternatively, see Fig. 9In another exemplary embodiment of the present application, the depth prediction model based on binary search is trained through the following steps 2041 to 2047: Step 2041, collect image data sets of different scenes to obtain a training set.

[0076] Step 2042: Send the first cost volume of each reference image in the training set to the initial binary search-based depth prediction model, so that the initial binary search-based depth prediction model outputs a predicted depth image.

[0077] Step 2043, sending the depth image predicted by the training data set at each stage and the actual depth image to the adaptive weight adjustment loss function, so that the adaptive weight adjustment loss function calculates the loss value of each stage, and reflects and propagates based on the loss value to adjust the weight parameters of the initial binary search-based depth prediction model.

[0078] Step 2044, for the predicted depth map corresponding to each image in the training set, execute the above steps 2041 to 2043 one by one until the loss value no longer decreases, stop training, and determine the weight parameter at this time as the optimal weight parameter model of the depth prediction model based on binary search, so as to obtain the depth prediction model based on binary search.

[0079] Finally, the accuracy of the binary search-based depth prediction model can be tested using any data in the test set. When the accuracy is less than the expected result, steps 2041 to 2044 are re-executed. When the accuracy is greater than or equal to the expected result, no further training is required.

[0080] In addition, the training set can use a public training data set or build it yourself. Similarly, the test set can be collected by yourself or use a public one.

[0081] See also Fig.10 , introduce the adaptive weight adjustment loss function: In the MVS task dataset, it is found that samples with richer textures generally have smaller loss values, while samples with weaker textures generally have larger loss values. Pixels with smaller loss values ​​are called simple samples, and samples with larger loss values ​​are called difficult samples.

[0082] The adaptive weight adjustment loss function is used to calculate the loss value of the depth map generated at each stage of the network and the true depth map, and perform backpropagation to optimize the network parameters to achieve the optimization of the depth prediction of samples in weak texture areas. This loss function has two main operations: adaptive classification of easy / difficult samples, and dynamic transfer of sample attention.

[0083] Adaptive classification of easy / hard samples Based on the standard cross entropy loss, the sample adaptive classification loss function (Adaptive Weight Adjustment Loss, referred to as AWA-Loss) is expressed as follows (10): (10) in, is the weight coefficient of the cross entropy loss for each pixel sample; is the valid pixel set determined by formula (9); is the probability of the label location for each pixel sample; is the standard cross entropy loss.

[0084] SAC-Loss can be calculated based on The size of the adaptive feature vector can be used to distinguish simple or difficult samples and reduce the relative loss value of simple samples, thereby giving relatively greater attention to difficult samples, making the model more focused on optimizing difficult samples.

[0085] The following is a brief introduction to the dynamic transfer of sample attention: Since only adaptive classification of samples will cause SAC-Loss to ignore the optimization of simple samples, and the early loss value of training is generally high, it cannot be used as the classification basis for simple or difficult samples. Therefore, a dynamic transfer strategy of sample attention from easy to difficult is proposed to further optimize the loss function. The final AWA-Loss can be expressed as formula (11): (11) in, It is a weight reduction function that decreases monotonically with the training process; It is a weighting function that increases monotonically with the training process; is the current epoch.

[0086] and is defined as formula (12) and (13): (12) (13) in, Indicates the total number of epochs; and is the weight function curve.

[0087] Alternatively, see Fig.11In another exemplary embodiment of the present application, after the above step 204, the method further includes the following step 300: Step 300: Filter the m final depth images to obtain m filtered depth images.

[0088] This application uses two constraints, geometric consistency and photometric consistency, to filter pixels in the depth map that exceed a predetermined threshold.

[0089] The geometric consistency constraint is used to measure the depth consistency between m depth maps. Projection onto source image On, then through Depth estimation ,Will Reproject back to the reference image. If the reprojection coordinates and reprojection depth If formula (14) and formula (15) are satisfied, the depth estimation d1 of p1 is called dual-view Figure 1 In the experiment, all depths should be at least three views. Figure 1 To.

[0090] (14) (15) in,

[0091] is the pixel on the reference feature; is the reprojection depth; is the depth value.

[0092] The photometric consistency constraint filters low-confidence pixel depth values ​​through the confidence map. The confidence map is obtained by the probability body in step 203. , the maximum value is obtained along the depth D dimension, and the maximum probability value of each pixel is the confidence value. Depth predictions with confidence values ​​lower than 0.8 are filtered out. To further suppress reconstruction noise, the visible view of each pixel is determined in the filtering step, and all reprojected depth The average value of is taken as the final depth estimate of the pixel.

[0093] The filtered m depth images are sent to the point cloud fusion algorithm as new depth images.

[0094] Optionally, in another exemplary embodiment of the present application, the point cloud fusion algorithm is a visibility-based depth map fusion algorithm, and step 206 includes: sending the m depth images to the visibility-based depth map fusion algorithm, so that the visibility-based depth map fusion algorithm outputs a three-dimensional point cloud model.

[0095] Depth occlusions and violations between different viewpoints are minimized through a visibility-based depth map fusion algorithm, thereby integrating depth maps from different views into a unified point cloud representation.

[0096] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image-based three-dimensional reconstruction related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional reconstruction method based on an image is implemented.

[0097] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0099] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0100] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0102] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0103] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A three-dimensional reconstruction method based on an image, characterized in that: include: Acquire m original images of a target object, wherein the m original images are images taken from different angles of the target object, and the target object is an object to be three-dimensionally reconstructed; Obtain one image from the m original images as a reference image, and then obtain N-1 images as source images; Sending the one reference image and N-1 source images to a multi-scale feature extraction model, wherein the multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, where a is the number of layers of the pre-trained feature pyramid network; Sending the a reference feature images of different scales and a*(N-1) source feature images of different scales to a pre-trained cost volume-depth map model, so that the cost volume-depth map model outputs a final depth image; For each of the m original images, take turns as a reference image, and perform the steps of "obtaining one image from the m original images as a reference image, and then obtaining N-1 images as source images" to "so that the cost volume-depth map model outputs a final depth image" to obtain m final depth images; The m ultimate depth images are sent to a point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object.

2. The image-based three-dimensional reconstruction method according to claim 1, characterized in that: The weight parameters of the visual encoder of the pre-trained segmentation-everything model are in a frozen state.

3. The image-based three-dimensional reconstruction method according to claim 2, characterized in that: The step of sending the reference image and N-1 source images to a multi-scale feature extraction model, wherein the multi-scale feature extraction model includes a pre-trained segmentation model and a pre-trained feature pyramid network, so that the multi-scale feature extraction model outputs a reference feature images of different scales and a*(N-1) source feature images of different scales, including: Sending the one reference image and N-1 source images to the pre-trained segmentation model so that the segmentation model outputs a first feature image corresponding to each of the one reference image and the N-1 source images; The one reference image, N-1 source images and the first feature image are sent to the pre-trained feature pyramid network, so that the pre-trained feature pyramid network outputs the a reference feature images of different scales and a*(N-1) source feature images of different scales.

4. The image-based three-dimensional reconstruction method according to claim 1, characterized in that: The pre-trained cost volume-depth map model includes a pre-trained cost volume construction model, a regularization model and a depth prediction model, and the a reference feature images of different scales and a*(N-1) source feature images of different scales are sent to the pre-trained cost volume-depth map model so that the cost volume-depth map model outputs a final depth image, including: Obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales, and sending them to the cost volume construction model, so that the cost volume construction model outputs a first cost volume; Regularizing the first cost body to obtain a first probability body; Sending the first probability volume to a depth prediction model so that the depth prediction model outputs a first depth image; Upsampling the first depth image to transform the first depth image into a secondary depth image, wherein a scale of the secondary depth image is consistent with a scale of a secondary coarse-scale feature image; Based on the secondary depth image, the one reference feature image of the secondary coarse scale and the N-1 source feature images, re-execute the steps of "obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales" to "the scale of the secondary depth image is consistent with the scale of the feature image of the secondary coarse scale", to obtain a second depth image; Repeat the steps of "obtaining a reference feature image of the coarsest scale and N-1 source feature images from the a reference feature images of different scales and a*(N-1) source feature images of different scales" to "obtaining a second depth image" until the ultimate depth image is obtained based on the reference image of the finest scale and the N-1 source feature images.

5. The image-based 3D reconstruction method according to claim 4, characterized in that: If the pre-trained depth prediction model is a pre-trained binary search-based depth prediction model, sending the first probability body to the depth prediction model so that the depth prediction model outputs a first depth image includes: sending the first probability body to the pre-trained binary search-based depth prediction model so that the pre-trained binary search-based depth prediction model outputs the first depth image.

6. The image-based three-dimensional reconstruction method according to claim 5, characterized in that: The depth prediction model based on binary search is trained by the following method: Collect image data sets of different scenes to obtain training sets; Sending the first cost volume of each reference image in the training set to the initial binary search-based depth prediction model, so that the initial binary search-based depth prediction model outputs a predicted depth image; Sending the depth image predicted by the training data set at each stage and the real depth image to the adaptive weight adjustment loss function, so that the adaptive weight adjustment loss function calculates the loss value of each stage, and reflects and propagates based on the loss value to adjust the weight parameters of the initial binary search-based depth prediction model; For the predicted depth map corresponding to each image in the training set, the above-mentioned "collecting image data sets of different scenes" to "adjusting the weight parameters of the initial depth prediction model based on binary search based on reflection propagation of the loss value" are executed one by one until the loss value no longer decreases, and the training is stopped. The weight parameters at this time are determined as the optimal weight parameter model of the depth prediction model based on binary search, and the depth prediction model based on binary search is obtained.

7. The image-based three-dimensional reconstruction method according to any one of claims 1 to 6, characterized in that: If the point cloud fusion algorithm is a visibility-based depth map fusion algorithm, then sending the m ultimate depth images to the point cloud fusion algorithm so that the point cloud fusion algorithm outputs a three-dimensional point cloud model of the target object includes: sending the m depth images to the visibility-based depth map fusion algorithm so that the visibility-based depth map fusion algorithm outputs a three-dimensional point cloud model.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image-based three-dimensional reconstruction method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image-based three-dimensional reconstruction method described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image-based three-dimensional reconstruction method described in any one of claims 1 to 7 is implemented.

Images