Three-dimensional reconstruction method and system based on contrast learning and dense comparison, and medium

Abstract

This invention discloses a 3D reconstruction method, system, and medium based on contrastive learning and dense comparison. The method involves: acquiring a 2D query image of an object and its corresponding 3D supporting shape to form an original dataset; constructing a 3D reconstruction model, including a contrastive learning module and a feature fusion module; pre-training the contrastive learning module and fixing its parameters; iteratively training the 3D reconstruction model on the original dataset using gradient descent to obtain a trained 3D reconstruction model; and inputting the query object image into the trained 3D reconstruction model to obtain the predicted 3D shape. The 3D reconstruction model of this invention can effectively fuse features extracted from 2D images and features extracted from 3D shapes, extracting features closely related to 3D information from the image, thereby achieving high-precision 3D reconstruction results.

Description

Technical Field

[0001] This invention belongs to the technical field of three-dimensional reconstruction, specifically relating to a three-dimensional reconstruction method, system, and medium based on contrastive learning and dense comparison. Background Technology

[0002] 3D reconstruction is a crucial topic in computer vision, enabling the creation of 3D models from images and revealing the 3D position / coordinates of each point within the image. 3D reconstruction methods have applications in numerous fields, including computer-aided geometric design, computer graphics, computer animation, computer vision, medicine, virtual reality, and digital media. However, most existing 3D reconstruction techniques require multi-view images, complex camera equipment and environments, resulting in high costs. Existing single-view 3D reconstruction techniques have not yet achieved satisfactory results, or require large training datasets and exhibit poor generalization ability, particularly weak reconstruction capabilities for new object categories. Therefore, many existing 3D reconstruction methods have not yet reached satisfactory application standards. Improving the accuracy and generalization ability of 3D reconstruction, and reducing the cost of these techniques, are pressing challenges that need to be addressed. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a 3D reconstruction method, system, and medium based on contrastive learning and dense comparison. By constructing a 3D reconstruction model, the contrastive learning module is pre-trained, its parameters are fixed, and features are extracted from 2D images and 3D shapes. Then, a feature fusion module is used to fuse the extracted features and output the 3D reconstruction result. This invention can effectively align features in 2D images and 3D shapes, enabling the 2D encoder to learn more features closely related to 3D shapes, greatly improving the accuracy of 3D reconstruction and the generalization ability of the model.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] On the one hand, the present invention provides a three-dimensional reconstruction method based on contrastive learning and dense comparison, comprising the following steps:

[0006] The original dataset is constructed by collecting two-dimensional query images of objects and their corresponding three-dimensional supporting shapes;

[0007] Construct a 3D reconstruction model, including a contrastive learning module and a feature fusion module;

[0008] The contrastive learning module is pre-trained, and its parameters are then fixed, including:

[0009] Contrastive learning sample set is constructed on the original dataset. Two-dimensional encoder and three-dimensional encoder are used to encode the samples in the contrastive learning sample set respectively, and the two-dimensional feature vector and three-dimensional feature vector of the sample are obtained and projected into the same embedding space.

[0010] Construct the loss function of the contrastive learning module, calculate the loss value and update the parameters of the contrastive learning module in reverse until convergence, fix the parameters of the two-dimensional encoder and the three-dimensional encoder in the contrastive learning module to obtain the pre-trained contrastive learning module.

[0011] The 3D reconstruction model is iteratively trained on the original dataset using gradient descent, including:

[0012] The original dataset is input into a pre-trained contrastive learning model with fixed parameters to obtain the two-dimensional and three-dimensional feature vectors of the object.

[0013] Based on the object's two-dimensional and three-dimensional feature vectors, the feature fusion module performs a Cross-Attention operation to generate Q, K, and V matrices. It then uses Attention operations to re-represent the object's two-dimensional and three-dimensional features and finally uses dense comparison to fuse the features to obtain the object's fused features.

[0014] Construct a binary cross-entropy loss function for the 3D reconstruction model, calculate the binary cross-entropy loss value and update the 3D reconstruction model parameters until the binary cross-entropy loss function converges, and obtain the trained 3D reconstruction model.

[0015] Input the image of the object being queried into the trained 3D reconstruction model to obtain the predicted 3D shape.

[0016] As a preferred technical solution, the original dataset is represented as follows:

[0017]

[0018] Where k represents the object category, I ik S is the two-dimensional query image of the i-th object of category k in the original dataset. ik Let N be the 3D supporting shape of the i-th object of category k in the original dataset. k N represents the number of objects of category k in the original dataset. This represents the total number of objects in the original dataset.

[0019] The two-dimensional query images are captured using a high-resolution camera, and the shooting angle is uniform during capture.

[0020] As a preferred technical solution, the step of constructing a contrastive learning sample set on the original dataset, and using a two-dimensional encoder and a three-dimensional encoder to encode the samples in the contrastive learning sample set respectively, obtaining the two-dimensional feature vectors and three-dimensional feature vectors of the samples and projecting them onto the same embedding space, specifically involves:

[0021] Construct a contrastive learning sample set on the original dataset: randomly select Nc categories from the categories in the original dataset, and randomly select a pair of 2D query images and their corresponding 3D support shapes from each category to form the contrastive learning sample set, represented as:

[0022]

[0023] in, This represents the two-dimensional query image of the i-th sample in the comparison learning sample set. This represents the three-dimensional support shape of the i-th sample in the contrastive learning sample set;

[0024] The two-dimensional query images of the samples in the contrastive learning sample set are input into a two-dimensional encoder for encoding, resulting in a two-dimensional feature vector of the sample, represented as:

[0025]

[0026] Among them, E 2d This represents a two-dimensional encoder. Represents the two-dimensional feature vector of the i-th sample;

[0027] The 3D support shapes of the samples in the contrastive learning sample set are input into the 3D encoder for encoding, resulting in the 3D feature vectors of the samples, represented as:

[0028]

[0029] Among them, E 3d Indicates a 3D encoder. Represents the three-dimensional feature vector of the i-th sample;

[0030] Projecting the two-dimensional and three-dimensional feature vectors of the sample into the same embedding space yields the two-dimensional and three-dimensional embedding features of the sample, represented as:

[0031]

[0032]

[0033] in, Represents two-dimensional embedded features, P represents the 3D embedding feature. 2d P 3d It is a single-layer, fully connected network.

[0034] As a preferred technical solution, the loss function of the contrastive learning module is expressed as:

[0035]

[0036] in,

[0037]

[0038]

[0039] in, Let the loss be the two-dimensional query image for the i-th sample. Let be the 3D support shape loss of the i-th sample, sim be the cosine distance, and τ be the temperature coefficient.

[0040] As a preferred technical solution, the re-representation of the two-dimensional and three-dimensional features of the object specifically includes:

[0041] Cross-Attention operation is performed on the two-dimensional and three-dimensional feature vectors of the object to generate Q, K, and V matrices:

[0042] For two-dimensional eigenvectors:

[0043]

[0044] For three-dimensional feature vectors:

[0045]

[0046] Among them, W Q W K W V ∈R d×d All are trainable parameters; the shape of both the two-dimensional and three-dimensional feature vectors of the object is r. 3 ×d,r 3 d is the feature resolution, and d is the feature dimension;

[0047] The two-dimensional and three-dimensional features of an object are re-represented using the Attention operation. The two-dimensional features of the object are:

[0048]

[0049]

[0050]

[0051] The three-dimensional features of the object are:

[0052]

[0053]

[0054]

[0055] in, For the re-represented two-dimensional features, The three-dimensional features are represented as a new representation, where d is the feature dimension and FC represents a fully connected layer.

[0056] As a preferred technical solution, the method of using dense comparison to perform feature fusion to obtain the fused features of the object specifically includes:

[0057] The re-represented 3D shape features The shape is deformed into 1×r 3 ×d;

[0058] Deformed Through a fully connected layer:

[0059]

[0060] Output a global descriptor of shape 1×d;

[0061] Repeat the output of this descriptor r 3 Next, we obtain F″′ S The shape is r 3 ×d;

[0062] F″′ S With the re-represented two-dimensional features By splicing the data, we obtain the splicing feature F. qS The shape is r 3 ×2d;

[0063] splicing feature F qS Inputting into a 3D convolutional layer Conv3D(2d, d, 3), the output is a fused feature F″′. q The shape is r 3 ×d.

[0064] As a preferred technical solution, the binary cross-entropy loss function of the three-dimensional reconstruction model is expressed as:

[0065]

[0066] Where n is the number of pixels in the fused feature, y i ∈{0,1} represents the actual occupancy of the i-th pixel in the fused feature, where 1 indicates that the pixel is occupied and 0 indicates that the pixel is empty; p i This indicates the probability that the i-th pixel in the output is occupied.

[0067] As a preferred technical solution, during the pre-training process of the contrastive learning module, the temperature coefficient τ is set to 0.1, the iteration is 20 epochs, and the learning rate is 0.001.

[0068] During the iterative training of the 3D reconstruction model, the number of iterations was set to 100,000, the batch size to 32, and the learning rate to 0.0001, which was halved every 20,000 iterations; p was set to i When the value is greater than 0.3, it is predicted that the i-th pixel is occupied by an object.

[0069] On the other hand, the present invention provides a three-dimensional reconstruction system based on contrastive learning and dense comparison, which is applied to the above-mentioned three-dimensional reconstruction method based on contrastive learning and dense comparison, including a data acquisition module, a model building module, a pre-training module, an iterative training module and a result prediction module;

[0070] The data acquisition module is used to acquire two-dimensional query images of objects and their corresponding three-dimensional supporting shapes to form the original dataset;

[0071] The model building module is used to build a 3D reconstruction model, including a contrast learning module and a feature fusion module;

[0072] The pre-training module is used to pre-train the contrastive learning module, and after completion, the parameters of the contrastive learning module are fixed, including:

[0073] Contrastive learning sample set is constructed on the original dataset. Two-dimensional encoder and three-dimensional encoder are used to encode the samples in the contrastive learning sample set respectively, and the two-dimensional feature vector and three-dimensional feature vector of the sample are obtained and projected into the same embedding space.

[0074] Construct the loss function of the contrastive learning module, calculate the loss value and update the parameters of the contrastive learning module in reverse until convergence, fix the parameters of the two-dimensional encoder and the three-dimensional encoder in the contrastive learning module to obtain the pre-trained contrastive learning module.

[0075] The iterative training module is used to iteratively train the 3D reconstruction model on the original dataset using the gradient descent method, including:

[0076] The original dataset is input into a pre-trained contrastive learning model with fixed parameters to obtain the two-dimensional and three-dimensional feature vectors of the object.

[0077] Based on the object's two-dimensional and three-dimensional feature vectors, the feature fusion module performs a Cross-Attention operation to generate Q, K, and V matrices. It then uses Attention operations to re-represent the object's two-dimensional and three-dimensional features and finally uses dense comparison to fuse the features to obtain the object's fused features.

[0078] Construct a binary cross-entropy loss function for the 3D reconstruction model, calculate the binary cross-entropy loss value and update the 3D reconstruction model parameters until the binary cross-entropy loss function converges, and obtain the trained 3D reconstruction model.

[0079] The result prediction module is used to input the image of the queried object into the trained 3D reconstruction model to obtain the predicted 3D shape.

[0080] In another aspect, the present invention provides a computer-readable storage medium storing a program, characterized in that, when the program is executed by a processor, it implements the above-described three-dimensional reconstruction method based on contrastive learning and dense comparison.

[0081] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0082] 1. This invention uses neural networks and deep learning methods. After training, the model only needs one image to achieve a high-precision 3D reconstruction effect. Compared with traditional 3D reconstruction methods, this method does not require high-cost equipment and facilities.

[0083] 2. This invention utilizes contrastive learning technology to align the outputs of the 3D encoder and the 2D encoder, enabling the 2D encoder to learn more features closely related to the 3D shape. Compared with traditional encoder-decoder networks, this invention can greatly improve the accuracy of 3D reconstruction.

[0084] 3. In addition to using real results as supervision, the network model constructed in this invention also introduces support shape as prior; this prior information can effectively guide the 2D encoder to learn general rules, thereby improving the model's generalization ability and achieving good reconstruction results for objects of unseen categories.

[0085] 4. The contrast learning module and feature fusion module proposed in this invention can effectively integrate features extracted from two-dimensional images and features extracted from three-dimensional shapes. Therefore, after training, even without providing a three-dimensional support shape, features closely related to three-dimensional information can be extracted from a single image, thereby achieving a high-precision three-dimensional reconstruction effect. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 This is a flowchart illustrating the three-dimensional reconstruction method based on contrastive learning and dense comparison in an embodiment of the present invention.

[0088] Figure 2 This is a flowchart of the iterative training of the 3D reconstruction model in an embodiment of the present invention;

[0089] Figure 3 This is a network structure diagram of the three-dimensional reconstruction model in an embodiment of the present invention;

[0090] Figure 4 This is a schematic diagram illustrating the use of Attention operations to re-represent the two-dimensional and three-dimensional features of an object in an embodiment of the present invention.

[0091] Figure 5 This is a schematic diagram illustrating feature fusion using a dense comparison method in an embodiment of the present invention;

[0092] Figure 6 This is a schematic diagram of the structure of the 3D reconstruction system based on contrastive learning and dense comparison in an embodiment of the present invention;

[0093] Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0094] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0095] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0096] like Figure 1 As shown, this embodiment of the 3D reconstruction method based on contrastive learning and dense comparison includes the following steps:

[0097] S1. The original dataset, consisting of two-dimensional query images and three-dimensional supporting shapes of the collected objects, is represented as follows:

[0098]

[0099] Where k represents the object category, such as k∈{0(car), 1(airplane), ..., n(...)}, I ik S is the two-dimensional query image of the i-th object of category k in the original dataset. ik Let N be the 3D supporting shape of the i-th object of category k in the original dataset. k This represents the number of objects of category k in the original dataset. This represents the total number of objects in the original dataset. The two-dimensional query images were captured using high-resolution imaging equipment to ensure clarity and a consistent shooting angle. For example, using a 128×128 image resolution and a 32×32×32 shape resolution, a three-channel color two-dimensional query image I... ik The format is 3×128×128, and the 3D supported shape is S. ik The format is 1×32×32×32; open-source datasets such as ShapeNet and ModelNet can also be used as the original dataset.

[0100] S2. Construct a 3D reconstruction model, including a contrastive learning module (CMCL module) and a feature fusion module (CMFF module); the CMCL module includes a 2D encoder and a 3D encoder for extracting the object's feature vectors; the CMFF module includes a cross-attention network and a decoder for acquiring the fused features of the object to predict its 3D shape; for example... Figure 3 As shown, the 3D reconstruction model in this embodiment includes a contrastive learning module and three feature fusion modules.

[0101] S3. The next step is to train the 3D reconstruction model, such as... Figure 2 As shown, the contrastive learning module is first pre-trained, and then its parameters are fixed, including:

[0102] S31. Construct a contrastive learning sample set on the original dataset. Use a two-dimensional encoder and a three-dimensional encoder to encode the samples in the contrastive learning sample set, respectively, to obtain the two-dimensional feature vectors and three-dimensional feature vectors of the samples, and project them into the same embedding space, specifically:

[0103] Construct a contrastive learning sample set on the original dataset: randomly select N from the categories of the original dataset. c For each of the following categories, a pair of two-dimensional query images and their corresponding three-dimensional supporting shapes are randomly selected to form a contrastive learning sample set, represented as follows:

[0104]

[0105] in, This represents the two-dimensional query image of the i-th sample in the comparison learning sample set. This represents the three-dimensional support shape of the i-th sample in the contrastive learning sample set;

[0106] The two-dimensional query images of the samples in the contrastive learning sample set are input into a two-dimensional encoder for encoding, resulting in a two-dimensional feature vector of the sample, represented as:

[0107]

[0108] Among them, E 2d This represents a two-dimensional encoder. Represents the two-dimensional feature vector of the i-th sample;

[0109] The 3D support shapes of the samples in the contrastive learning sample set are input into the 3D encoder for encoding, resulting in the 3D feature vectors of the samples, represented as:

[0110]

[0111] Among them, E 3d Indicates a 3D encoder. Represents the three-dimensional feature vector of the i-th sample;

[0112] Projecting the two-dimensional and three-dimensional feature vectors of the sample into the same embedding space yields the two-dimensional and three-dimensional embedding features of the sample, represented as:

[0113]

[0114]

[0115] in, Represents two-dimensional embedded features, P represents the 3D embedding feature. 2d P 3d It is a single-layer, fully connected network.

[0116] S32. Construct the loss function of the contrastive learning module, calculate the loss value and update the parameters of the contrastive learning module in reverse until convergence, fix the parameters of the two-dimensional encoder and the three-dimensional encoder in the contrastive learning module, and obtain the pre-trained contrastive learning module.

[0117] The loss function for the contrastive learning module is expressed as follows:

[0118]

[0119] In the formula,

[0120]

[0121]

[0122] in, Let the loss be the two-dimensional query image for the i-th sample. Let be the 3D support shape loss of the i-th sample, sim be the cosine distance, and τ be the temperature coefficient.

[0123] In this embodiment, during the pre-training process of the contrast learning module, the temperature coefficient τ is set to 0.1, the iteration is 20 epochs, and the learning rate is 0.001.

[0124] S4. Next, the 3D reconstruction model is iteratively trained on the original dataset using gradient descent, including:

[0125] S41. Input the original dataset into a pre-trained contrastive learning model with fixed parameters to obtain the two-dimensional and three-dimensional feature vectors of the object.

[0126] S42. Based on the two-dimensional and three-dimensional feature vectors of the object, the feature fusion module performs a Cross-Attention operation to generate Q, K, and V matrices, and uses Attention operations to re-represent the two-dimensional and three-dimensional features of the object. Then, the dense comparison method is used to perform feature fusion to obtain the fused features of the object.

[0127] like Figure 3 As shown, in this embodiment, three feature fusion modules are used to fuse the input features. The input shapes of the three feature fusion modules are 128×4×4×4, 64×8×8×8 and 32×16×16×16, respectively.

[0128] For any one of the feature fusion modules, such as Figure 4 As shown, the two-dimensional and three-dimensional features of the object are re-represented, specifically as follows:

[0129] Cross-Attention operation is performed on the two-dimensional and three-dimensional feature vectors of the object to generate Q, K, and V matrices:

[0130] For the two-dimensional feature vector of an object:

[0131]

[0132] For the three-dimensional feature vector of an object:

[0133]

[0134] Among them, W Q W K W V All are trainable parameters; the shapes of the object's two-dimensional and three-dimensional feature vectors are both r. 3 ×d,r 3 d is the feature resolution, and d is the feature dimension;

[0135] The two-dimensional and three-dimensional features of an object are re-represented using the Attention operation. For the two-dimensional features of the object:

[0136]

[0137]

[0138]

[0139] For the three-dimensional features of an object:

[0140]

[0141]

[0142]

[0143] in, For the re-represented two-dimensional features, The three-dimensional features are represented as a new representation, where d is the feature dimension and FC represents a fully connected layer.

[0144] like Figure 5 As shown, the dense comparison method is used to perform feature fusion to obtain the fused features of the object, specifically:

[0145] The re-represented 3D shape features The shape is deformed into 1×r 3 ×d;

[0146] Deformed Through a fully connected layer:

[0147]

[0148] Output a global descriptor of shape 1×d;

[0149] Repeat the output of this descriptor r 3 Next, we obtain F″′ S The shape is r 3 ×d;

[0150] F″′ S With the re-represented two-dimensional features By splicing the data, we obtain the splicing feature F. qs The shape is r 3 ×2d;

[0151] splicing feature F qS Input a 3D convolutional layer Conv3D(2d, d, 3), output fused features F″′q The shape is r 3 ×d.

[0152] S43. Construct the binary cross-entropy loss function for the 3D reconstruction model, calculate the binary cross-entropy loss value and update the parameters of the 3D reconstruction model until the binary cross-entropy loss function converges, and obtain the trained 3D reconstruction model.

[0153] The binary cross-entropy loss function is expressed as:

[0154]

[0155] Where n is the number of pixels in the fused feature, for example, if the resolution of the fused feature is 1×32×32×32, then n=32×32×32=32768; y i ∈{0,1} represents the actual occupancy of the i-th pixel in the fused feature, where 1 indicates that the pixel is occupied and 0 indicates that the pixel is empty; p i This represents the probability that the i-th pixel in the output is occupied. The entire 3D reconstruction model can be considered as a function: P = F. θ (q, S), where q is the two-dimensional query image, S is the three-dimensional support shape used as prior, and P is the three-dimensional reconstruction result; if the resolution of the obtained fused features is 1×32×32×32, then the three-dimensional shape of P is 1×32×32×32, where the output probability value p of each pixel is... i This indicates the probability that the pixel is occupied by an object.

[0156] In this embodiment, during the iterative training of the 3D reconstruction model, the number of iterations is set to 100,000, the batch size to 32, and the learning rate to 0.0001, which is halved every 20,000 iterations; p is set to i When the value is greater than 0.3, it is predicted that the i-th pixel is occupied by an object.

[0157] S5. Input the image of the object being queried into the trained 3D reconstruction model to obtain the predicted 3D shape.

[0158] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0159] Based on the same ideas as the 3D reconstruction method based on contrastive learning and dense comparison in the above embodiments, the present invention also provides a 3D reconstruction system based on contrastive learning and dense comparison, which can be used to execute the above-described 3D reconstruction method based on contrastive learning and dense comparison. For ease of explanation, the structural diagram of the 3D reconstruction system embodiment based on contrastive learning and dense comparison only shows the parts relevant to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0160] like Figure 6 As shown, another embodiment of the present invention provides a three-dimensional reconstruction system based on contrastive learning and dense comparison, including a data acquisition module, a model building module, a pre-training module, an iterative training module, and a result prediction module;

[0161] The data acquisition module is used to collect two-dimensional query images of objects and their corresponding three-dimensional supporting shapes to form the original dataset.

[0162] The model building module is used to build 3D reconstruction models, including a contrastive learning module and a feature fusion module;

[0163] The pre-training module is used to pre-train the contrastive learning module. After completion, the parameters of the contrastive learning module are fixed, including:

[0164] Contrastive learning sample set is constructed on the original dataset. Two-dimensional encoder and three-dimensional encoder are used to encode the samples in the contrastive learning sample set respectively, and the two-dimensional feature vector and three-dimensional feature vector of the sample are obtained and projected into the same embedding space.

[0165] Construct the loss function of the contrastive learning module, calculate the loss value and update the parameters of the contrastive learning module in reverse until convergence, fix the parameters of the two-dimensional encoder and the three-dimensional encoder in the contrastive learning module to obtain the pre-trained contrastive learning module.

[0166] The iterative training module is used to iteratively train the 3D reconstruction model on the original dataset using gradient descent, including:

[0167] The original dataset is input into a pre-trained contrastive learning model with fixed parameters to obtain the two-dimensional and three-dimensional feature vectors of the object.

[0168] Based on the object's two-dimensional and three-dimensional feature vectors, the feature fusion module performs a Cross-Attention operation to generate Q, K, and V matrices. It then uses Attention operations to re-represent the object's two-dimensional and three-dimensional features and finally uses dense comparison to fuse the features to obtain the object's fused features.

[0169] Construct a binary cross-entropy loss function for the 3D reconstruction model, calculate the binary cross-entropy loss value and update the 3D reconstruction model parameters until the binary cross-entropy loss function converges, and obtain the trained 3D reconstruction model.

[0170] The result prediction module is used to input the image of the queried object into the trained 3D reconstruction model to obtain the predicted 3D shape.

[0171] It should be noted that the 3D reconstruction system based on contrastive learning and dense comparison of the present invention corresponds one-to-one with the 3D reconstruction method based on contrastive learning and dense comparison of the present invention. The technical features and beneficial effects described in the embodiments of the 3D reconstruction method based on contrastive learning and dense comparison described above are applicable to the embodiments of the 3D reconstruction system based on contrastive learning and dense comparison. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0172] Furthermore, in the above embodiments of the 3D reconstruction system based on contrastive learning and dense comparison, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the 3D reconstruction system based on contrastive learning and dense comparison can be divided into different program modules to complete all or part of the functions described above.

[0173] like Figure 7 As shown, in one embodiment, a computer-readable storage medium is provided, storing a program in a memory. When the program is executed by a processor, it implements the aforementioned 3D reconstruction method based on contrastive learning and dense comparison, specifically:

[0174] The original dataset is constructed by collecting two-dimensional query images of objects and their corresponding three-dimensional supporting shapes;

[0175] Construct a 3D reconstruction model, including a contrastive learning module and a feature fusion module;

[0176] The contrastive learning module is pre-trained, and its parameters are then fixed, including:

[0177] Contrastive learning sample set is constructed on the original dataset. Two-dimensional encoder and three-dimensional encoder are used to encode the samples in the contrastive learning sample set respectively, and the two-dimensional feature vector and three-dimensional feature vector of the sample are obtained and projected into the same embedding space.

[0178] Construct the loss function of the contrastive learning module, calculate the loss value and update the parameters of the contrastive learning module in reverse until convergence, fix the parameters of the two-dimensional encoder and the three-dimensional encoder in the contrastive learning module to obtain the pre-trained contrastive learning module.

[0179] The 3D reconstruction model is iteratively trained on the original dataset using gradient descent, including:

[0180] The original dataset is input into a pre-trained contrastive learning model with fixed parameters to obtain the two-dimensional and three-dimensional feature vectors of the object.

[0181] Based on the object's two-dimensional and three-dimensional feature vectors, the feature fusion module performs a Cross-Attention operation to generate Q, K, and V matrices. It then uses Attention operations to re-represent the object's two-dimensional and three-dimensional features and finally uses dense comparison to fuse the features to obtain the object's fused features.

[0182] Construct a binary cross-entropy loss function for the 3D reconstruction model, calculate the binary cross-entropy loss value and update the 3D reconstruction model parameters until the binary cross-entropy loss function converges, and obtain the trained 3D reconstruction model.

[0183] Input the image of the object being queried into the trained 3D reconstruction model to obtain the predicted 3D shape.

[0184] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A 3D reconstruction method based on contrastive learning and dense comparison, characterized in that, Includes the following steps: The original dataset is constructed by collecting two-dimensional query images of objects and their corresponding three-dimensional supporting shapes; Construct a 3D reconstruction model, including a contrastive learning module and a feature fusion module; The contrastive learning module is pre-trained, and its parameters are then fixed, including: Contrastive learning sample set is constructed on the original dataset. Two-dimensional encoder and three-dimensional encoder are used to encode the samples in the contrastive learning sample set respectively, and the two-dimensional feature vector and three-dimensional feature vector of the sample are obtained and projected into the same embedding space. Construct the loss function of the contrastive learning module, calculate the loss value and update the parameters of the contrastive learning module in reverse until convergence, fix the parameters of the two-dimensional encoder and the three-dimensional encoder in the contrastive learning module to obtain the pre-trained contrastive learning module. The 3D reconstruction model is iteratively trained on the original dataset using gradient descent, including: The original dataset is input into a pre-trained contrastive learning model with fixed parameters to obtain the two-dimensional and three-dimensional feature vectors of the object. Based on the object's two-dimensional and three-dimensional feature vectors, the feature fusion module performs a Cross-Attention operation to generate Q, K, and V matrices. It then uses Attention operations to re-represent the object's two-dimensional and three-dimensional features and finally uses dense comparison to fuse the features to obtain the object's fused features. Construct a binary cross-entropy loss function for the 3D reconstruction model, calculate the binary cross-entropy loss value and update the 3D reconstruction model parameters until the binary cross-entropy loss function converges, and obtain the trained 3D reconstruction model. Input the image of the object being queried into the trained 3D reconstruction model to obtain the predicted 3D shape.

2. The 3D reconstruction method based on contrastive learning and dense comparison according to claim 1, characterized in that, The original dataset is represented as follows: Where k represents the object category, I ik S is the two-dimensional query image of the i-th object of category k in the original dataset. ik Let N be the 3D supporting shape of the i-th object of category k in the original dataset. k N represents the number of objects of category k in the original dataset. This represents the total number of objects in the original dataset. The two-dimensional query images are captured using a high-resolution camera, and the shooting angle is uniform during capture.

3. The 3D reconstruction method based on contrastive learning and dense comparison according to claim 2, characterized in that, The step involves constructing a contrastive learning sample set on the original dataset, and using a two-dimensional encoder and a three-dimensional encoder to encode the samples in the contrastive learning sample set, respectively, to obtain the two-dimensional feature vectors and three-dimensional feature vectors of the samples and project them into the same embedding space. Specifically: Construct a contrastive learning sample set on the original dataset: randomly select N from the categories of the original dataset. c For each of the following categories, a pair of two-dimensional query images and their corresponding three-dimensional supporting shapes are randomly selected to form a contrastive learning sample set, represented as follows: in, This represents the two-dimensional query image of the i-th sample in the comparison learning sample set. This represents the three-dimensional support shape of the i-th sample in the contrastive learning sample set; The two-dimensional query images of the samples in the contrastive learning sample set are input into a two-dimensional encoder for encoding, resulting in a two-dimensional feature vector of the sample, represented as: Among them, E 2d This represents a two-dimensional encoder. Represents the two-dimensional feature vector of the i-th sample; The 3D support shapes of the samples in the contrastive learning sample set are input into the 3D encoder for encoding, resulting in the 3D feature vectors of the samples, represented as: Among them, E 3d Indicates a 3D encoder. Represents the three-dimensional feature vector of the i-th sample; Projecting the two-dimensional and three-dimensional feature vectors of the sample into the same embedding space yields the two-dimensional and three-dimensional embedding features of the sample, represented as: in, Represents two-dimensional embedded features, P represents the 3D embedding feature. 2d P 3d It is a single-layer, fully connected network.

4. The 3D reconstruction method based on contrastive learning and dense comparison according to claim 3, characterized in that, The loss function of the contrastive learning module is expressed as: in, in, Let the loss be the two-dimensional query image for the i-th sample. Let be the 3D support shape loss of the i-th sample, sim be the cosine distance, and τ be the temperature coefficient.

5. The 3D reconstruction method based on contrastive learning and dense comparison according to claim 3, characterized in that, The re-representation of the object's two-dimensional and three-dimensional features specifically includes: Cross-Attention operation is performed on the two-dimensional and three-dimensional feature vectors of the object to generate Q, K, and V matrices: For two-dimensional eigenvectors: For three-dimensional feature vectors: Among them, W Q W K W V ∈R d×d All are trainable parameters; the shape of both the two-dimensional and three-dimensional feature vectors of the object is r. 3 ×d,r 3 d is the feature resolution, and d is the feature dimension; The two-dimensional and three-dimensional features of an object are re-represented using the Attention operation. The two-dimensional features of the object are: The three-dimensional features of the object are: in, For the re-represented two-dimensional features, The three-dimensional features are represented as a new representation, where d is the feature dimension and FC represents a fully connected layer.

6. The 3D reconstruction method based on contrastive learning and dense comparison according to claim 5, characterized in that, The method of using dense comparison to perform feature fusion to obtain the fused features of an object specifically involves: The re-represented 3D shape features The shape is deformed into 1×r 3 ×d; Deformed Through a fully connected layer: Output a global descriptor of shape 1×d; Repeat the output of this descriptor r 3 Next, we obtain F″′ S The shape is r 3 ×d; F″′ S With the re-represented two-dimensional features By splicing the data, we obtain the splicing feature F. qS The shape is r 3 ×2d; splicing feature F qS The input is a 3D convolutional layer Conv3D(2d,d,3), and the output is a fused feature F″′. q The shape is r 3 ×d.

7. The 3D reconstruction method based on contrastive learning and dense comparison according to claim 6, characterized in that, The binary cross-entropy loss function of the 3D reconstruction model is expressed as: Where n is the number of pixels in the fused feature, y i ∈{0,1} represents the actual occupancy of the i-th pixel in the fused feature, where 1 indicates that the pixel is occupied and 0 indicates that the pixel is empty; p i This indicates the probability that the i-th pixel in the output is occupied.

8. The 3D reconstruction method based on contrastive learning and dense comparison according to claim 7, characterized in that, During the pre-training process of the contrastive learning module, the temperature coefficient τ is set to 0.1, the iteration is 20 epochs, and the learning rate is 0.

001. During the iterative training of the 3D reconstruction model, the number of iterations was set to 100,000, the batch size to 32, and the learning rate to 0.0001, which was halved every 20,000 iterations; p was set to i When the value is greater than 0.3, it is predicted that the i-th pixel is occupied by an object.

9. A three-dimensional reconstruction system based on contrastive learning and dense comparison, characterized in that, The 3D reconstruction method based on contrastive learning and dense comparison, applied to any one of claims 1-8, includes a data acquisition module, a model building module, a pre-training module, an iterative training module, and a result prediction module; The data acquisition module is used to acquire two-dimensional query images of objects and their corresponding three-dimensional supporting shapes to form the original dataset; The model building module is used to build a 3D reconstruction model, including a contrast learning module and a feature fusion module; The pre-training module is used to pre-train the contrastive learning module, and after completion, the parameters of the contrastive learning module are fixed, including: Contrastive learning sample set is constructed on the original dataset. Two-dimensional encoder and three-dimensional encoder are used to encode the samples in the contrastive learning sample set respectively, and the two-dimensional feature vector and three-dimensional feature vector of the sample are obtained and projected into the same embedding space. Construct the loss function of the contrastive learning module, calculate the loss value and update the parameters of the contrastive learning module in reverse until convergence, fix the parameters of the two-dimensional encoder and the three-dimensional encoder in the contrastive learning module to obtain the pre-trained contrastive learning module. The iterative training module is used to iteratively train the 3D reconstruction model on the original dataset using the gradient descent method, including: The original dataset is input into a pre-trained contrastive learning model with fixed parameters to obtain the two-dimensional and three-dimensional feature vectors of the object. Based on the object's two-dimensional and three-dimensional feature vectors, the feature fusion module performs a Cross-Attention operation to generate Q, K, and V matrices. It then uses Attention operations to re-represent the object's two-dimensional and three-dimensional features and finally uses dense comparison to fuse the features to obtain the object's fused features. Construct a binary cross-entropy loss function for the 3D reconstruction model, calculate the binary cross-entropy loss value and update the 3D reconstruction model parameters until the binary cross-entropy loss function converges, and obtain the trained 3D reconstruction model. The result prediction module is used to input the image of the queried object into the trained 3D reconstruction model to obtain the predicted 3D shape.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the three-dimensional reconstruction method based on contrastive learning and dense comparison as described in any one of claims 1-8.

Images