Self-supervised image-3D point cloud cross-modal hash code extraction model establishment method based on probability contrast learning and application thereof

Abstract

The invention belongs to the technical field of cross-modal retrieval, and discloses a self-supervised image-3D point cloud cross-modal hash code extraction model establishment method based on probability comparative learning and application thereof, and the method comprises the steps of establishing a prototype comparative learning task, a distribution comparative learning task and a cross-modal feature reconstruction task. The prototype comparison learning task is used for pairing the image and the 3D point cloud data and explicitly constraining the image and the 3D point cloud data to a combined Hamming space; a distributed contrast learning task re-parameterization image and a point cloud feature are in probability distribution, a one-to-many modal relationship in a real scene is fit, modal semantic uncertainty is adapted, and meanwhile, robustness of cross-modal matching is improved; and the reconstruction task reconstructs original data according to the fused cross-modal information features to promote modelling to align different modal semantic features. According to the three loss function training models, Hash codes output by the models are extracted and discretized when the loss functions converge; according to the method, the modal gap between the image and the 3D point cloud can be effectively bridged, and the accuracy of cross-modal retrieval is improved.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal retrieval technology, and more specifically, relates to a method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model based on probabilistic contrastive learning and its application. Background Technology

[0002] The background technology of this invention mainly revolves around the need for efficient cross-modal retrieval of multimedia retrieval and processing, particularly image and 3D point cloud data. 3D point cloud data captures the spatial layout and geometric attributes of objects or scenes in 3D space. In real-world applications such as autonomous driving, augmented reality, and robotics, rapidly retrieving 3D point cloud data from 2D images (and vice versa) is crucial. Traditional methods, such as direct retrieval based on high-dimensional features, are often inefficient. Hash methods, because they can compress high-dimensional features into binary code, significantly improve retrieval speed and reduce storage overhead while maintaining sample similarity, becoming the mainstream solution in the field of cross-modal retrieval.

[0003] Traditional cross-modal hashing methods primarily focus on scenarios such as text-video and image-text. However, point cloud data has an irregular and unordered data structure, making it difficult for traditional cross-modal hashing methods to effectively capture meaningful semantics. Furthermore, feature variations and semantic gaps exist between 2D pixels and 3D coordinates, hindering the learning of precise correspondences between the two modes. Traditional point-to-point comparative learning only focuses on similarity measurements between individual samples, failing to uncover potential connections between modalities at the overall data distribution level. It cannot effectively capture the semantic information of 3D point cloud and image data, nor can it eliminate the gaps between modalities, resulting in limited retrieval performance. Summary of the Invention

[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model based on probabilistic contrastive learning and its application. The aim is to reduce the modal gap between images and 3D point clouds and improve the accuracy of cross-modal retrieval.

[0005] To achieve the above objectives, this invention provides a method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model based on probabilistic contrastive learning, comprising: The point cloud in the 3D point cloud dataset is rendered into images from multiple different angles to obtain a training set. The training samples in the training set are matching image data and 3D point cloud data. The first hash model is used to extract features from image data in training samples and convert them into hash codes. The second hash model is used to extract features from 3D point cloud data in training samples and convert them into hash codes. The features of the image data are reconstructed into 3D point cloud data using a first reconstruction model, and the features of the 3D point cloud data are reconstructed into image data using a second reconstruction model. A loss function is constructed, and the first hash model, the second hash model, the first reconstruction model, and the second reconstruction model are trained based on the loss function. The loss function includes prototype contrast loss, distribution contrast learning loss, and reconstruction loss. The prototype contrast loss is used to reduce the Hamming distance between the hash codes of similar image data and 3D point cloud data after conversion, and to increase the Hamming distance between the hash codes of dissimilar image data and 3D point cloud data after conversion. By modeling the features of the image data and the features of the 3D point cloud data as probability distributions using a reparameterization method, and randomly sampling the probability distributions, the following is obtained: Based on the sampling results, The distribution contrast learning loss is constructed based on the sampling results; the reconstruction loss includes the feature loss between the reconstructed 3D point cloud data and the original point cloud data, as well as the feature loss between the reconstructed image data and the original image data. After multiple training sessions, when the loss function converges, the first hash model and the second hash model are used as the image hash code extraction model and the 3D point cloud hash code extraction model, respectively.

[0006] Furthermore, the distribution contrastive learning loss for:

[0007] The random contrastive loss is expressed as follows:

[0008] in, For training batch size, , These are the th in the current training batch. The sampling results corresponding to the image data in each sample, and the sampling results corresponding to the 3D point cloud data. , Each represents the The first in the sampling results Regarding the sampling results; For the The set of sampling results corresponding to the 3D point cloud data in the sampling results is the set of positive samples; In the current training batch, excluding the first The set of sampling results corresponding to other 3D point cloud data outside of the sample is the negative sample set; Indicates the similarity between features; These are the weighting coefficients; The KL divergence loss is calculated for two modal distributions: image and 3D point cloud, and unit normal distribution.

[0009] Furthermore, the prototype contrast loss for:

[0010] in, For training batch size, , These are the th in the current training batch. Hash codes generated from 3D point cloud data and image data in each sample To compare the losses:

[0011] in, They are positive and negative samples, respectively. For the negative sample set, for hour, For the current batch excluding the first The set of hash codes generated from image data other than the image data in each sample; for hour, For the current batch excluding the first A set of hash codes generated from 3D point cloud data other than 3D point cloud data in a sample; For temperature parameters, This indicates the similarity between features.

[0012] Furthermore, the reconstruction loss The expression is:

[0013] in, , These are the th in the current training batch. 3D point cloud data and image data from each sample , These are the reconstructed 3D point cloud data and the reconstructed image data, respectively. It represents the second norm.

[0014] Furthermore, the first hash model includes a Transformer-based image encoder and an image hash layer. The image encoder is used to extract image features and uses the Transformer's classification token as a global information marker for the image features. The image hash layer is used to convert the global information marker of the image features into a corresponding hash code. The second hash model includes a Transformer-based 3D point cloud encoder and a 3D point cloud hash layer. The 3D point cloud encoder is used to extract 3D point cloud features and uses the Transformer's classification token as a global information marker for the 3D point cloud features. The 3D point cloud hash layer is used to convert the global information marker of the 3D point cloud features into the corresponding hash code.

[0015] Furthermore, the first reconstruction model includes a first cross-modal feature fusion module and an image decoder, and the second reconstruction model includes a second cross-modal feature fusion module and a 3D point cloud decoder; The first cross-modal feature fusion module includes a first self-attention layer, a first cross-attention layer and a first feedforward layer connected in sequence, and the second cross-modal feature fusion module includes a second self-attention layer, a second cross-attention layer and a second feedforward layer connected in sequence. The image features, after undergoing internal self-attention interaction in the first self-attention layer, are output as the Q matrix input of the first cross-attention layer, and also as the K and V matrices of the second cross-attention layer; the 3D point cloud features, after undergoing internal self-attention interaction in the second self-attention layer, are output as the Q matrix input of the second cross-attention layer, and also as the K and V matrix inputs of the first cross-attention layer. The first cross-attention layer performs cross-attention interaction based on the input Q matrix, K matrix, and V matrix, and the output features are input to the first feedforward layer to obtain image features that fuse 3D point cloud modal information. The second cross-attention layer performs cross-attention interaction based on the input Q matrix, K matrix, and V matrix, and the output features are input to the second feedforward layer to obtain 3D point cloud features that fuse image modal information. The image decoder is used to decode the image features of the fused 3D point cloud modal information to obtain the reconstructed 3D point cloud data; The 3D point cloud decoder is used to decode the 3D point cloud features of the fused image modal information to obtain the reconstructed image data.

[0016] This invention also provides a method for extracting cross-modal hash codes from image-3D point clouds, comprising: For an object from which a hash code is to be extracted, determine its modality. If it is an image, use the image hash code extraction model to extract its hash code and discretize it. If it is a 3D point cloud, use the 3D point cloud hash code extraction model to extract its hash code and discretize it. The image hash code extraction model and the 3D point cloud hash code extraction model are obtained by the self-supervised image-3D point cloud cross-modal hash code extraction model establishment method described above.

[0017] This invention also provides a method for establishing an image-3D point cloud cross-modal hash code library, comprising: For a dataset consisting of image-3D point cloud pairs, the discrete hash codes of each image are extracted using the image-3D point cloud cross-modal hash code extraction method described above to obtain an image hash code library. The discrete hash codes of each 3D point cloud are then extracted using the image-3D point cloud cross-modal hash code extraction method described above to obtain a 3D point cloud hash code library.

[0018] This invention also provides a cross-modal retrieval method for image-3D point cloud, comprising: For the target object to be retrieved, determine its modality; If it is an image, the discrete hash code of the target object is extracted using the image-3D point cloud cross-modal hash code extraction method described above. Then, the similarity between the discrete hash code and each 3D point cloud hash code in the 3D point cloud hash code library is calculated. The 3D point cloud corresponding to the top R 3D point cloud hash codes with the highest similarity is taken as the retrieval result. If it is a 3D point cloud, the discrete hash code of the target object is extracted using the image-3D point cloud cross-modal hash code extraction method described above. Then, the similarity between the discrete hash code and each image hash code in the image hash code library is calculated. The images corresponding to the top R image hash codes with the highest similarity are used as the search results. The 3D point cloud hash code library and the image hash code library are established using the image-3D point cloud cross-modal hash code library establishment method described above.

[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the self-supervised image-3D point cloud cross-modal hash code extraction model establishment method as described in any of the preceding claims, or the image-3D point cloud cross-modal hash code extraction method as described in the preceding claims, or the image-3D point cloud cross-modal hash code library establishment method as described in the preceding claims, or the image-3D point cloud cross-modal retrieval method as described in the preceding claims.

[0020] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) This invention, by designing three complementary losses—prototype contrast loss, distribution contrast learning loss, and cross-modal feature reconstruction—can fully mine the semantic information of images and 3D point clouds, effectively bridging the gap between modalities. Specifically, the prototype contrast loss pairs image and 3D point cloud data and explicitly constrains them to a joint Hamming space. The image and 3D point cloud features are reparameterized into probability distributions, and the distribution contrast learning loss, constructed based on the probability distribution, mines the potential connections between modalities from the overall data distribution level, conforming to the real "one-to-many" modal relationship (one 3D point cloud corresponds to multiple images from different angles), adapting to the semantic uncertainty of modalities while improving the robustness of cross-modal matching; at the same time, through random sampling comparison, it not only ensures the learning effectiveness of a large number of sample (image sample) categories, but also significantly improves the semantic representation ability of a small number of sample (3D point cloud sample) categories. The reconstruction loss is constructed by using 3D point cloud data reconstructed from image data and original 3D point cloud data, and the reconstruction loss is constructed by using image data reconstructed from 3D point cloud data and original image data, which can enable the model to learn to align semantic features of different modalities. The hash code extraction model established based on this enables hash codes from different modalities with similar semantics to be closely mapped together, reducing the gap between image and 3D point cloud modalities, which is beneficial to improving the accuracy of subsequent image-3D point cloud cross-modal retrieval.

[0021] (2) The method of the present invention does not rely on manually labeled data and can be trained on large-scale unlabeled images and 3D point cloud data pairs, solving the problem of difficulty in obtaining sufficient labels for large-scale data in practical applications and reducing training costs.

[0022] (3) The present invention specifically designed a distributed contrastive learning loss. By using the probability-based contrastive loss, the rich information of the point cloud can be further utilized to reduce the modal gap. By adopting distributed contrastive learning, the features of two different modalities, image and 3D point cloud, are mapped to the same space. In the same space, samples with similar hash codes are brought closer to each other, while samples with dissimilar hash codes are kept away from each other. The potential relationship between modalities is mined from the overall data distribution level. It can capture the semantic information of 3D point cloud and image data well, so as to eliminate the gap between the two modalities and improve the retrieval performance.

[0023] In summary, this invention integrates three complementary learning tasks (prototype contrastive learning, distribution contrastive learning, and cross-modal feature reconstruction) into a unified learning framework, fully mining and integrating global and local feature information of images and 3D point clouds. This significantly improves the accuracy of cross-modal retrieval of large-scale unlabeled images and 3D point clouds without the need for manual labeling, while maintaining high-efficiency coding performance. Attached Figure Description

[0024] Figure 1This is a schematic diagram of the overall framework of the self-supervised image-3D point cloud cross-modal hash code extraction model establishment method based on probabilistic contrastive learning in an embodiment of the present invention.

[0025] Figure 2 This is a flowchart illustrating the workflow of the self-supervised image-3D point cloud cross-modal retrieval hashing method based on probabilistic contrastive learning in this embodiment of the invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0027] In this invention, the terms "first," "second," etc., used in the invention and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0028] This invention innovatively integrates three complementary learning tasks: prototype contrastive learning, distribution contrastive learning, and cross-modal feature reconstruction. This effectively extracts local features from the data, mines global distribution information, and deeply fuses data from different modalities, significantly bridging modal gaps. It significantly improves the accuracy of large-scale unlabeled point cloud retrieval without the need for manual labeling.

[0029] In the experiments of this invention, through testing on multiple public benchmark datasets and comparing the performance with existing cross-modal hashing models, the constructed hashing model outperforms existing cross-modal hashing retrieval methods in metrics such as mAP@2000 and Precision@K in image retrieval of point clouds and point clouds retrieval of images. This is due to the innovative learning task design: by introducing three complementary tasks—prototype contrastive learning, distribution contrastive learning, and cross-modal feature reconstruction—the model can comprehensively capture global and local feature information of image and 3D point cloud data during training and effectively bridge the gap between the two modalities; and efficient feature extraction and encoding: using an advanced Transformer encoder combined with an attention mechanism, image and 3D point cloud features are effectively extracted and converted into high-quality binary hash codes through a carefully designed hash layer. Specific embodiments are as follows.

[0030] Example 1 like Figure 1 and Figure 2As shown, this invention provides a method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model (PCH) based on probabilistic contrastive learning, including multimodal feature extraction, prototype and distribution contrastive learning loss construction, and cross-modal data reconstruction. This invention pre-establishes a first hash model and a second hash model. The first hash model is used to extract hash codes from each image in the training set, and the second hash model is used to extract hash codes from each point cloud in the training set. The training samples in the training set include matched image data and 3D point cloud data. The first hash model is used to extract image features and convert them into continuous hash codes, and the second hash model is used to extract 3D point cloud features and convert them into continuous hash codes. In this embodiment, the first hash model includes an image encoder and an image hash layer, and the second hash model includes a 3D point cloud encoder and a 3D point cloud hash layer.

[0031] The first reconstruction model is used to reconstruct image features into point cloud data, and the second reconstruction model is used to reconstruct 3D point cloud features into image data. In this embodiment of the invention, the first reconstruction model includes a first cross-modal feature fusion module and an image decoder, and the second reconstruction model includes a second cross-modal feature fusion module and a 3D point cloud decoder.

[0032] A loss function is constructed, and the first hash model, second hash model, first reconstruction model, and second reconstruction model are trained based on this loss function; the loss function includes prototype contrast loss. Distribution contrastive learning loss and reconstruction losses Among them, prototype contrast loss This is used to reduce the Hamming distance between similar image data and 3D point cloud data after hash code conversion, and to increase the Hamming distance between dissimilar image data and 3D point cloud data after hash code conversion; reconstruction loss This includes the loss between the reconstructed point cloud data and the original point cloud data, as well as the loss between the reconstructed image data and the original image data; by modeling the features of the image and 3D point cloud as probability distributions, and randomly sampling the probability distributions, we obtain... For the sampling results, construct a distribution contrastive learning loss based on the sampling results. To reduce the differences between modes.

[0033] After multiple training sessions, when the loss function converges, the first hash model and the second hash model are used as the image hash code extraction model and the 3D point cloud hash code extraction model, respectively.

[0034] The following is one of the specific implementation methods: 1. Multimodal feature extraction includes: (1-1) Data Loading: First, in this embodiment of the invention, preprocessing is performed on a large-scale 3D point cloud dataset, which will... The original 3D point cloud was rendered into multiple images from different angles, and all rendered images were standardized and resized to obtain a dataset. During training and testing, an image is randomly selected and paired with its corresponding original 3D point cloud to form a data pair. The loaded image and point cloud data are then processed using... and express.

[0035] (1-2) Feature encoding: Given a dataset ,in, For dataset The total number of sample pairs in the middle. For the first One sample pair, For the first Point cloud data in a sample pair This represents the number of points (coordinates) in a 3D point cloud sample. Indicates the first Image data in each sample pair , These represent the length and width of the image data, respectively, in pixels.

[0036] In this embodiment of the invention, two multi-layer pre-trained Transformers are used as an image encoder and a 3D point cloud encoder, respectively, to extract image data. Image features and extracting point cloud data 3D point cloud features In this embodiment of the invention, before inputting the data to the 3D point cloud encoder, the point cloud data is further reduced in dimensionality using a farthest point sampling (FPS) algorithm. In other embodiments of the invention, other image and 3D point cloud feature extraction models such as SwingTransformer and PointNet may also be used.

[0037] Taking image data as an example, the input image data After position encoding, it is obtained .feature The calculation formula can be expressed as:

[0038] in, Representation layer normalization, This represents a feedforward neural network. This represents a multi-head self-attention mechanism composed of multiple self-attention mechanisms, and its specific calculation is as follows:

[0039] in, Indicates the first Linear transformation parameters of an attention head, For the total number of attention heads, This indicates the output weighting parameters; For the hidden layer dimension of Transformer, These are the query vector, key, and value.

[0040] Finally, the Transformer's special classification token [CLS] was used as the image feature. and 3D point cloud features Global information markers , In a sample pair, the image data and its corresponding point cloud data each have their own global information labels; these global information labels are then... Input image hash layer, The input 3D point cloud is hashed in a hash layer to obtain a hash code corresponding to the image and the 3D point cloud (this hash code is a consecutive hash code), i.e. , ;in, This represents hash encoding. In this embodiment of the invention, the image hash layer and the point cloud hash layer are both single-layer fully connected layers, with the number of hidden layer neurons equal to the number of bits in the hash code. When the loss function converges, the output is converted into a binary hash code using the sign function sgn. 2. Constructing the prototype and distribution comparison learning loss, and comparing the learning loss. Including prototype contrast loss Comparison of loss with distribution : .

[0041] (2-1) Prototype contrastive learning loss : Connecting the image with the sequential hash code of the 3D point cloud , By inputting the contrastive learning framework separately, the prototype contrastive learning loss is obtained. .

[0042] In this embodiment of the invention, the prototype contrastive learning loss for:

[0043] Where B is the training batch size. , Hash codes generated from 3D point clouds and images, respectively. To compare the loss, describing the difference between positive and negative samples across different modalities, its expression is:

[0044] in, To represent the similarity between features, as an optional method, this embodiment specifically uses cosine similarity; In this embodiment, the samples are respectively positive and negative. This represents the hash code of a modality in the current sample pair. These are the hash codes of the corresponding modal data in the sample pair and the hash codes of the corresponding modal data in all other remaining sample pairs in the current batch, excluding the current sample pair; for example, for , express , express , express ,in, , , This is the current batch size; for , express , express , express , ; This refers to the temperature parameter. The negative sample set, i.e. or The set that constitutes the composition.

[0045] (2-2) Distribution contrast learning loss Features of images and 3D point clouds and Modeling as Gaussian distributions using reparameterization methods respectively and Then, respectively, in the Gaussian distribution of the image Gaussian distribution of point clouds Random sampling (probabilistic coding) is performed on the above to obtain Sampling results ,in, To The first obtained by random sampling Each sampling result To The first obtained by random sampling Each sampling result; the sampling results By inputting into a probability-based contrastive loss framework, a distributed contrastive learning loss is obtained. In other embodiments, image and point cloud features can also be reparameterized to other probability distributions.

[0046] In this embodiment of the invention, the distributed contrastive learning loss for:

[0047] The random contrastive loss is expressed as follows:

[0048] in, Image modality in the current sample pair The sampling probability encoding (sampling result). 3D point cloud modality in the current sample pair The sampling results; 3D point cloud modality in the current sample pair The set of sampling results is the set of positive samples, i.e. All sampled point cloud probability codes, ; Excluding the current batch The set of probability codes of other point clouds is the negative sample set. This is the weighting coefficient, with empirically determined values.

[0049] The KL divergence loss is calculated using two modal distributions and a unit normal distribution, and is used to constrain image and point cloud encoding. and The latent distribution of is expressed as:

[0050] in, Represents the identity matrix. and The distributions of image and point cloud modes are represented as follows:

[0051]

[0052] in, For consisting of fully connected layers, LayerNorm layers and A mapping module consisting of normalization layers; It is a separate fully connected layer; This represents a Gaussian distribution. In other embodiments, other prior distributions such as the Von-Mises-Fisher distribution may also be used. This indicates taking the diagonal elements.

[0053] 3. Cross-modal data reconstruction includes: (3-1) Cross-modal feature fusion: combining image features obtained during the multimodal feature extraction stage After self-attention and cross-attention fusion are performed in the first cross-modal feature fusion module, image features with fused 3D point cloud modal information are obtained. Point cloud features After self-attention and cross-attention fusion are performed in the second cross-modal feature fusion module, 3D point cloud features of fused image modal information are obtained. In this embodiment of the invention, both the first cross-modal feature fusion module and the second cross-modal feature fusion module include a self-attention layer, a cross-attention layer, and a feedforward layer connected in sequence. Image features After the self-attention layer of the first cross-modal feature fusion module performs internal self-attention interaction, the output features are used as the Q-matrix input to the cross-attention layer of the first cross-modal feature fusion module, while the point cloud features... After internal self-attention interaction in the self-attention layer of the second cross-modal feature fusion module, the output features are used as the K and V matrices of the cross-attention layer of the first cross-modal feature fusion module. After cross-attention interaction in the cross-attention layer of the first cross-modal feature fusion module, the resulting output is input to the feedforward layer of the first cross-modal feature fusion module to obtain image features that fuse 3D point cloud modal information. Similarly, point cloud features After the self-attention layer of the second cross-modal feature fusion module performs internal self-attention interaction, the output features are used as the Q-matrix input to the cross-attention layer of the second cross-modal feature fusion module, while the image features... After internal self-attention interaction in the self-attention layer of the first cross-modal feature fusion module, the output features are used as the K and V matrices of the cross-attention layer of the second cross-modal feature fusion module. After cross-attention interaction in the cross-attention layer of the second cross-modal feature fusion module, the output is input to the feedforward layer of the second cross-modal feature fusion module to obtain the 3D point cloud features of the fused image modal information. .

[0054] (3-2) Cross-modal data reconstruction: The fused image and 3D point cloud features are input into the image decoder and 3D point cloud decoder, respectively, to obtain the reconstructed point cloud data and reconstructed image data. In this embodiment of the invention, the image decoder and 3D point cloud decoder are single-layer Transformers.

[0055] The difference between the decoded data and the original data is calculated to obtain the 2D and 3D reconstruction losses. .

[0056] In this embodiment of the invention, reconstruction loss for:

[0057] in, This represents the high-dimensional feature encoding obtained from the image decoder, i.e., the reconstructed 3D point cloud data; This represents the high-dimensional feature encoding obtained from the 3D point cloud decoder, i.e., the reconstructed image data; It represents the second norm.

[0058] Model optimization process: To compare learning loss and reconstruction losses The resulting loss function, which combines the losses from various complementary learning tasks, represents the model's total loss.

[0059] The established total loss function To achieve the overall optimization goal, an image-3D point cloud hash code extraction model is trained. The model parameters are updated through backpropagation and iterative training, enabling the model to learn more effective image and 3D point cloud hash codes in a self-supervised, end-to-end manner. When the loss converges, the trained image-3D point cloud hash code extraction model is obtained.

[0060] In this embodiment of the invention, by combining three complementary tasks—prototype contrastive learning, distributed contrastive learning, and cross-modal data reconstruction—the model learns a shared Hamming space in which hash codes from different modalities with similar semantics can be closely mapped together, effectively reducing modality gap.

[0061] Example 2 This invention provides a method for extracting cross-modal hash codes from image-3D point clouds, comprising: For an object from which hash codes are to be extracted, its modality is determined. If it is an image, its continuous hash codes are extracted using an image hash code extraction model and then discretized. If it is a 3D point cloud, its continuous hash codes are extracted using a 3D point cloud hash code extraction model and then discretized. In this embodiment of the invention, a sign function is used. Discretize consecutive hash codes into binary hash codes.

[0062] The image hash code extraction model and the 3D point cloud hash code extraction model are established by the method in Example 1 above.

[0063] The relevant technical solutions are the same as above, and will not be repeated here.

[0064] Example 3 This invention provides a method for establishing an image-3D point cloud cross-modal hash code library, including: For a dataset consisting of image-3D point cloud pairs, the discrete hash codes of each image are extracted using the image-3D point cloud cross-modal hash code extraction method in Example 2 to obtain an image hash code library. The discrete hash codes of each 3D point cloud are also extracted using the image-3D point cloud cross-modal hash code extraction method in Example 2 to obtain a 3D point cloud hash code library.

[0065] The relevant technical solutions are the same as above, and will not be repeated here.

[0066] Example 4 This invention provides a cross-modal retrieval method for image-3D point cloud, including: For the target object to be retrieved, determine its modality; If the image is a 3D point cloud cross-modal hash code extraction method as described in Example 2, the discrete hash code of the target object (target image) is extracted. Then, the similarity between this discrete hash code and each 3D point cloud hash code in the 3D point cloud hash code library is calculated. The 3D point clouds corresponding to the top R most similar 3D point cloud hash codes are used as the retrieval results. In this embodiment of the invention, the top R 3D point cloud hash codes with the closest Hamming distance are used as the top R most similar 3D point cloud hash codes, where R is a positive integer.

[0067] If it is a 3D point cloud, the discrete hash code of the target object (target 3D point cloud) is extracted using the image-3D point cloud cross-modal hash code extraction method in Example 2. Then, the similarity between the discrete hash code and each image hash code in the image hash code library is calculated. The images corresponding to the top R image hash codes with the highest similarity are used as the search results.

[0068] The relevant technical solutions are the same as above, and will not be repeated here.

[0069] Example 5 This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any one or more of the embodiments 1-4 described above.

[0070] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0071] The relevant technical solutions are the same as above, and will not be repeated here.

[0072] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model based on probabilistic contrastive learning, characterized in that, include: The point cloud in the 3D point cloud dataset is rendered into images from multiple different angles to obtain a training set. The training samples in the training set are matching image data and 3D point cloud data. The first hash model is used to extract features from image data in training samples and convert them into hash codes. The second hash model is used to extract features from 3D point cloud data in training samples and convert them into hash codes. The features of the image data are reconstructed into 3D point cloud data using a first reconstruction model, and the features of the 3D point cloud data are reconstructed into image data using a second reconstruction model. A loss function is constructed, and the first hash model, the second hash model, the first reconstruction model, and the second reconstruction model are trained based on the loss function. The loss function includes prototype contrast loss, distribution contrast learning loss, and reconstruction loss. The prototype contrast loss is used to reduce the Hamming distance between the hash codes of similar image data and 3D point cloud data after conversion, and to increase the Hamming distance between the hash codes of dissimilar image data and 3D point cloud data after conversion. By modeling the features of the image data and the features of the 3D point cloud data as probability distributions using a reparameterization method, and randomly sampling the probability distributions, the following is obtained: Based on the sampling results, The distribution contrast learning loss is constructed based on the sampling results; the reconstruction loss includes the feature loss between the reconstructed 3D point cloud data and the original point cloud data, as well as the feature loss between the reconstructed image data and the original image data. After multiple training sessions, when the loss function converges, the first hash model and the second hash model are used as the image hash code extraction model and the 3D point cloud hash code extraction model, respectively.

2. The method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model according to claim 1, characterized in that, The distribution contrast learning loss for: The random contrastive loss is expressed as follows: in, For training batch size, , These are the th in the current training batch. The sampling results corresponding to the image data in each sample, and the sampling results corresponding to the 3D point cloud data. , Each represents the The first in the sampling results Regarding the sampling results; For the The set of sampling results corresponding to the 3D point cloud data in the sampling results is the set of positive samples; In the current training batch, excluding the first The set of sampling results corresponding to other 3D point cloud data outside of the sample is the negative sample set; Indicates the similarity between features; These are the weighting coefficients; The KL divergence loss is calculated for two modal distributions: image and 3D point cloud, and unit normal distribution.

3. The method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model according to claim 1, characterized in that, The prototype contrast loss for: in, For training batch size, , These are the th in the current training batch. Hash codes generated from 3D point cloud data and image data in each sample To compare the losses: in, They are positive and negative samples, respectively. For the negative sample set, for hour, For the current batch excluding the first The set of hash codes generated from image data other than the image data in each sample; for hour, For the current batch excluding the first A set of hash codes generated from 3D point cloud data other than 3D point cloud data in a sample; For temperature parameters, This indicates the similarity between features.

4. The method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model according to claim 1, characterized in that, The reconstruction loss The expression is: in, , These are the th in the current training batch. 3D point cloud data and image data from each sample , These are the reconstructed 3D point cloud data and the reconstructed image data, respectively. It represents the second norm.

5. The method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model according to any one of claims 1-4, characterized in that, The first hash model includes a Transformer-based image encoder and an image hash layer. The image encoder is used to extract image features and uses the Transformer's classification token as a global information marker for the image features. The image hash layer is used to convert the global information marker of the image features into a corresponding hash code. The second hash model includes a Transformer-based 3D point cloud encoder and a 3D point cloud hash layer. The 3D point cloud encoder is used to extract 3D point cloud features and uses the Transformer's classification token as a global information marker for the 3D point cloud features. The 3D point cloud hash layer is used to convert the global information marker of the 3D point cloud features into the corresponding hash code.

6. The method for establishing a self-supervised image-3D point cloud cross-modal hash code extraction model according to claim 5, characterized in that, The first reconstruction model includes a first cross-modal feature fusion module and an image decoder, and the second reconstruction model includes a second cross-modal feature fusion module and a 3D point cloud decoder; The first cross-modal feature fusion module includes a first self-attention layer, a first cross-attention layer and a first feedforward layer connected in sequence, and the second cross-modal feature fusion module includes a second self-attention layer, a second cross-attention layer and a second feedforward layer connected in sequence. The image features, after undergoing internal self-attention interaction in the first self-attention layer, are output as the Q matrix input of the first cross-attention layer, and also as the K and V matrices of the second cross-attention layer; the 3D point cloud features, after undergoing internal self-attention interaction in the second self-attention layer, are output as the Q matrix input of the second cross-attention layer, and also as the K and V matrix inputs of the first cross-attention layer. The first cross-attention layer performs cross-attention interaction based on the input Q matrix, K matrix, and V matrix, and the output features are input to the first feedforward layer to obtain image features that fuse 3D point cloud modal information. The second cross-attention layer performs cross-attention interaction based on the input Q matrix, K matrix, and V matrix, and the output features are input to the second feedforward layer to obtain 3D point cloud features that fuse image modal information. The image decoder is used to decode the image features of the fused 3D point cloud modal information to obtain the reconstructed 3D point cloud data; The 3D point cloud decoder is used to decode the 3D point cloud features of the fused image modal information to obtain the reconstructed image data.

7. A method for extracting cross-modal hash codes from image-3D point clouds, characterized in that, include: For an object from which a hash code is to be extracted, determine its modality. If it is an image, use the image hash code extraction model to extract its hash code and discretize it. If it is a 3D point cloud, use the 3D point cloud hash code extraction model to extract its hash code and discretize it. The image hash code extraction model and the 3D point cloud hash code extraction model are established by the self-supervised image-3D point cloud cross-modal hash code extraction model establishment method according to any one of claims 1-6.

8. A method for establishing an image-3D point cloud cross-modal hash code library, characterized in that, include: For a dataset consisting of image-3D point cloud pairs, the discrete hash codes of each image are extracted using the image-3D point cloud cross-modal hash code extraction method described in claim 7 to obtain an image hash code library, and the discrete hash codes of each 3D point cloud are extracted using the image-3D point cloud cross-modal hash code extraction method described in claim 7 to obtain a 3D point cloud hash code library.

9. A cross-modal retrieval method for image-3D point cloud, characterized in that, include: For the target object to be retrieved, determine its modality; If it is an image, the discrete hash code of the target object is extracted using the image-3D point cloud cross-modal hash code extraction method of claim 7. Then, the similarity between the discrete hash code and each 3D point cloud hash code in the 3D point cloud hash code library is calculated. The 3D point cloud corresponding to the top R 3D point cloud hash codes with the highest similarity is taken as the retrieval result. If it is a 3D point cloud, the discrete hash code of the target object is extracted using the image-3D point cloud cross-modal hash code extraction method described in claim 7. Then, the similarity between the discrete hash code and each image hash code in the image hash code library is calculated. The images corresponding to the top R image hash codes with the highest similarity are used as the search results. The 3D point cloud hash code library and the image hash code library are established using the image-3D point cloud cross-modal hash code library establishment method described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the self-supervised image-3D point cloud cross-modal hash code extraction model establishment method as described in any one of claims 1-6, or the image-3D point cloud cross-modal hash code extraction method as described in claim 7, or the image-3D point cloud cross-modal hash code library establishment method as described in claim 8, or the image-3D point cloud cross-modal retrieval method as described in claim 9.

Images