Unsupervised cross-modal retrieval method, system, medium and device based on hypergraph convolution

Abstract

The present invention belongs to the field of cross-modal information retrieval, and provides an unsupervised cross-modal retrieval method, system, medium and device based on hypergraph convolution. The technical solution is as follows: fine-grained semantic features are extracted, and the semantic representation of each modality is further enhanced through a multimodal fusion converter. This deep feature extraction and fusion strategy enables our model to more comprehensively capture the complementary and symbiotic information of multimodal data; an effective fusion method is proposed to construct a semantic complementary similarity matrix to maximize the potential semantic correlation between different modal instances, which helps to overcome the shortcomings of existing methods in capturing the comprehensive semantic information of multimodal data and enhance the model's understanding and measurement of multimodal content correlation; an adaptive hypergraph neural network is introduced, which helps learn hash codes by encoding high-order relationships and local clustering structures between vertices through hypergraph convolution, thereby generating more discriminative hash codes.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal information retrieval, and in particular to unsupervised cross-modal retrieval methods, systems, media and devices based on hypergraph convolution. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the booming development of the Internet and social media, massive multimodal data, such as text, images, videos, etc., are growing at an unprecedented rate. Efficient and accurate cross-modal retrieval systems have become an urgent need in the field of information retrieval. Unsupervised cross-modal hashing technology has attracted widespread attention due to its computational and storage advantages in cross-modal data retrieval. The core challenge of this technology lies in how to reduce the semantic gap between heterogeneous modalities during the hash code learning process and effectively encode the correlation of different modal data into binary codes. Although deep learning methods have shown excellent performance in cross-modal tasks, they are still limited in semantic interaction between modalities and the ability to generalize to new data. Existing unsupervised cross-modal hashing still has some limitations, such as inaccurate similarity measures and modality imbalance, resulting in unsatisfactory retrieval performance. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the above-mentioned background technology, the present invention provides an unsupervised cross-modal retrieval method, system, medium and device based on hypergraph convolution, which can more comprehensively capture the complementary and symbiotic information of multimodal data, overcome the shortcomings of existing methods in capturing the comprehensive semantic information of multimodal data, and enhance the model's understanding and measurement of the relevance of multimodal content.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A first aspect of the present invention provides an unsupervised cross-modal retrieval method based on hypergraph convolution, comprising the following steps:

[0007] Obtain a multimodal training dataset;

[0008] The cross-modal retrieval model is trained based on the multi-modal training data set to obtain a trained cross-modal retrieval model; specifically including:

[0009] Perform cross-modal fusion on the image features and text features extracted based on the multimodal training data set to obtain cross-modal fused image features and text features;

[0010] An image modality similarity matrix is ​​constructed based on image features and text features, and a text modality similarity matrix is ​​constructed based on image features and text features after cross-modal fusion, and the image modality similarity matrix and the text modality similarity matrix are unified into a robust similarity matrix;

[0011] Using the robust similarity matrix, we introduce a hypergraph to aggregate the common features of similar samples into hyperedges to obtain the hypergraph association matrix. We use the association matrix to perform hypergraph convolution on image features and text features, mine high-order semantic information between nodes, and obtain the hash code in the hypergraph learning process.

[0012] A reconstruction loss function is constructed according to the generated hash code and the robust similarity matrix, and the parameters of the hash coding network are updated based on the reconstruction loss function;

[0013] The retrieval results are obtained based on the task data to be retrieved and the trained cross-modal retrieval model.

[0014] Furthermore, the cross-modal fusion of the image features and text features extracted based on the multimodal training data set to obtain the cross-modal fused image features and text features includes:

[0015] The CLIP image feature extractor is used to extract image features, and the text feature extractor is used to extract text features;

[0016] Concatenate the output of the CLIP image feature extractor and the output of the text feature extractor to obtain a concatenated tensor;

[0017] The concatenated tensor is input into the multimodal fusion Transformer, and the self-attention mechanism is used to capture the correlation within and between modalities and the semantic correlation between features to obtain the fused image features and text features.

[0018] Furthermore, the loss function for cross-modal fusion of image features and text features extracted from the multimodal training dataset is:

[0019] ,

[0020] ,

[0021] ,

[0022] in, is the contrast loss of the image modality, is the contrast loss of the text modality, is the cross-modal contrast loss that combines the first two. is the temperature coefficient, represents the similarity calculation function, and Features representing truly aligned image-text pairs, and Indicates Samples and samples, For each batch For training samples.

[0023] Furthermore, the robust similarity matrix is ​​expressed as:

[0024] ,

[0025] ,

[0026] ,

[0027] in, is the weight to measure the similarity information of different modalities. is the image modality similarity matrix, is the text modality similarity matrix, is a symmetric matrix, is the robust similarity matrix Middle i Samples and j The similarity value of samples, m For each batch For training samples.

[0028] Furthermore, the robust similarity matrix is ​​used to introduce a hypergraph to aggregate the common features of similar samples into hyperedges, and obtain the association matrix of the hypergraph, including:

[0029] The image features , text features And the robust similarity matrix As input, each feature vector is regarded as a node and represented as , using the similarity matrix Identify each node most similar nodes; combine the identified nodes into hyperedges , which can be expressed as ,in, Representation and Most similar A collection of nodes;

[0030] Incidence Matrix of Hypergraph It is expressed as:

[0031] .

[0032] Furthermore, the method of using the association matrix to perform hypergraph convolution on the image features and the text features, mining the high-order semantic information between the nodes, and obtaining the hash code in the hypergraph learning process includes:

[0033] Introduce a standard Laplacian matrix for the incidence matrix of the constructed hypergraph;

[0034] Combined with the standard Laplacian matrix to calculate the hypergraph convolutional layer representation;

[0035] A hypergraph convolutional network is constructed based on the obtained hypergraph convolutional layer representation to generate hash codes in the hypergraph learning process.

[0036] Furthermore, the expression of the reconstruction loss function is:

[0037] ,

[0038] ,

[0039] ,

[0040] ,

[0041] ,

[0042] in, is the reconstruction loss, is a hyperparameter that adjusts the scaling range of the similarity matrix, symbol represents the Hadamard product, is a hyperparameter that measures the weight between different losses, is the robust similarity matrix, , , , is the learned hash code.

[0043] A second aspect of the present invention provides unsupervised cross-modal retrieval based on hypergraph convolution, comprising:

[0044] A multimodal data acquisition module, which is used to acquire a multimodal training data set;

[0045] The cross-modal retrieval model training module is used to train the cross-modal retrieval model based on the multimodal training data set to obtain a trained cross-modal retrieval model; specifically includes:

[0046] Perform cross-modal fusion on the image features and text features extracted based on the multimodal training data set to obtain cross-modal fused image features and text features;

[0047] An image modality similarity matrix is ​​constructed based on image features and text features, and a text modality similarity matrix is ​​constructed based on image features and text features after cross-modal fusion, and the image modality similarity matrix and the text modality similarity matrix are unified into a robust similarity matrix;

[0048] Using the robust similarity matrix, we introduce a hypergraph to aggregate the common features of similar samples into hyperedges to obtain the hypergraph association matrix. We use the association matrix to perform hypergraph convolution on image features and text features, mine high-order semantic information between nodes, and obtain the hash code in the hypergraph learning process.

[0049] A reconstruction loss function is constructed according to the generated hash code and the robust similarity matrix, and the parameters of the hash coding network are updated based on the reconstruction loss function;

[0050] The retrieval module is used to obtain retrieval results based on the task data to be retrieved and the trained cross-modal retrieval model.

[0051] A third aspect of the present invention provides a computer-readable storage medium.

[0052] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the hypergraph convolution-based unsupervised cross-modal retrieval method as described above.

[0053] A fourth aspect of the present invention provides a computer device.

[0054] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the unsupervised cross-modal retrieval method based on hypergraph convolution as described above are implemented.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. After fusing the extracted fine-grained semantic features, the present invention constructs a semantic complementary similarity matrix to maximize the potential semantic correlation between instances of different modalities. It then helps learn hash codes by encoding high-order relationships and local clustering structures between vertices through hypergraph convolution, thereby generating more discriminative hash codes and improving retrieval accuracy.

[0057] 2. The present invention uses the CLIP multimodal model to extract fine-grained semantic features, and further enhances the semantic representation of each modality through a multimodal fusion converter. This deep feature extraction and fusion strategy enables the model of the present invention to more comprehensively capture the complementary and symbiotic information of multimodal data.

[0058] 3. The present invention proposes an effective fusion method to construct a semantic complementary similarity matrix to maximize the potential semantic correlation between instances of different modalities. This method helps to overcome the shortcomings of existing methods in capturing the comprehensive semantic information of multimodal data and enhance the model's understanding and measurement of the correlation of multimodal content.

[0059] 4. The present invention introduces an adaptive hypergraph neural network, which helps learn hash codes by encoding high-order relationships and local clustering structures between vertices through hypergraph convolution, thereby generating more discriminative hash codes.

[0060] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0062] Figure 1 It is a flow chart of an unsupervised cross-modal retrieval method based on hypergraph convolution provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0064] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0065] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0066] Explanation of terms

[0067] CLIP (Contrastive Language-Image Pre-Training): is a multimodal pre-trained neural network model released by OpenAI in 2021. It maps images and texts to a unified vector space through contrastive learning, allowing the model to directly calculate the similarity between images and texts in the vector space. The core idea of ​​the CLIP model is to use a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts. This model is particularly suitable for zero-shot learning tasks, that is, the model does not need to see new training examples of images or texts to make predictions. The CLIP model performs well in many fields, such as image-text retrieval, image-text generation, etc.

[0068] Hypergraph Neural Networks (HGNN) is an advanced graph representation learning method that handles complex graph structures by encoding high-order data correlations. This network achieves effective representation learning through hyper-edge convolution operations and is able to capture complex relationships between nodes. The concept of hypergraphs has been introduced into cross-modal hash retrieval to overcome the limitations of ordinary pairwise graphs in describing high-order relationships between samples. Hypergraphs connect any number of samples through hyperedges, which can more comprehensively describe the similarities between samples, i.e., high-order relationships. The mining of this high-order information provides a more robust modal representation for unsupervised cross-modal hashing methods.

[0069] In view of the limitations of the existing unsupervised cross-modal hashing mentioned in the background technology, the present invention combines the CLIP model with hypergraph learning for the first time and proposes an unsupervised cross-modal retrieval method based on hypergraph convolution, including:

[0070] (1) The present invention uses the CLIP multimodal model to extract fine-grained semantic features and further enhances the semantic representation of each modality through a multimodal fusion transformer. This deep feature extraction and fusion strategy enables the model of the present invention to more comprehensively capture the complementary and symbiotic information of multimodal data;

[0071] (2) This paper proposes an effective fusion method to construct a semantic complementary similarity matrix to maximize the potential semantic relevance between instances of different modalities. This method helps to overcome the shortcomings of existing methods in capturing the comprehensive semantic information of multimodal data and enhances the model's understanding and measurement of the relevance of multimodal content;

[0072] (3) The present invention introduces an adaptive hypergraph neural network, which helps learn hash codes by encoding high-order relationships and local clustering structures between vertices through hypergraph convolution, thereby generating more discriminative hash codes;

[0073] (4) The present invention adopts an iterative approximate optimization strategy to reduce information loss in the binarization process.

[0074] Embodiment 1

[0075] like Figure 1 As shown, this embodiment provides an unsupervised cross-modal retrieval method based on hypergraph convolution, comprising the following steps:

[0076] Step 1: Obtain a multimodal dataset;

[0077] In this embodiment, the acquired multimodal data set is represented as ,in, Respectively represent The image data and text data in the sample pairs, is the number of sample pairs;

[0078] The task of cross-modal retrieval is to use data from one modality to retrieve data from another modality.

[0079] In this embodiment, image-to-text retrieval and text-to-image retrieval are used as retrieval tasks to build and optimize a cross-modal retrieval model.

[0080] The multimodal dataset used for training is represented as ,in, Respectively represent The image data and text data in the sample pairs in each batch For the training samples, the image features extracted based on CLIP and text features The constructed similarity matrix is ​​expressed as , image features after cross-modal Transformer fusion and text features The constructed similarity matrix is ​​expressed as The final representations of image and text are defined as and ,in and Represent the dimensions of image features and text features respectively. The hash code generated by the hash coding network is recorded as and , the hash code generated by the hypergraph convolutional network is recorded as and ,in The length of the hash code.

[0081] Step 2: Fuse the image features and text features extracted based on the multimodal dataset to obtain fused image features and text features;

[0082] Step 2 specifically includes the following steps:

[0083] Step 201: In each batch For the training samples, the CLIP image feature extractor is used to extract image features , the text features are extracted using the text feature extractor .

[0084] Step 202: Output result of CLIP image feature extractor And the output of the text feature extractor Concatenate to get a new tensor ,in and Represent the dimensions of visual features and text features respectively, .

[0085] Step 203: The spliced ​​result The input is sent to the multimodal fusion Transformer, and the self-attention mechanism is used to capture the correlation within and between modalities and the semantic correlation between features to obtain the fused image features and text features. The specific steps include:

[0086] In this example, the attention score is obtained by calculating the similarity between the query and the key, and then using it to weight the sum of the values

[0087] This method effectively identifies the degree of correlation between each feature and other features and integrates these relationships into the feature fusion process. Finally, through the multimodal fusion Transformer, the effective fusion of image and text features is achieved, thereby improving the performance of multimodal learning.

[0088] Step 2031, construct the query, key and value in the self-attention mechanism;

[0089] In the self-attention mechanism, queries, keys, and values ​​are constructed using multimodal features, expressed as:

[0090] ,

[0091] ,

[0092] ,

[0093] ,

[0094] in, For images Features, For Text Features, Yes and The result of splicing is similar. It is for images Features and text Features The result of splicing, 、 and are all trainable network parameters. For tensors and The query vector generated by multiplication is For tensors and The key vector produced by multiplication, For tensors and The vector of values ​​produced by the multiplication.

[0095] Step 2032: For any image-text pair, a multimodal feature with stronger representation capability is generated through the self-attention mechanism of the multimodal Transformer, which is expressed as:

[0096] ,

[0097] in, is the dimension of the key, is the self-attention mechanism, and are the key matrix and the value matrix respectively.

[0098] Step 2033: Based on the multimodal features generated by the self-attention mechanism, a feedforward neural network is used to generate multimodal features, which can be expressed as:

[0099] ,

[0100] ,

[0101] ,

[0102] in, and Represent the normalization layer and the dropout layer respectively, It is the output of the multimodal fusion Transformer.

[0103] Step 2034: In order to ensure that the data representation of the same category in the same modality contains consistent category semantics, the modality contrast loss is defined as follows:

[0104] ,

[0105] ,

[0106] in, is the contrast loss of the image modality, is the contrastive loss of the text modality, is the cross-modal contrast loss that combines the first two. is the temperature coefficient, represents the similarity calculation function, and Features that represent truly aligned image-text pairs.

[0107] Therefore, the multimodal contrast loss can be formulated as:

[0108] ;

[0109] Step 2035: Adopt and , and The parameters of the hash coding network are used as the final representations of the image and text modalities respectively. This not only retains the effective information of the original features, but also extracts higher-level semantic information, thereby better capturing and utilizing the complex associations between images and texts, and improving the performance and accuracy of the model.

[0110] Step 3: construct an image modality similarity matrix based on image features and text features, construct a text modality similarity matrix based on image features and text features after cross-modal fusion, and unify the image modality similarity matrix and the text modality similarity matrix into a robust similarity matrix; specifically including:

[0111] Due to the lack of sample labels, the unsupervised hashing method cannot construct a multi-label similarity matrix to guide the learning of hash codes. This embodiment proposes a similarity matrix construction scheme based on aggregation and dynamic adjustment. Assuming that there are visual features , construct the image modality cosine similarity matrix ,in Calculated as ,in ;

[0112] Using text features Constructing a text modality similarity matrix ,in Calculated as .

[0113] Then, the image modality similarity matrix and text modality similarity matrix The similarity matrix is ​​integrated into a unified matrix to maintain the semantic relationship between instances of different modalities and complement each other.

[0114] To this end, a joint modal similarity matrix is ​​constructed To integrate and The multimodal information in is expressed as:

[0115] ,

[0116] ,

[0117] in, It is the weight to measure the similarity information of different modalities. Larger image-text contrast Image-text pairs with smaller values ​​have higher semantic similarity;

[0118] is a symmetric matrix, represented as:

[0119] ,

[0120] in, is the image modality similarity matrix No. OK, is the text modality similarity matrix No. List.

[0121] Through experimental observation, it is noted that for the above similarity matrix , the distances between unpaired instances are not well separated. One reason is that the features are learned through contrast of a large amount of data, which keeps the learned distances of unpaired instances within a small range. Previous hashing algorithms only focused on the values ​​on the diagonal of the similarity matrix, that is, they paid more attention to the pairwise relationship between images and texts. However, in the hashing learning process, the relationship between unpaired images and texts should not be ignored. Therefore, the remapping method is used to enhance the values ​​of the non-diagonal elements in the similarity matrix (reflecting the relative relationship between unpaired images and texts) to make the similarity matrix more discriminative.

[0122] First, get each batch The average, minimum, and maximum values ​​of all elements in , and );

[0123] Then, by comparing with the average, we determine the The corresponding image-text pair is a "similar sample pair" or a "dissimilar sample pair", and different weights are applied to them according to the judgment results, expressed as:

[0124] ,

[0125] The weight and The calculation method can be expressed as: , ;

[0126] and Represent the weights of "similar sample pairs" and "different sample pairs" respectively. These weights can "stretch" the original elements nonlinearly. They help the hash function learn the common features between similar samples and the distinguishing features between different samples, thereby generating more discriminative and accurate hash codes. Finally, the new cross-modal similarity matrix can be expressed as .

[0127] According to the above method, the similarity matrix is ​​constructed respectively. and ,in It is the feature extracted by CLIP encoder and Constructed, It is based on the features fused by cross-modal Transformer and The two similarity matrices are unified into the final robust similarity matrix by weighting It is used to supervise and guide the generation process of hash codes.

[0128] Step 4: Using the robust similarity matrix, we introduce a hypergraph to aggregate the common features of similar samples into hyperedges and obtain the association matrix of the hypergraph.

[0129] The image features , text features And the robust similarity matrix As input, each feature vector is regarded as a node and represented as , using the similarity matrix Identify each node The identified nodes are then combined into hyperedges , which can be expressed as ,in, Representation and Most similar A collection of nodes;

[0130] Therefore, the incidence matrix of the hypergraph is It is expressed as:

[0131] .

[0132] Step 5: Use the association matrix to perform hypergraph convolution on image features and text features, mine high-order semantic information between nodes, and obtain the hash code in the hypergraph learning process;

[0133] Hypergraph learning requires aggregating feature information from different neighborhood structures and generating node representations. To this end, we construct a hypergraph Introduce a standard Laplace matrix, expressed as:

[0134] ,

[0135] in, is a diagonal matrix, each element represents the degree of each node, is another diagonal matrix, each element of which represents the size of each hyperedge.

[0136] The hypergraph convolutional layer is expressed as:

[0137] ,

[0138] in, Represents the first The input of the layer, Indicates The weight matrix of the layer, and Respectively represent Layer and The dimensions represented by the nodes in the layer.

[0139] The hash code generated by the hypergraph convolutional network is expressed as:

[0140] .

[0141] in represents the number of iterations. We use the iterative approximate optimization strategy to optimize the hash code, namely , thus converting the discrete problem into a series of continuous optimization problems, which can effectively alleviate the information loss and instability problems in the binarization process.

[0142] Step 6: Final representation of image and text modalities , Input into a hash coding network composed of multi-layer perceptrons to generate hash codes for retrieval and , and use contrastive loss Supervised hash code generation.

[0143] Use two modality-specific hash functions implemented by a multilayer perceptron , generate hash codes for samples of each modality, expressed as:

[0144] ,

[0145] ,

[0146] in For a binary function, the common subspace learned by contrast loss optimization is introduced, and the formula is as follows:

[0147] ,

[0148] ,

[0149] ,

[0150] and for truly aligned image-text pairs.

[0151] Step 7: Utilize the learned hash code 、 、 、 And the comprehensive similarity matrix Construct a loss function and update the parameters of the hash coding network based on the loss function;

[0152] Using the learned hash code 、 、 、 And the comprehensive similarity matrix Construct the loss function:

[0153] ,

[0154] ,

[0155] ,

[0156] ,

[0157] ,

[0158] in, is a hyperparameter that adjusts the scaling range of the similarity matrix, symbol represents the Hadamard product, is a hyperparameter that measures the weight between different losses.

[0159] Step 8: Obtain retrieval results based on the task data to be retrieved and the trained cross-modal retrieval model.

[0160] Embodiment 2

[0161] This embodiment provides a CLIP-based hypergraph convolutional unsupervised cross-modal hashing system, including the following:

[0162] A multimodal data acquisition module, which is used to acquire a multimodal training data set;

[0163] The cross-modal retrieval model training module is used to train the cross-modal retrieval model based on the multimodal training data set to obtain a trained cross-modal retrieval model; specifically includes:

[0164] Perform cross-modal fusion on the image features and text features extracted based on the multimodal training data set to obtain cross-modal fused image features and text features;

[0165] An image modality similarity matrix is ​​constructed based on image features and text features, and a text modality similarity matrix is ​​constructed based on image features and text features after cross-modal fusion, and the image modality similarity matrix and the text modality similarity matrix are unified into a robust similarity matrix;

[0166] Using the robust similarity matrix, we introduce a hypergraph to aggregate the common features of similar samples into hyperedges to obtain the hypergraph association matrix. We use the association matrix to perform hypergraph convolution on image features and text features, mine high-order semantic information between nodes, and obtain the hash code in the hypergraph learning process.

[0167] A reconstruction loss function is constructed according to the generated hash code and the robust similarity matrix, and the parameters of the hash coding network are updated based on the reconstruction loss function;

[0168] The retrieval module is used to obtain retrieval results based on the task data to be retrieved and the trained cross-modal retrieval model.

[0169] Embodiment 3

[0170] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the unsupervised cross-modal retrieval method based on hypergraph convolution as described above.

[0171] Embodiment 4

[0172] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the unsupervised cross-modal retrieval method based on hypergraph convolution as described above are implemented.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Unsupervised cross-modal retrieval method based on hypergraph convolution, characterized by: The steps include: Obtain a multimodal training dataset; The cross-modal retrieval model is trained based on the multi-modal training data set to obtain a trained cross-modal retrieval model; specifically including: Perform cross-modal fusion on the image features and text features extracted based on the multimodal training data set to obtain cross-modal fused image features and text features; An image modality similarity matrix is ​​constructed based on image features and text features, and a text modality similarity matrix is ​​constructed based on image features and text features after cross-modal fusion, and the image modality similarity matrix and the text modality similarity matrix are unified into a robust similarity matrix; Using the robust similarity matrix, we introduce a hypergraph to aggregate the common features of similar samples into hyperedges to obtain the hypergraph association matrix. We use the association matrix to perform hypergraph convolution on image features and text features, mine high-order semantic information between nodes, and obtain the hash code in the hypergraph learning process. A reconstruction loss function is constructed according to the generated hash code and the robust similarity matrix, and the parameters of the hash coding network are updated based on the reconstruction loss function; Obtain retrieval results based on the task data to be retrieved and the trained cross-modal retrieval model; The robust similarity matrix is ​​used to introduce a hypergraph to aggregate the common features of similar samples into hyperedges, and the association matrix of the hypergraph is obtained, including: The image features , text features And the robust similarity matrix As input, each feature vector is regarded as a node and represented as , using the similarity matrix Identify each node most similar nodes; combine the identified nodes into hyperedges , which can be expressed as ,in, Representation and Most similar A collection of nodes; Incidence Matrix of Hypergraph It is expressed as: ； The method of using the association matrix to perform hypergraph convolution on image features and text features, mining high-order semantic information between nodes, and obtaining hash codes in the hypergraph learning process includes: Introduce a standard Laplacian matrix for the incidence matrix of the constructed hypergraph; Combined with the standard Laplacian matrix to calculate the hypergraph convolutional layer representation; A hypergraph convolutional network is constructed based on the obtained hypergraph convolutional layer representation to generate a hash code in the hypergraph learning process.

2. The unsupervised cross-modal retrieval method based on hypergraph convolution as claimed in claim 1, characterized in that: The cross-modal fusion of the image features and text features extracted based on the multimodal training data set to obtain the cross-modal fused image features and text features includes: The CLIP image feature extractor is used to extract image features, and the text feature extractor is used to extract text features; Concatenate the output of the CLIP image feature extractor and the output of the text feature extractor to obtain a concatenated tensor; The concatenated tensor is input into the multimodal fusion Transformer, and the self-attention mechanism is used to capture the correlation within and between modalities and the semantic correlation between features to obtain the fused image features and text features.

3. The unsupervised cross-modal retrieval method based on hypergraph convolution as claimed in claim 1, characterized in that: The loss function for cross-modal fusion of image features and text features extracted from the multimodal training dataset is: ， ， ， in, is the contrast loss of the image modality, is the contrastive loss of the text modality, is the cross-modal contrast loss that combines the first two. is the temperature coefficient, represents the similarity calculation function, and Features representing truly aligned image-text pairs, and Indicates Samples and samples, For each batch For training samples.

4. The unsupervised cross-modal retrieval method based on hypergraph convolution as claimed in claim 1, characterized in that: The robust similarity matrix is ​​expressed as: ， ， ， in, is the weight to measure the similarity information of different modalities. is the image modality similarity matrix, is the text modality similarity matrix, is a symmetric matrix, is the robust similarity matrix Middle i Samples and j The similarity value of samples, m For each batch For training samples.

5. The unsupervised cross-modal retrieval method based on hypergraph convolution as claimed in claim 1, characterized in that: The expression of the reconstruction loss function is: ， ， ， ， ， in, is the reconstruction loss, is a hyperparameter that adjusts the scaling range of the similarity matrix, symbol represents the Hadamard product, is a hyperparameter that measures the weight between different losses, is the robust similarity matrix, , , , is the learned hash code.

6. The hypergraph convolution-based unsupervised cross-modal retrieval system adopts the hypergraph convolution-based unsupervised cross-modal retrieval method according to claims 1-5, characterized in that: include: A multimodal data acquisition module, which is used to acquire a multimodal training data set; The cross-modal retrieval model training module is used to train the cross-modal retrieval model based on the multimodal training data set to obtain a trained cross-modal retrieval model; specifically includes: Perform cross-modal fusion on the image features and text features extracted based on the multimodal training data set to obtain cross-modal fused image features and text features; An image modality similarity matrix is ​​constructed based on image features and text features, and a text modality similarity matrix is ​​constructed based on image features and text features after cross-modal fusion, and the image modality similarity matrix and the text modality similarity matrix are unified into a robust similarity matrix; Using the robust similarity matrix, we introduce a hypergraph to aggregate the common features of similar samples into hyperedges to obtain the hypergraph association matrix. We use the association matrix to perform hypergraph convolution on image features and text features, mine high-order semantic information between nodes, and obtain the hash code in the hypergraph learning process. A reconstruction loss function is constructed according to the generated hash code and the robust similarity matrix, and the parameters of the hash coding network are updated based on the reconstruction loss function; The retrieval module is used to obtain retrieval results based on the task data to be retrieved and the trained cross-modal retrieval model.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the hypergraph convolution-based unsupervised cross-modal retrieval method as described in any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the unsupervised cross-modal retrieval method based on hypergraph convolution as described in any one of claims 1-5 are implemented.

Images