An unsupervised cross-modal hashing model training method and a retrieval method thereof

Abstract

The application relates to the technical field of multimedia information retrieval, and discloses an unsupervised cross-modal hash model training method and a retrieval method thereof. First, an unlabeled cross-modal data set is acquired, initial semantic features of all image samples and text samples are extracted, and features of a current batch are obtained through small batch sampling according to a preset batch size; the image features and the text features of the current batch are spliced to construct a node set of a cross-modal unified graph, initial similarity between nodes is calculated to generate an initial adjacency matrix; the initial adjacency matrix is subjected to batch structure correction to obtain a robust adjacency matrix; based on the robust adjacency matrix, a multi-layer graph convolution network is used to iteratively update a graph topological structure and node feature representation layer by layer; features output by the last layer of the graph convolution are input into a hash mapping network to generate binary hash codes; and an end-to-end training is performed on the entire network by using a joint loss function, so that the unsupervised cross-modal hash model is obtained. The method improves cross-modal retrieval precision and reduces computational complexity.

Description

Technical Field

[0001] This application relates to the field of multimedia information retrieval technology, specifically to an unsupervised cross-modal hash model training method and its retrieval method. Background Technology

[0002] With the rapid development of internet technology, multimedia data such as images and text are experiencing explosive growth. Cross-modal retrieval technology enables mutual retrieval between data of different modalities and has broad application prospects in multimedia content management, intelligent search engines, and other fields. However, most existing cross-modal retrieval methods rely heavily on manually labeled data for model training, resulting in high labeling costs and difficulty in adapting to large-scale real-world data. Furthermore, cross-modal retrieval methods based on static graph structures cannot adapt to the dynamic changes in node features during training. The structural bias introduced by small-batch sampling is amplified as the number of network layers increases, leading to poor model generalization ability and retrieval accuracy that fails to meet practical application requirements. Summary of the Invention

[0003] To address, or at least partially address, the aforementioned technical problems, this application provides an unsupervised cross-modal hashing model training method and a retrieval method thereof.

[0004] Firstly, this application provides an unsupervised cross-modal hashing model training method, comprising the following steps:

[0005] S1. Obtain an unlabeled cross-modal dataset, which includes image samples and corresponding text samples;

[0006] S2. Extract the initial semantic features of all image samples and all text samples in the cross-modal dataset respectively; perform small-batch sampling on the cross-modal dataset according to a preset batch size to obtain the initial semantic features of the images and the initial semantic features of the text corresponding to the current batch;

[0007] S3. Concatenate the initial semantic features of the image corresponding to the current batch with the initial semantic features of the text to construct a node set of a cross-modal unified graph;

[0008] S4. Calculate the initial similarity between any two nodes in the node set and generate an initial adjacency matrix;

[0009] S5. Perform batch structure correction on the initial adjacency matrix to eliminate the structural bias and noise introduced by small batch sampling, and obtain a robust adjacency matrix.

[0010] S6. Based on the robust adjacency matrix, the graph topology and node feature representation are updated iteratively layer by layer through a multi-layer graph convolutional network, so that the graph topology and node feature representation are optimized synchronously.

[0011] S7. Input the node features output from the last layer of graph convolution into the hash mapping network to generate binary hash codes corresponding to the image modality and the text modality;

[0012] S8. The entire network is trained end-to-end using a joint loss function. After training, an unsupervised cross-modal hashing model is obtained.

[0013] Optionally, the step of extracting the initial semantic features of all image samples and all text samples in the cross-modal dataset specifically includes:

[0014] The pre-trained CLIP model is used to extract the initial semantic features of each image sample and each text sample in the cross-modal dataset.

[0015] Optionally, S4 specifically includes:

[0016] Calculate the cosine similarity between any two nodes in the node set, and use the cosine similarity as the edge weight between the corresponding nodes to generate the initial adjacency matrix.

[0017] Optionally, the batch structure correction of the initial adjacency matrix specifically includes:

[0018] The initial adjacency matrix is ​​linearly transformed by a learnable linear correction operator, and then the corrected adjacency matrix is ​​obtained by processing it with an activation function.

[0019] The linear correction operator is initialized as an identity matrix and its parameters are updated during training.

[0020] Optionally, obtaining the robust adjacency matrix specifically includes:

[0021] A weighted self-loop is added to the modified adjacency matrix, and the strength of message passing between nodes is controlled by adjusting the weight coefficients to obtain the robust adjacency matrix.

[0022] Optionally, the step of iteratively updating the graph topology and node feature representation through a multi-layer graph convolutional network specifically includes:

[0023] Before each graph convolution operation, the similarity between nodes is recalculated based on the node features output by the previous graph convolution. Then, the recalculated similarity matrix is ​​subjected to batch structural correction to obtain the adjacency matrix of the current layer.

[0024] Graph convolution is performed based on the adjacency matrix of the current layer to obtain the node feature representation of the current layer.

[0025] Optionally, the multilayer graph convolutional network adopts a two-layer structure.

[0026] Optionally, the joint loss function is obtained by weighted summation of cross-modal consistency loss, cross-modal contrast loss, and intramodal structure loss.

[0027] Optionally, the hash mapping network is a two-layer feedforward neural network, with the first layer using the Leaky ReLU activation function and the second layer using the tanh activation function;

[0028] The generated binary hash codes corresponding to the image modality and text modality specifically include:

[0029] The output of the hash mapping network is processed by a symbol function to generate the binary hash code.

[0030] Secondly, this application also provides an unsupervised cross-modal retrieval method, which is implemented using an unsupervised cross-modal hash model obtained by the unsupervised cross-modal hash model training method described in any one of the first aspects. The unsupervised cross-modal retrieval method includes the following steps:

[0031] S100, Obtain the cross-modal samples to be retrieved;

[0032] S200. The unsupervised cross-modal hashing model is used to process the cross-modal sample to be retrieved, and the corresponding binary hash code is generated.

[0033] S300. Calculate the Hamming distance based on the binary hash code, and complete the cross-modal retrieval based on the Hamming distance.

[0034] The method provided in this application has the following beneficial effects:

[0035] This application effectively eliminates structural bias and noise introduced by mini-batch sampling by performing batch structural correction on the initial adjacency matrix. This allows the generated adjacency matrix to more accurately reflect the inherent semantic relationships of the global data, avoiding overfitting of the model to local data features and improving the model's generalization ability. By iteratively updating the graph topology and node feature representation layer by layer in a multi-layer graph convolutional network, the graph structure can be continuously adjusted as node features are optimized, more accurately reflecting the true semantic relationships between nodes. This generates node features with stronger semantic discriminativeness, improving the accuracy of cross-modal retrieval. Using a joint loss function for end-to-end training of the network simultaneously ensures the semantic consistency of cross-modal paired samples, the discriminativeness of samples with different semantic meanings, and the discriminativeness of samples within the same modality, avoiding the model performance defects caused by a single loss function.

[0036] The pre-trained CLIP model extracts initial semantic features, leveraging the cross-modal semantic representation capabilities learned from large-scale pre-trained data to provide high-quality foundational features for subsequent relation modeling and accelerate model convergence. The multi-layer graph convolutional network employs a two-layer structure, ensuring feature representation while controlling the number of parameters and computational complexity, avoiding oversmoothing issues caused by excessive network layers. The hash mapping network uses a two-layer feedforward neural network structure, mapping high-dimensional graph convolutional features to a low-dimensional hash space, generating compact binary hash codes, significantly reducing the computational complexity and storage overhead of subsequent retrieval. The trained model supports independent processing of single-modal samples and mixed processing of unpaired cross-modal samples during the inference phase without additional model fine-tuning, adapting to various complex real-world application scenarios. Attached Figure Description

[0037] Figure 1 A schematic diagram of an unsupervised cross-modal hashing model training method provided in this application embodiment;

[0038] Figure 2 A schematic diagram of the error curve of an adjacency matrix update process provided in an embodiment of this application;

[0039] Figure 3 A heatmap of the adjacency matrix update process under different training rounds is provided for an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0041] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0042] This embodiment provides an unsupervised cross-modal hashing model training method and a corresponding unsupervised cross-modal retrieval method for handling retrieval tasks of heterogeneous modal data, including images and text. During training, no manually labeled information is required; model learning relies solely on the inherent semantic relationships within the data, making it adaptable to the processing needs of large-scale real-world multimedia data. This method integrates feature information from different modalities by constructing a unified cross-modal graph, correcting structural biases introduced by mini-batch sampling. Simultaneously, the graph topology and node feature representations are updated concurrently during multi-layer graph convolution operations, ensuring that the generated binary hash code accurately reflects the semantic relevance between samples from different modalities. The trained model has few parameters and high computational efficiency. During the inference phase, it supports independent processing of single-modal samples and mixed processing of unpaired cross-modal samples, meeting the needs of various practical application scenarios.

[0043] See Figures 1 to 3 This application provides an unsupervised cross-modal hashing model training method, including the following steps:

[0044] S1. Obtain an unlabeled cross-modal dataset, which includes image samples and corresponding text samples.

[0045] Specifically, the first step is to acquire an unlabeled cross-modal dataset. This dataset consists of multiple paired image and text samples, each pair corresponding to the same semantic topic. Image samples are stored as digital images, and text samples are stored as natural language descriptions. The cross-modal dataset can be a publicly available multimedia retrieval benchmark dataset or unlabeled multimedia data collected in real-world application scenarios. After acquiring the dataset, basic preprocessing operations are performed. For image samples, the image size is uniformly adjusted to a fixed size, and the pixel values ​​are normalized to ensure they fall within a preset range. For text samples, word segmentation, stop word removal, and encoding are performed to convert the natural language text into a vector format that the computer can process. After preprocessing, all image samples and their corresponding text samples are divided into training and validation sets according to a preset ratio. The training set is used for updating and learning model parameters, while the validation set is used to monitor the model's training process. When the retrieval performance on the validation set no longer improves for several consecutive training epochs, the model is considered converged, and training is stopped.

[0046] S2. Extract the initial semantic features of all image samples and all text samples in the cross-modal dataset respectively; perform small-batch sampling on the cross-modal dataset according to the preset batch size to obtain the initial semantic features of the images and the initial semantic features of the text corresponding to the current batch.

[0047] In some implementations, initial semantic features of all image samples and initial semantic features of all text samples in the cross-modal dataset are extracted, specifically including:

[0048] The pre-trained CLIP model was used to extract the initial semantic features of each image sample and each text sample in the cross-modal dataset.

[0049] First, feature extraction is performed on all image samples and all text samples in the training set to obtain the initial semantic features corresponding to each sample. The initial semantic features are high-level semantic abstractions of the original content of the samples and can reflect the core semantic information of the samples.

[0050] The initial semantic features can be extracted using a pre-trained CLIP model, which is pre-trained on large-scale image-text pairing data. The image encoder in the pre-trained CLIP model processes all image samples in the training set, outputting the initial semantic features for each image sample; similarly, the text encoder in the pre-trained CLIP model processes all text samples in the training set, outputting the initial semantic features for each text sample. The initial semantic features output by the image encoder and text encoder have the same dimension, enabling direct cross-modal similarity calculation. All extracted initial semantic features of the images are stored in the original sample order to generate a global initial semantic feature library for images; similarly, all extracted initial semantic features of the text are stored in the original sample order to generate a global initial semantic feature library for text. In both the global initial semantic feature library and the global initial semantic feature library, feature vectors at the same index position correspond to a pair of paired image and text samples, preserving the pairing relationships between the original samples.

[0051] At the start of each training cycle, the sample indices of the training set are randomly shuffled, and the shuffled indices are divided into multiple consecutive batches according to a preset batch size. For each batch, based on the sample indices contained within the batch, feature vectors at the corresponding index positions are retrieved from the global initial image semantic feature library and the global initial text semantic feature library, respectively, to obtain the initial image semantic features and initial text semantic features corresponding to the current batch. The initial image semantic features and initial text semantic features in each batch maintain a one-to-one pairing relationship to accurately capture the semantic associations between paired samples during subsequent cross-modal relationship modeling.

[0052] S3. Concatenate the initial semantic features of the images and the initial semantic features of the text corresponding to the current batch to construct a node set of a cross-modal unified graph.

[0053] Specifically, each initial semantic feature of an image and each initial semantic feature of text in the current batch can be treated as an independent graph node, and all graph nodes together form a node set of a cross-modal unified graph. The total number of nodes in the node set is twice the size of the current batch, where the first half of the nodes correspond to all image samples in the current batch, and the second half of the nodes correspond to all text samples in the current batch. Image nodes and text nodes at the same index position correspond to a pair of cross-modal samples.

[0054] This construction method integrates sample features from different modalities into a single graph structure, eliminating the need to build separate graph structures for each modality. In subsequent graph structure modeling, semantic similarities between image nodes, text nodes, and between image and text nodes can be calculated simultaneously, comprehensively capturing semantic relationships within and between modalities, and avoiding the fragmentation of cross-modal semantic information caused by separate modeling.

[0055] S4. Calculate the initial similarity between any two nodes in the node set and generate the initial adjacency matrix.

[0056] In some implementations, S4 specifically includes:

[0057] Calculate the cosine similarity between any two nodes in the node set, and use the cosine similarity as the edge weight between the corresponding nodes to generate the initial adjacency matrix.

[0058] After constructing the node set of the cross-modal unified graph, an initial adjacency matrix is ​​generated. This initial adjacency matrix describes the initial semantic relationships between nodes in the graph and serves as the foundation for subsequent graph convolution operations. The cosine similarity of the initial semantic features between any two nodes in the node set is calculated, and this cosine similarity is used as the weight of the edge between these two nodes. The initial adjacency matrix is ​​a square matrix, with both the number of rows and columns equal to the total number of nodes in the node set. Line number The element value of the column corresponds to the first The node and the first Edge weights between nodes.

[0059] Cosine similarity is suitable for measuring the similarity between high-dimensional semantic feature vectors, accurately reflecting the strength of semantic association between samples represented by different nodes. The value of cosine similarity ranges from [-1, 1], with a larger value indicating a higher semantic correlation between the two nodes and a smaller value indicating a lower semantic correlation. The initial adjacency matrix is ​​a symmetric matrix, and the nodes... With nodes The cosine similarity is equal to the node With nodes The cosine similarity is the similarity between nodes whose elements are on the diagonal of the matrix and themselves, and its value is always 1. Based on the generated initial adjacency matrix, subsequent batch structure correction operations are performed to eliminate structural biases and noise introduced by mini-batch sampling.

[0060] S5. Perform batch structure correction on the initial adjacency matrix to eliminate the structural bias and noise introduced by small batch sampling, and obtain a robust adjacency matrix.

[0061] In some implementations, batch structure correction is performed on the initial adjacency matrix, specifically including:

[0062] The initial adjacency matrix is ​​linearly transformed by a learnable linear correction operator, and then the corrected adjacency matrix is ​​obtained by processing it with an activation function.

[0063] The linear correction operator is initialized to the identity matrix, and its parameters are updated during training.

[0064] In some implementations, a robust adjacency matrix is ​​obtained, specifically including:

[0065] A weighted self-loop is added to the modified adjacency matrix, and the strength of message passing between nodes is controlled by adjusting the weight coefficients to obtain a robust adjacency matrix.

[0066] Because mini-batch sampling can only obtain a random subset of the global training dataset, the initial adjacency matrix calculated based on this subset can only reflect the similarity relationships of local data and cannot accurately represent the inherent manifold structure of the global data. Differences in sample distribution between different batches can lead to significant fluctuations in the initial adjacency matrices generated from different batches. This structural bias is amplified as it propagates through graph convolution operations layer by layer, ultimately resulting in noise in the features learned by the model and a decrease in cross-modal retrieval accuracy. If the uncorrected initial adjacency matrix is ​​directly used for graph convolution operations, the model will overfit the local data features of each batch and will fail to learn a cross-modal semantic representation with generalization capabilities.

[0067] In this embodiment, a learnable linear correction operator is used to linearly transform the initial adjacency matrix, and then an activation function is applied to obtain the corrected adjacency matrix. The linear correction operator is a square matrix with the same number of rows and columns as the initial adjacency matrix, initialized as an identity matrix, and its parameters are updated during model training using the backpropagation algorithm. The linear correction operator can learn common structural patterns between different batches, mapping the initial adjacency matrix with batch-specific noise to a batch-independent latent graph space, thus correcting structural biases caused by local sampling. The calculation method for the correction process is as follows:

[0068]

[0069] in, The initial adjacency matrix, For learnable linear correction operators, It is the sigmoid activation function. This is the corrected adjacency matrix. The sigmoid activation function maps the linearly transformed result to the interval between 0 and 1, ensuring a uniform range for all edge weights and preventing extreme values ​​from affecting the stability of subsequent graph convolution operations. The linear correction operator gradually converges to a stable matrix during training, enabling consistent structural correction of the initial adjacency matrix generated in any batch, eliminating structural differences between different batches.

[0070] Weighted self-loops are added to the corrected adjacency matrix. By adjusting the weight coefficients, the strength of message passing between nodes is controlled, resulting in a robust adjacency matrix. Since the cross-modal unified graph contains both image and text nodes, the feature distributions of these two types of nodes inherently differ. Direct graph convolution aggregation amplifies the cross-modal variance, leading to gradient instability or even representation collapse. Adding weighted self-loops enhances the proportion of node-specific features in the aggregation process, balances the contribution of node-specific information with neighborhood information, and suppresses excessive interference between heterogeneous nodes. The weighted self-loop is calculated as follows:

[0071]

[0072] in, For robust adjacency matrix, This is the message passing strength coefficient, with a value ranging from 0 to 1. This is an element-wise multiplication operation. The matrix represents the identity matrix. When the weight coefficient is 0, the node only receives its own feature information and does not aggregate neighborhood information; as the weight coefficient increases, the proportion of neighborhood information received by the node gradually increases. By adjusting the weight coefficient, the receptive field size of graph convolution can be flexibly controlled to adapt to datasets of different sizes and distributions.

[0073] Through the aforementioned weighted self-loops, the spectral radius of the graph convolution propagation operator is limited to a finite range, effectively avoiding gradient explosion during training and ensuring the stable operation of multi-layer graph convolution. The resulting robust adjacency matrix more accurately reflects the inherent semantic relationships of the global data, filtering out random noise introduced by mini-batch sampling.

[0074] S6. Based on the robust adjacency matrix, the graph topology and node feature representation are updated iteratively layer by layer through a multi-layer graph convolutional network, so that the graph topology and node feature representation are optimized synchronously.

[0075] In some implementations, the graph topology and node feature representations are iteratively updated layer by layer through a multi-layer graph convolutional network, specifically including:

[0076] Before each graph convolution operation, the similarity between nodes is recalculated based on the node features output by the previous graph convolution. Then, the recalculated similarity matrix is ​​subjected to batch structural correction to obtain the adjacency matrix of the current layer.

[0077] Graph convolution is performed based on the adjacency matrix of the current layer to obtain the node feature representation of the current layer.

[0078] In some implementations, multilayer graph convolutional networks employ a two-layer structure.

[0079] After obtaining the robust adjacency matrix, a multi-layer graph convolutional network is used to iteratively update the graph topology and node feature representations layer by layer, ensuring synchronous optimization of both. Traditional graph convolution methods use a fixed initial adjacency matrix for all layer operations, which cannot adapt to the dynamic changes in node features during training. The initial adjacency matrix is ​​constructed based on the original initial semantic features. As training progresses, node features gradually learn more accurate semantic information, at which point the initial adjacency matrix can no longer accurately reflect the semantic relationships between the updated nodes. If a static initial adjacency matrix is ​​continued for graph convolution operations, it will lead to incorrect neighborhood information transmission, limit the model's feature representation capabilities, and fail to fully explore the deep semantic relationships in the data.

[0080] Each layer operation in a multi-layer graph convolutional network consists of two steps: first, updating the adjacency matrix of the current layer; and then performing graph convolution operations based on the updated adjacency matrix to obtain the node feature representation of the current layer. The first-layer graph convolution operation uses the robust adjacency matrix obtained in S5 as the initial input, performs degree normalization on the robust adjacency matrix, and obtains the standard graph convolution propagation operator. The initial features of the node set are multiplied by the graph convolution propagation operator, then multiplied by the learnable weight matrix of the first layer, and after processing by the activation function, the node feature representation of the first layer is obtained.

[0081] Before performing the second-layer graph convolution operation, the cosine similarity between any two nodes is recalculated based on the node features output from the first layer, resulting in a new similarity matrix. The same batch structure correction operation as in S5 is performed on the new similarity matrix, using the same learnable linear correction operator for linear transformation, followed by processing with the sigmoid activation function, and then adding weighted self-loops to obtain the adjacency matrix for the second layer. The adjacency matrix for the second layer is then normalized to obtain the corresponding graph convolution propagation operator. The node features output from the first layer are multiplied by this propagation operator, then multiplied by the learnable weight matrix for the second layer, and finally processed by the activation function to obtain the node feature representation for the second layer. The iterative update calculation method is as follows:

[0082]

[0083]

[0084] in, For the first The node features output by the layer. For the first Normalized graph convolution propagation operator for layers, For the first The learnable weight matrix of the layer, It is the ReLU activation function. For the first The node features output by the layer. For the first The adjacency matrix after layer update. The same linear correction operator is reused throughout the training process when updating the adjacency matrix at all layers. and message passing strength coefficient This ensures the consistency of structural modifications.

[0085] Multilayer graph convolutional networks employ a two-layer structure, which can control the number of model parameters and computational complexity while ensuring feature representation capabilities, avoiding the oversmoothing problem caused by too many network layers. Oversmoothing refers to the phenomenon where, with too many graph convolutional layers, the features of all nodes gradually converge to the same value, losing their discriminative power. The two-layer structure effectively captures first- and second-order semantic relationships in the data while avoiding oversmoothing.

[0086] Through this layer-by-layer iterative update approach, the graph topology can be continuously adjusted as node features are optimized, more accurately reflecting the true semantic relationships between nodes. A more accurate graph structure can further improve the performance of graph convolution operations, generating node features with stronger semantic discriminative power, thus forming a positive optimization process. The node features output by the final layer of graph convolution contain rich cross-modal semantic information, accurately reflecting the semantic similarity between different samples.

[0087] like Figure 2As shown, this figure is a visualization curve of the adjacency matrix update process. The horizontal axis represents the training rounds, and the vertical axis represents the mean absolute error between the adjacency matrix and the real semantic association matrix. The figure contains three curves: the blue curve (ε(1)) corresponds to the adjacency matrix error constructed by the features generated by the first layer graph convolution, the red curve (ε(L)) corresponds to the adjacency matrix error constructed by the final joint representation, and the black flat curve (ε(CLIP)) corresponds to the adjacency matrix error constructed by the original CLIP representation. As can be seen from the figure, with the increase of the training rounds, the error values ​​of the blue and red curves show a continuous decreasing trend, and the red curve decreases more significantly, with the final error value being lower than that of the blue curve; while the error value of the black curve remains at a fixed level and does not decrease. This indicates that through the layer-by-layer topology update of this method, the adjacency matrix constructed by both the intermediate layer and the final layer can continuously approach the real semantic association structure as training progresses, while the adjacency matrix constructed based on fixed initial features cannot correct the initial deviation, and the error remains at a high level.

[0088] like Figure 3 As shown, this figure is a heatmap of the adjacency matrix update process under different training rounds. Each row (E1, E15, E40) corresponds to a training stage, and each column corresponds to the adjacency matrix of different processing stages (from ε0 to ε). Label The following are the components in sequence: the initial adjacency matrix generated in S4, the intermediate matrix after processing by the linear correction operator in S5 (without weighted self-loops), the similarity matrix recalculated from the first layer graph convolution output features in S6 (uncorrected), the correction result of the first layer intermediate feature similarity matrix in S6, the ideal output reference of the batch structure correction module, the similarity matrix calculated from the final features output by the second layer graph convolution in S6, and the ideal output baseline of the entire model. In the heatmap, darker colors represent higher similarity between nodes, with the diagonal positions always being the darkest. As can be seen from the figure, the initial adjacency matrix in the early stages of training has obvious noise and cluttered high-similarity regions; after processing by the batch structure correction module, the noise in the adjacency matrix is ​​significantly reduced, and the overall structure becomes smoother; as training progresses, the adjacency matrix updated by multi-layer graph convolution gradually exhibits a clear block structure, with the regions of semantically relevant samples becoming darker and the regions of semantically irrelevant samples becoming lighter, gradually aligning with the true adjacency matrix constructed based on label information. This demonstrates the optimization effect of batch structure correction and layer-by-layer topology update on the adjacency matrix, proving that the proposed method can effectively filter noise, correct biases, and enable the graph structure to gradually reflect the true semantic relationships of the data.

[0089] S7. Input the node features output from the last layer of graph convolution into the hash mapping network to generate binary hash codes corresponding to the image modality and the text modality.

[0090] In some implementations, the hash mapping network is a two-layer feedforward neural network, with the first layer using the Leaky ReLU activation function and the second layer using the tanh activation function;

[0091] Generating binary hash codes corresponding to image and text modalities, specifically including:

[0092] The output of the hash mapping network is processed by a symbol function to generate binary hash codes.

[0093] After obtaining the node features from the output of the last layer of graph convolution, these features are input into a hash mapping network to generate binary hash codes corresponding to the image and text modalities. The hash mapping network is used to map high-dimensional graph convolution features to a low-dimensional hash space, generating compact binary representations that significantly reduce the computational complexity and storage overhead of subsequent retrieval.

[0094] The hash mapping network employs a two-layer feedforward neural network structure. The first layer receives the node features output from the last layer's graph convolution as input, performs a linear transformation, and then processes them through the Leaky ReLU activation function to output intermediate features. The Leaky ReLU activation function can solve the neuron death problem inherent in the ReLU activation function, ensuring the stability of the network training process while preserving feature information on the negative half-axis and avoiding feature loss. The second layer receives the intermediate features output from the first layer, performs a linear transformation, and then processes them through the tanh activation function to output continuous value features. The tanh activation function maps the output values ​​to the interval [-1, 1], making the distribution of continuous value features close to the value range of binary hash codes, providing a foundation for subsequent discretization processing, and enhancing the non-linear expressive power of the features.

[0095] Since the node features input to the hash mapping network are composed of the initial semantic features of the images and the initial semantic features of the text in the current batch, the continuous value features output by the hash mapping network also correspond to both image and text components. The continuous value features are split into two parts according to node order: the first part corresponds to the continuous value features of the image modality, and the second part corresponds to the continuous value features of the text modality. This splitting process preserves the original sample pairing relationships, ensuring a one-to-one correspondence between the continuous value features of paired image samples and text samples.

[0096] The continuous value features of the split image and text are discretized separately to generate the final binary hash code. The discretization is implemented using a sign function, and the calculation method is as follows:

[0097]

[0098] in, The continuous value characteristics of the output of the hash mapping network, This is a sign function; it outputs 1 when the input value is greater than 0, and -1 when the input value is less than or equal to 0. This is the generated binary hash code.

[0099] The length of the generated binary hash code can be adjusted according to actual retrieval needs, flexibly balancing storage overhead and retrieval speed while ensuring retrieval accuracy. The binary hash codes of image modalities and text modalities reside in the same hash space. The binary hash codes generated by paired image samples and text samples have high similarity, while the binary hash codes generated by unpaired samples have low similarity. Cross-modal retrieval can be quickly achieved by calculating Hamming distance.

[0100] S8. The entire network is trained end-to-end using a joint loss function. After training, an unsupervised cross-modal hashing model is obtained.

[0101] In some implementations, the joint loss function is obtained by weighted summation of cross-modal consistency loss, cross-modal contrast loss, and intramodal structure loss.

[0102] After generating binary hash codes corresponding to image and text modalities, the entire network is trained end-to-end using a joint loss function, resulting in an unsupervised cross-modal hashing model. The joint loss function integrates three different optimization objectives, simultaneously ensuring semantic consistency across modal samples, discriminability across modal samples, and discriminability within modal samples, thus avoiding the performance defects caused by a single loss function.

[0103] The joint loss function is obtained by weighted summation of cross-modal consistency loss, cross-modal contrast loss, and intramodal structure loss, and is calculated as follows:

[0104]

[0105] in, and These are adjustable weighting coefficients used to balance the contribution of the three loss terms to the total loss, and can be adjusted according to the distribution characteristics of different datasets.

[0106] Cross-modal consistency loss is used to constrain the distance between paired image and text samples in a continuous feature space, making the continuous value features of paired samples as similar as possible and reducing the semantic gap between different modalities. The cross-modal consistency loss is calculated as follows:

[0107]

[0108] in, For batch size, For continuous value features of the image output by the hash mapping network, For continuous value features of the text output by the hash mapping network, It is the Frobenius norm. By minimizing the cross-modal consistency loss, paired cross-modal samples can be placed in close proximity in the hash space.

[0109] Cross-modal contrastive loss is used to enhance the model's ability to distinguish between samples with different semantic meanings, bringing semantically related cross-modal samples closer together and semantically unrelated cross-modal samples further apart. Within the current batch, paired image and text samples are treated as positive sample pairs, and all unpaired samples are treated as negative sample pairs. The cross-modal contrastive loss is calculated as follows:

[0110]

[0111] in, For the sample The continuous value characteristic, For the sample The corresponding continuous-value features of paired cross-modal samples, This is a temperature coefficient used to adjust the smoothness of the similarity distribution. Representation and Sample The set consists of all samples belonging to the same training batch. By optimizing the cross-modal contrastive loss, the model's ability to distinguish between different semantic categories can be effectively improved, reducing false matches during the retrieval process.

[0112] Intramodal structure loss is used to prevent oversmoothing in graph convolutional networks and enhance the semantic discriminability of samples within the same modality. When the number of graph convolutional layers is too large, oversmoothing can easily occur, causing all node features to become uniform, resulting in samples with different semantics being unable to be effectively distinguished. The intramodal structure loss is calculated as follows:

[0113]

[0114] in, These are the weighting coefficients for the text modality, used to balance the contributions of the image and text modalities to the loss. This indicates that all image samples in the current batch are traversed. This indicates that all text samples within the current batch are traversed. The first part of the formula corresponds to the structural loss within the image modality, and the second part corresponds to the structural loss within the text modality. By maximizing the distance between dissimilar samples within a modality, it is possible to effectively prevent features from converging to the same value and maintain the semantic discriminativeness of the samples.

[0115] In each training batch, the joint loss function value for the current batch is calculated, the gradients of all learnable parameters in the network are computed using the backpropagation algorithm, and then all parameters are updated using gradient descent. Parameters requiring updating include the linear correction operator, the weight matrices of all graph convolutional layers, and all parameters of the hash map network. During training, the model's retrieval performance is evaluated using a validation set at fixed training epochs. When the retrieval performance on the validation set no longer improves for several consecutive training epochs, the model is considered to have converged, training is stopped, and the final model parameters are saved.

[0116] Table 1 shows the ablation experiment results of the core module of the model, used to verify the independent contribution and synergistic effect of each key technical feature in this method on the model's retrieval performance. The first column of the table represents different model variants, the second column I→T represents the average retrieval accuracy of the image-to-text search task, the third column T→I represents the average retrieval accuracy of the text-to-image search task, and the fourth column Avg. represents the average retrieval accuracy of the two retrieval directions.

[0117] The model variants in the first column are: the baseline model CLIP, which uses only native CLIP features for hash mapping; the model CGC-GCN, which removes the batch structure correction module and retains only the static graph convolutional network; the model CGC-BSA, which removes the graph convolutional network and retains only the batch structure correction module; the model CGC-AGG, which removes the layer-by-layer graph topology update mechanism and uses only the fixed initial adjacency matrix; the model CGC-CLIP, which removes the graph convolutional network but retains the batch structure correction module and layer-by-layer feature update; and the full version method CGC(Full), which includes all core modules.

[0118] The data in the table shows that the baseline model using only native CLIP features has the lowest retrieval accuracy, indicating that relying solely on pre-trained features cannot fully uncover deep cross-modal semantic relationships in the data. Removing the graph convolutional network significantly reduces model performance, demonstrating the crucial role of graph structure modeling in capturing semantic associations between samples. Removing the batch structure correction module results in a substantial performance drop, indicating that batch structure correction effectively eliminates structural biases introduced by mini-batch sampling, improving the accuracy of the graph structure. Removing layer-by-layer graph topology updates causes the most significant performance decline, proving that dynamically updating the graph topology is the core innovation of this method, enabling the graph structure to continuously adjust as node features are optimized, more accurately reflecting true semantic associations. The complete method achieves the highest retrieval accuracy, indicating that the various core modules work together to enhance the model's cross-modal semantic representation capability and retrieval performance.

[0119] Table 1

[0120]

[0121] Additionally, considering that the linear correction operator used is a square matrix bound to the batch size, when the batch size needs to be adjusted during training, or when the pre-trained structural correction parameters need to be transferred to tasks with different batch sizes, the original operator may not be directly reusable, limiting the training flexibility of the model. This embodiment can also consider changing the global matrix correction to element-wise parameterized correction, using a lightweight correction network with two linear layers to replace the original square matrix M. This network takes the initial similarity value of a single edge as input and outputs the corrected edge weights. The correction network has only a few dozen parameters, is independent of the batch size, and can maintain consistent structural correction effects under any batch size. It also supports parameter transfer across tasks, further improving the model's generalization ability. Furthermore, since the cosine similarity of all node pairs needs to be recalculated before each layer of graph convolution, the computational complexity increases quadratically with the number of nodes, limiting further expansion of the batch size. Therefore, after calculating the complete adjacency matrix in the first layer, only the 32 most similar neighbor nodes are retained for each node to construct a sparse adjacency topology and cache it. In the subsequent topology update process of all layers, only the similarity values ​​of edges existing in the cached topology are calculated, and the full set of node pairs is no longer calculated. This scheme can reduce the computational complexity of topology updates by an order of magnitude. Furthermore, by filtering out a large number of semantically irrelevant weak connections, it can further reduce noise interference and improve the training speed and feature discriminative power of the model. The above considerations are feasible directions and do not affect the implementation examples corresponding to S1 to S8 above.

[0122] This application also provides an unsupervised cross-modal retrieval method. The unsupervised cross-modal retrieval method is implemented based on an unsupervised cross-modal hash model obtained through any of the above-mentioned unsupervised cross-modal hash model training method embodiments. The unsupervised cross-modal retrieval method includes the following steps:

[0123] S100, Obtain the cross-modal samples to be retrieved;

[0124] S200. An unsupervised cross-modal hashing model is used to process the cross-modal samples to be retrieved and generate the corresponding binary hash codes.

[0125] S300: Calculate Hamming distance based on binary hash code, and complete cross-modal retrieval based on Hamming distance.

[0126] This embodiment provides an unsupervised cross-modal retrieval method. Based on the unsupervised cross-modal hash model training method described above, it implements an unsupervised cross-modal hash model that can quickly complete cross-modal semantic retrieval tasks in large-scale multimedia datasets. It supports two core retrieval modes: image-to-text search and text-to-image search, and is adaptable to various complex real-world application input scenarios. This method uses the trained model to map samples from different modalities to a unified binary hash space and achieves fast similarity matching by calculating Hamming distance. The retrieval speed is significantly higher than retrieval methods based on original features, meeting the real-time requirements of applications.

[0127] First, the cross-modal samples to be retrieved are acquired. The form of these samples is unrestricted; they can be single image samples, single text samples, or a mixed set of multiple unpaired image and text samples. The samples to be retrieved do not need to correspond to any samples in the training set, nor do they require any manually labeled information; they can directly process any multimedia data collected in the real world. After acquiring the samples, the samples undergo the same preprocessing operations as in the training phase. For image samples, the image size is uniformly adjusted to the fixed size used during training, and the image pixel values ​​are normalized. For text samples, word segmentation, stop word removal, and encoding are performed to convert the natural language text into a vector form consistent with the format used in the training phase.

[0128] An unsupervised cross-modal hashing model, already trained, is used to process the preprocessed samples to be retrieved, generating corresponding binary hash codes. During the inference phase, no model parameters need to be updated, nor is it necessary to construct a batch cross-modal unified graph; computation can be completed using only the pre-trained feature extraction and hash mapping parts of the model. For unimodal image samples, the image encoder input to the model generates initial semantic features, which are then used to generate the corresponding binary hash codes via a hash mapping network and a symbolic function. For unimodal text samples, the text encoder input to the model generates initial semantic features, which are then used to generate the corresponding binary hash codes via a hash mapping network and a symbolic function. For mixed, unpaired cross-modal sample sets, each sample is processed independently to generate its own corresponding binary hash code; pairing relationships between samples are not required.

[0129] Hamming distance is calculated based on the generated binary hash codes, and cross-modal retrieval is performed based on the Hamming distance. Beforehand, binary hash codes are generated for all multimedia samples in the retrieval database using a trained model, constructing a hash retrieval database. This database stores the binary hash code of each sample along with its corresponding original sample information. When a retrieval request is received, the Hamming distance is calculated between the binary hash code generated for the query sample and all hash codes in the hash retrieval database. A smaller Hamming distance indicates a higher semantic relevance between the two samples. The samples in the database are sorted in ascending order of Hamming distance, and a predetermined number of samples at the top of the sorted list are returned as the final retrieval results. This method can complete the retrieval of millions of data points in milliseconds while maintaining high retrieval accuracy, making it suitable for various practical applications such as multimedia content management, intelligent search engines, and personalized recommendation systems.

[0130] Table 2 shows the comparison results of the inference robustness of cross-modal retrieval methods, used to verify the retrieval performance stability of this method under different real-world input scenarios. The data in the table is based on tests on the standard MIRFLICKR-25K dataset, and MAP@50 is used as the performance evaluation index, which is the average retrieval accuracy of the top 50 retrieval results. The first column of the table lists the methods being compared, namely the existing mainstream unsupervised cross-modal hashing method UCMFH and our method CGC; the second column is Scenario I, corresponding to a single-modal independent input scenario, where I→T represents the retrieval accuracy of the image-to-text search task, and T→I represents the retrieval accuracy of the text-to-image search task; the third column is Scenario II, corresponding to an unpaired mixed input scenario, which also includes two retrieval directions: image-to-text search and text-to-image search.

[0131] The data in the table shows that in single-modal independent input scenarios, our method achieves a retrieval accuracy of 0.940 in image-to-text search and 0.905 in text-to-image search, both significantly outperforming the comparative method UCMFH. In unpaired mixed input scenarios, while the retrieval accuracy decreases slightly, it remains at a high level of 0.889 and 0.866, respectively, and is significantly higher than the performance of the comparative method in the same scenarios. This indicates that our method not only exhibits excellent retrieval performance in standard paired input scenarios but also adapts well to common real-world single-modal independent input and unpaired mixed input scenarios, demonstrating strong inference robustness and practical application value.

[0132] Table 2

[0133]

[0134] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application.

Claims

1. A method for training an unsupervised cross-modal hashing model, characterized in that, Includes the following steps: S1. Obtain an unlabeled cross-modal dataset, which includes image samples and corresponding text samples; S2. Extract the initial semantic features of all image samples and all text samples in the cross-modal dataset, respectively; The cross-modal dataset is sampled in small batches according to a preset batch size to obtain the initial semantic features of the image and the initial semantic features of the text corresponding to the current batch. S3. Concatenate the initial semantic features of the image corresponding to the current batch with the initial semantic features of the text to construct a node set of a cross-modal unified graph; S4. Calculate the initial similarity between any two nodes in the node set and generate an initial adjacency matrix; S5. Perform batch structure correction on the initial adjacency matrix to eliminate the structural bias and noise introduced by small batch sampling, and obtain a robust adjacency matrix. S6. Based on the robust adjacency matrix, the graph topology and node feature representation are updated iteratively layer by layer through a multi-layer graph convolutional network, so that the graph topology and node feature representation are optimized synchronously. S7. Input the node features output from the last layer of graph convolution into the hash mapping network to generate binary hash codes corresponding to the image modality and the text modality; S8. The entire network is trained end-to-end using a joint loss function. After training, an unsupervised cross-modal hashing model is obtained.

2. The unsupervised cross-modal hashing model training method according to claim 1, characterized in that, The step of extracting the initial semantic features of all image samples and all text samples in the cross-modal dataset specifically includes: The pre-trained CLIP model is used to extract the initial semantic features of each image sample and each text sample in the cross-modal dataset.

3. The unsupervised cross-modal hashing model training method according to claim 1, characterized in that, S4 specifically includes: Calculate the cosine similarity between any two nodes in the node set, and use the cosine similarity as the edge weight between the corresponding nodes to generate the initial adjacency matrix.

4. The unsupervised cross-modal hashing model training method according to claim 1, characterized in that, The batch structure correction of the initial adjacency matrix specifically includes: The initial adjacency matrix is ​​linearly transformed by a learnable linear correction operator, and then the corrected adjacency matrix is ​​obtained by processing it with an activation function. The linear correction operator is initialized as an identity matrix and its parameters are updated during training.

5. The unsupervised cross-modal hashing model training method according to claim 4, characterized in that, The process of obtaining the robust adjacency matrix specifically includes: A weighted self-loop is added to the modified adjacency matrix, and the strength of message passing between nodes is controlled by adjusting the weight coefficients to obtain the robust adjacency matrix.

6. The unsupervised cross-modal hashing model training method according to claim 1, characterized in that, The method of iteratively updating the graph topology and node feature representation layer by layer through a multi-layer graph convolutional network specifically includes: Before each graph convolution operation, the similarity between nodes is recalculated based on the node features output by the previous graph convolution. Then, the recalculated similarity matrix is ​​subjected to batch structural correction to obtain the adjacency matrix of the current layer. Graph convolution is performed based on the adjacency matrix of the current layer to obtain the node feature representation of the current layer.

7. The unsupervised cross-modal hashing model training method according to claim 6, characterized in that, The multilayer graph convolutional network adopts a two-layer structure.

8. The unsupervised cross-modal hashing model training method according to claim 1, characterized in that, The joint loss function is obtained by weighted summation of cross-modal consistency loss, cross-modal contrast loss, and intramodal structure loss.

9. The unsupervised cross-modal hashing model training method according to claim 1, characterized in that, The hash mapping network is a two-layer feedforward neural network. The first layer uses the Leaky ReLU activation function, and the second layer uses the tanh activation function. The generated binary hash codes corresponding to the image modality and text modality specifically include: The output of the hash mapping network is processed by a symbol function to generate the binary hash code.

10. An unsupervised cross-modal retrieval method, characterized in that, The unsupervised cross-modal retrieval method is implemented using an unsupervised cross-modal hash model obtained by the unsupervised cross-modal hash model training method according to any one of claims 1 to 9. The unsupervised cross-modal retrieval method includes the following steps: S100, Obtain the cross-modal samples to be retrieved; S200. The unsupervised cross-modal hashing model is used to process the cross-modal sample to be retrieved, and the corresponding binary hash code is generated. S300. Calculate the Hamming distance based on the binary hash code, and complete the cross-modal retrieval based on the Hamming distance.

Images