Cross-media retrieval method and model training method based on multi-bit hash code

Abstract

This application discloses a cross-media retrieval method and model training method based on multi-bit hash codes. The method constructs a multi-bit hash code network model to fully capture the global features and local semantic features of multiple media information. By hierarchically aligning the global features and local semantic features of multiple media information, it fully utilizes fine-grained, more semantically rich local features to align heterogeneous multimedia data, effectively reducing the heterogeneity and semantic gaps between different media. At the same time, the application uses the aligned features to collaboratively and simultaneously generate multiple high-quality hash codes of different lengths, overcoming the problem of very limited flexibility, accuracy, and scalability of cross-media retrieval caused by using fixed-length hash codes, thereby improving the accuracy and flexibility of cross-media retrieval.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a cross-media retrieval method and model training method based on multi-bit hash codes. Background Art

[0002] Cross-media retrieval is an important similarity search technology, which refers to retrieving data semantically related to a given media query term (e.g., text) from another media representation (e.g., an image). In particular, among cross-media retrieval methods, the cross-media hashing method, as one of the efficient retrieval methods, can convert the original data into a compact binary code representation, thereby performing fast retrieval. However, one of the problems with cross-media hashing methods today is that due to the huge amount of multimedia data and the complex heterogeneity, where heterogeneity refers to the fact that different media data exist in "heterogeneous" feature spaces with different distributions, the accuracy of cross-media retrieval is greatly affected. In addition, another problem is that almost all existing cross-media hashing methods can only learn single-bit hash codes, that is, a hash code of a fixed length, such as 32 bits, which also affects the accuracy of cross-media retrieval. Summary of the Invention

[0003] The technical problem to be solved by this application is to provide a cross-media retrieval method and model training method based on multi-bit hash codes in response to the shortcomings of the existing technology.

[0004] In order to solve the above technical problems, the first aspect of the embodiments of the present application provides a training method for a multi-bit hash code network model, wherein the training method for the multi-bit hash code network model specifically includes:

[0005] Obtaining a training sample set, and constructing a semantic similarity matrix using the semantic label of each training sample in the training sample set, wherein the training sample set includes a plurality of training batches, and each training batch includes a plurality of image-text pairs;

[0006] Each image-text pair in the training batch is input into the initial hash model, and the initial hash model is used to determine the text global features, text local feature sequence, image global features, and image local feature sequence of the training sample;

[0007] Aligning the text global features and the image global features to obtain aligned text global features and aligned image global features, learning text semantic features of the text local feature sequence and image semantic features of the image local feature sequence based on shared concept embedding, and constructing a cross-modal contrast loss term based on the aligned text global features, the aligned image global features, the text semantic features, and the image semantic features;

[0008] Determining a text fusion feature based on the aligned text global feature and the text semantic feature, and determining an image fusion feature based on the aligned image global feature and the image semantic feature;

[0009] Determining at least two first hash features based on the text fusion features, and determining at least two second hash features based on the image fusion features;

[0010] constructing a hash loss term based on the semantic similarity matrix and at least two first hash features and at least two second hash features of each image-text pair in the training batch;

[0011] Based on the cross-modal contrast loss term and the hash loss term, the parameters of the initial hash model are updated to obtain a multi-bit hash code network model.

[0012] The training method of the multi-bit hash code network model, wherein the multi-bit hash code network model includes a feature extraction module, a cross-media contrastive alignment module and a multi-hash collaborative learning module, the feature extraction module is connected to the cross-media contrastive alignment module, and the cross-media contrastive alignment module is connected to the multi-hash collaborative learning module, wherein the feature extraction module is used to extract text global features, text local feature sequences, image global features and image local feature sequences; the cross-media contrastive alignment module is used to determine text fusion features and image fusion features based on the text global features, the image global features, the text local feature sequences and the image local feature sequences; the multi-hash collaborative learning module is used to determine at least two first hash features based on the text fusion features, and determine at least two first hash codes based on the at least two first hash features, and determine at least two second hash features based on the image fusion features, and determine at least two second hash codes based on the at least two second hash features.

[0013] The training method of the multi-bit hash code network model, wherein the cross-media contrastive alignment module includes a residual MLP unit, a local learning unit, a pooling layer and an adder, the residual MLP unit and the adder are connected, and the local learning unit is connected to the adder through the pooling layer, wherein the local learning unit includes a cross-attention layer and a Transformer layer connected in sequence, and the query vector of the cross-attention layer is a shared concept embedding.

[0014] The training method of the multi-bit hash code network model, wherein the cross-modal contrast loss term is constructed based on aligned text global features, aligned image global features, text semantic features, and image semantic features, is specifically:

[0015] For each image-text pair in the training batch, determine the aligned text global features of the image-text pair and the global features of each aligned image in the training batch The first similarity of the image-text pair; the global features of the aligned images and the global features of each aligned text in the training batch The second similarity;

[0016] Determining a global contrast loss term based on all the determined first similarities and all the determined second similarities;

[0017] Determine each text semantic feature in the text semantic feature sequence for each image-text pair Each image semantic feature in the image semantic feature sequence of the image-text pair The third similarity, and each image semantic feature in the image semantic feature sequence of the image-text pair Each text semantic feature in the text semantic feature sequence of the image-text pair The fourth similarity;

[0018] Determining a local contrast loss term based on all the determined third similarities and all the determined fourth similarities;

[0019] According to the global contrast loss term and the local contrast loss term, a cross-modal contrast loss term is determined.

[0020] The training method of the multi-bit hash code network model, wherein the constructing of the hash loss term based on the at least two first hash features, the at least two second hash features, and the semantic similarity matrix specifically includes:

[0021] For each image-text pair in the training batch, determine each first hash feature of the image-text pair and the first hash feature of each image-text pair in the training set The inner product of the first hash feature, each second hash feature of the image-text pair and the second hash feature of each image-text pair in the training set Determine the inner product of the second hash feature of the image-text pair The second hash feature of each image-text pair in the training set The inner product of the third hash feature of the image-text pair and each second hash feature of the image-text pair and the first hash feature of each image-text pair in the training set The inner product of the fourth hash feature;

[0022] Determining a media loss item based on the determined inner products of all first hash features, the inner products of all second hash features, and a semantic similarity matrix;

[0023] determining an inter-media loss term based on the determined inner products of all third hash features, the inner products of all fourth hash features, and a semantic similarity matrix;

[0024] For each image-text pair in the training batch, according to each first hash feature and its corresponding second hash feature Determine hash signature b i,w , and determine all hash features b based on i,w , all first hash features and all second hash features Determine the hash quantization term;

[0025] A hash loss term is determined according to the intra-media loss term, the inter-media loss term, and the hash quantization term.

[0026] The training method of the multi-bit hash code network model, wherein, before updating the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain the multi-bit hash code network model, the method further includes:

[0027] For each image-text pair in the training batch, determining a text-assisted hash feature based on the text fusion feature, and determining an image-assisted hash feature based on the image fusion feature;

[0028] Learning a text hash code according to the text auxiliary hash feature, learning an image hash code according to the image auxiliary hash feature, and determining an auxiliary hash code according to the text hash code and the image hash code;

[0029] Mapping the at least two first hash features and the at least two second hash features to the Hamming space where the text auxiliary hash feature is located to obtain at least two first hash codes and at least two second hash codes;

[0030] determining an auxiliary hash loss term based on the text auxiliary hash features and the image auxiliary hash features of the image-text pairs in the training batch, and determining a reconstruction loss term based on the auxiliary hash code, the at least two first hash codes, and the at least two second hash codes;

[0031] A modified loss term is determined according to the auxiliary hash loss term and the reconstruction loss term, and the hash loss term is modified based on the modified loss term.

[0032] A second aspect of an embodiment of the present application provides a cross-media retrieval method based on a multi-bit hash code, wherein the cross-media retrieval method based on a multi-bit hash code specifically includes:

[0033] Obtaining a query sample, and determining at least a query hash code sequence of the query sample using the multi-bit hash code network model, wherein the query hash code sequence includes at least two hash codes;

[0034] Obtaining a target data set corresponding to the query sample, and determining a hash code sequence set corresponding to the target data set using the multi-bit hash code network model;

[0035] According to the query hash code sequence and the hash code sequence set, target data corresponding to the query sample is determined in the target data set.

[0036] A third aspect of an embodiment of the present application provides a training device for a multi-bit hash code network model, wherein the device specifically includes:

[0037] an acquisition module, configured to acquire a training sample set and construct a semantic similarity matrix using the semantic labels of each training sample in the training sample set, wherein the training sample set includes a plurality of training batches, and each training batch includes a plurality of image-text pairs;

[0038] An initial hash model is used to input each image-text pair in a training batch into the initial hash model, determine the text global features, text local feature sequences, image global features, and image local feature sequences of the training samples through the initial hash model; align the text global features and the image global features to obtain aligned text global features and aligned image global features, learn text semantic features of the text local feature sequences and image semantic features of the image local feature sequences based on shared concept embedding, and construct a cross-modal contrast loss term based on the aligned text global features, aligned image global features, text semantic features, and image semantic features; determine a text fusion feature based on the aligned text global features and the text semantic features, and determine an image fusion feature based on the aligned image global features and the image semantic features; determine at least two first hash features based on the text fusion features, and determine at least two second hash features based on the image fusion features; and construct a hash loss term based on a semantic similarity matrix and the at least two first hash features and at least two second hash features of each image-text pair in the training batch;

[0039] A parameter updating module is used to update the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain a multi-bit hash code network model.

[0040] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of any multi-bit hash code network model as described above.

[0041] A fifth aspect of the embodiments of the present application provides a terminal device, comprising: a processor and a memory;

[0042] The memory stores a computer-readable program executable by the processor;

[0043] When the processor executes the computer-readable program, the processor implements the steps in any of the above-described methods for training a multi-bit hash code network model.

[0044] Beneficial Effects: Compared with existing technologies, this application constructs a multi-bit hash code network model to obtain global features and local semantic tokens of multiple media information. By hierarchically aligning the global features and local semantic tokens of multiple media information, it effectively reduces the heterogeneity and semantic gaps between different media. At the same time, multiple high-quality hash codes of different lengths are generated simultaneously using the aligned global features and local semantic tokens of each media information, thereby improving the efficiency and accuracy of cross-media retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a network model structure diagram of the training method of the multi-bit hash code network model provided in an embodiment of the present application.

[0047] Figure 2 This is a flowchart of an embodiment of a method for training a multi-bit hash code network model provided in an embodiment of the present application.

[0048] Figure 3 A schematic diagram of the structure of a training device for a multi-bit hash code network model provided in an embodiment of the present application.

[0049] Figure 4 This is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] This application provides a cross-media retrieval method and model training method based on multi-bit hash codes. To make the purpose, technical solutions, and effects of this application clearer and more explicit, the application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to explain this application and are not intended to limit this application.

[0051] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0053] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0054] Research has found that with the rapid development of smart devices and social media, multimedia data is experiencing an unprecedented explosive growth. Furthermore, multimedia data is collected from diverse sources and represented in diverse formats, resulting in heterogeneous multimedia data, such as text, images, and videos. The sheer volume and complex heterogeneity of multimedia data—heterogeneous data structures in which different media data reside in differently distributed "heterogeneous" feature spaces, resulting in inconsistent feature representations and distributions—significantly impacts the efficiency and accuracy of cross-media retrieval.

[0055] Cross-media retrieval is an important similarity search technology, which refers to retrieving data semantically related to a given media query term (e.g., text) from a media representation (e.g., an image), that is, retrieving text semantically related to the image based on the image. In particular, among cross-media retrieval methods, the cross-media hashing method, as one of the efficient retrieval methods, can convert the original data into a compact binary code representation, thereby performing fast retrieval. Nowadays, one of the problems with cross-media hashing methods is how to alleviate the heterogeneity between multiple media data to improve the accuracy of cross-media retrieval. In addition, another problem is how to use a unified architecture to simultaneously generate high-quality hash codes of multiple lengths to improve the accuracy and efficiency of cross-media retrieval, thereby flexibly meeting the different requirements of different platform systems for retrieval performance and retrieval efficiency.

[0056] Currently, many hashing-based cross-media retrieval methods have been proposed to support fast similarity searches across different media data. Existing cross-media retrieval methods can be divided into different subcategories from different perspectives. Typically, they can be divided into unsupervised and supervised methods based on whether they utilize supervised information, shallow and deep methods based on whether they are based on deep learning, and single-bit and multi-bit methods based on the length and number of hash codes generated by the model.

[0057] For unsupervised and supervised cross-media hashing methods, according to whether manually annotated semantic tags are involved, existing cross-media hashing methods can be roughly divided into two categories, namely unsupervised cross-media hashing methods and supervised cross-media hashing methods.

[0058] Unsupervised cross-media hashing methods focus on preserving semantic similarity by exploring the correlations between cross-media data pairs without using supervised information (i.e., pairwise similarity matrices or semantic labels). Unsupervised methods can be easily extended to handle situations where label information is scarce or expensive to obtain for cross-media retrieval tasks.

[0059] It's worth noting that unsupervised cross-media hashing methods focus on capturing the correlations between multiple media data embedded in the original space and fail to consider high-level semantic supervision to learn discriminative binary hash codes. In contrast, supervised cross-media hashing fully captures the value of supervised information, leveraging semantic labels to enhance the discriminative power of the learned hash codes. As a result, supervised cross-media hashing typically achieves superior retrieval performance.

[0060] Existing cross-media hashing methods can be divided into two categories, shallow and deep, depending on whether they utilize deep learning techniques. Shallow cross-media hashing methods often employ a two-stage learning paradigm: feature extraction and hash code learning. However, the separate feature extraction and hash code learning processes lead to incompatibility issues, meaning that the extracted features are not optimal for learning hash codes.

[0061] In recent years, due to the powerful representational and nonlinear modeling capabilities of deep networks, many deep cross-media hashing methods based on multi-layer neural networks have been proposed. These methods utilize end-to-end training to generate informative deep features and learn high-quality hash codes based on these features. Empirical evidence shows that end-to-end deep learning architectures are more suitable for learning more discriminative hash codes than traditional shallow cross-media hashing methods that use handcrafted features.

[0062] For both single-bit and multi-bit cross-media hashing methods, almost all existing cross-media hashing methods can only learn single-bit hash codes—hash codes of a fixed length, such as 32 bits. When learning hash codes of varying lengths, such as 16, 32, or 64 bits, these single-bit methods must modify the dimensionality of the output layer and retrain the entire learning network for each high-quality hash code, which consumes significant time and resources. To mitigate heterogeneity across different media, most deep cross-media hashing methods primarily maintain pairwise cross-media similarity by utilizing only global embeddings. However, leveraging global representations alone is insufficient for discriminative hash learning. Intuitively, a target object containing important semantics may occupy only a small portion of the original data, which can be easily overlooked by global features. Consequently, these frameworks inevitably produce suboptimal hash codes. Furthermore, this learning paradigm is severely limited in flexibility and scalability in practice, as real-world systems must balance efficiency and performance. Typically, short hash codes enable faster retrieval but suffer from information loss. Conversely, long hash codes achieve better retrieval performance but are less computationally efficient than short hash codes. Therefore, it is extremely difficult to determine the desired length of the hash code to meet the performance and storage requirements in real scenarios, which inevitably leads to tediously adjusting the code length by retraining the network.

[0063] In order to solve the above problems, in an embodiment of the present application, a training sample set is obtained, and a semantic similarity matrix is ​​constructed using the semantic labels of each training sample in the training sample set, wherein the training sample set includes several training batches, and each training batch includes several image-text pairs; each image-text pair in the training batch is input into an initial hash model, and the text global features, text local feature sequence, image global features and image local feature sequence of the training sample are determined by the initial hash model; the text global features and the image global features are aligned to obtain aligned text global features and aligned image global features, and the text semantic features of the text local feature sequence and the image semantic features of the image local feature sequence are learned according to the shared concept embedding, and based on the aligned text, the text global features and the image local feature sequence are learned. The present invention constructs a cross-modal contrast loss term based on the global features, aligned image global features, text semantic features, and image semantic features; determines a text fusion feature based on the aligned text global features and the text semantic features, and determines an image fusion feature based on the aligned image global features and the image semantic features; determines at least two first hash features based on the text fusion feature, and determines at least two second hash features based on the image fusion feature; constructs a hash loss term based on the semantic similarity matrix and at least two first hash features and at least two second hash features of each image-text pair in the training batch; updates the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain a multi-bit hash code network model. The present invention obtains global features and local semantic tokens of multiple media information by constructing a multi-bit hash code network model, and effectively reduces the heterogeneity and semantic gap between different media by hierarchically aligning the global features and local semantic tokens of multiple media information. At the same time, multiple high-quality hash codes of different lengths are generated simultaneously through the aligned global features and local semantic tokens of each media information, thereby improving the efficiency and accuracy of cross-media retrieval.

[0064] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0065] To illustrate the specific implementation process of the training method of the multi-bit hash code network model provided in the embodiment of the present application, the multi-bit hash code network model is first described here. The multi-bit hash code network model is used to obtain multiple hash codes of different lengths of media information, such as 16 bits, 32 bits, 64 bits, etc.

[0066] Specifically, if Figure 1As shown, the multi-bit hash code network model includes a feature extraction module, a cross-media contrast alignment module and a multi-hash collaborative learning module, the feature extraction module is connected to the cross-media contrast alignment module, and the cross-media contrast alignment module is connected to the multi-hash collaborative learning module, wherein the feature extraction module is used to extract text global features, text local feature sequences, image global features and image local feature sequences; the cross-media contrast alignment module is used to determine text fusion features and image fusion features based on text global features, the image global features, text local feature sequences and image local feature sequences; the multi-hash collaborative learning module is used to determine at least two first hash features based on the text fusion features, and determine at least two first hash codes based on the at least two first hash features, and determine at least two second hash features based on the image fusion features, and determine at least two second hash codes based on the at least two second hash features.

[0067] Specifically, the feature extraction module may include multiple Transformer encoders, wherein the Transformer encoders are used to encode media information to obtain global features and local features of the media. In an embodiment of the present application, the feature extraction module may include two Transformer encoders, wherein one Transformer encoder is used to extract global text features and a sequence of local text features, and the other Transformer encoder is used to extract global image features and a sequence of local image features.

[0068] The cross-media comparative alignment module includes a residual MLP unit, a local learning unit, a pooling layer, and an adder. The residual MLP unit and the adder are connected, and the local learning unit is connected to the adder through the pooling layer. The local learning unit includes a cross-attention layer and a Transformer layer connected in sequence. The query vector of the cross-attention layer is a shared concept embedding. The residual MLP unit includes a weight-shared residual multi-layer perceptron block (Residual Multi-layer Perceptron, ResMLP). The residual multi-layer perceptron block is composed of one or more identical blocks stacked together, and each block contains a multi-layer perceptron with a residual connection. In this embodiment, the residual multi-layer perceptron block is composed of a stack of two identical blocks.

[0069] In an embodiment of the present application, the cross-media comparative alignment module includes two residual MLP units, local learning units, pooling layers and adders. The two residual MLP units, local learning units, pooling layers and adders have the same composition structure, one of which is used to perform data processing on the global features of the text and the local feature sequence of the text to determine the text fusion features, and the other is used to perform data processing on the global features of the image and the local feature sequence of the image to determine the image fusion features.

[0070] The multi-hash collaborative learning module includes a hash linear projection layer, wherein the hash linear projection layer includes a plurality of hash encoders, and the hash encoders are used to encode the text fusion features and the image fusion features to obtain a plurality of high-quality hash codes of different lengths. In an embodiment of the present application, the multi-hash collaborative learning module includes a two-way hash linear projection layer, one of which is used to determine at least two first hash codes based on the text fusion features, and the other is used to determine at least two second hash codes based on the image fusion features.

[0071] like Figure 2 As shown, the training method of a multi-bit hash code network model provided in this embodiment specifically includes steps S10-S70.

[0072] S10: Obtain a training sample set, and construct a semantic similarity matrix using the semantic label of each training sample in the training sample set, wherein the training sample set includes several training batches, and each training batch includes several image-text pairs.

[0073] Specifically, an image-text pair includes an image and text that are semantically related. For example, if image A is a truck driving on a street, and the truck is red and yellow, and text B is "a truck driving on a street with red and yellow colors", image A and text B can be an image-text pair. In this embodiment, the image-text pair is i It can represent: in, represents the image of the i-th image-text pair, represents the dimension of the image, represents the text of the i-th image-text pair, Represents the dimension of the text, l i ∈{0,1} 1×Crepresents the semantic label of the i-th image-text pair, C represents the number of semantic category labels in the training sample set, and the semantic label is the semantic category label based on the semantic annotation of the image-text pair. The image-text pair can be a single-label sample or a multi-label sample, that is, the semantic label can include one semantic category label or multiple semantic category labels. In this embodiment of the present application, the image-text pair has multiple semantic category labels, that is, the semantic label of the image-text pair is a multi-label sample.

[0074] The training sample set includes several training batches, each of which is used as training data for one training cycle of the initial hash model to train the initial hash model to obtain a multi-bit hash code network model. In other words, the image-text pair is a training sample for training the initial hash model. For ease of explanation, in the embodiment of the present application, the training sample set is represented as Where N represents the number of training samples in the training sample set.

[0075] The semantic similarity matrix is ​​constructed based on the semantic labels of the image-text pairs, and is used to reflect the similarity of each image-text pair in the training sample set in the semantic space. Specifically, for N training samples in the training sample set, an N×N semantic similarity matrix S can be generated. ij , where S ij The value of S is 0 or 1. ij =1 indicates the i-th image-text pair o in the training sample set i and the jth image-text pair o j have at least one identical semantic category label, S ij = 0 represents the i-th image-text pair o in the training sample set i and the jth image-text pair o j Do not have the same semantic category labels.

[0076] S20. Input each image-text pair in the training batch into the initial hash model, and determine the text global features, text local feature sequence, image global features, and image local feature sequence of the training sample through the initial hash model.

[0077] Specifically, the initial hash model and the multi-bit hash code network model have the same model structure, differing only in model parameters. The initial hash model uses the initial network model, while the multi-bit hash code network model uses the network model trained using the training sample set. In other words, by training the initial hash model using the training sample set, a trained multi-bit hash code network model can be obtained.

[0078] The global text feature is a global feature obtained by extracting features from the entire global information of the text. The local text feature sequence includes a plurality of local text features, wherein the local text features are local region features obtained by dividing the text into a plurality of phrases and extracting features from each phrase. For example, if text A is divided into five phrases and feature extraction is performed on each of these five phrases, five local text features can be obtained, and these five local text features can constitute the local text feature sequence of the text.

[0079] The global image features are global features obtained by extracting features from the entire global image information, such as shape, texture, color, etc. The local image feature sequence includes a plurality of local image features, wherein the local image features are local region features obtained by segmenting the image into a plurality of image blocks and extracting features from each image block.

[0080] In an embodiment of the present application, the text is divided into several phrases, and combined with the position code obtained by pre-encoding the text position information, feature extraction is performed on the entire text global information through one of the Transformer encoders of the feature extraction module (for example, the GPT-2 encoder) to obtain the global features of the text, and feature extraction is performed on each phrase to obtain several local features of the text, thereby combining the several local features of the text into a local feature sequence of the text.

[0081] The image is divided into 3×3 image blocks, and combined with the position code obtained by pre-encoding the image position information, another Transformer encoder (for example, ViT encoder) is used to extract features from the global information of the entire image to obtain global image features. Feature extraction is then performed on each image block to obtain several local image features, which are then combined into an image local feature sequence.

[0082] For the sake of convenience, in this embodiment of the present application, the text features in the i-th image-text pair in the training sample set are and image features They can be expressed as:

[0083]

[0084]

[0085] in, and They represent the global text features and image global features in the i-th image-text pair in the training sample set, R 1×d The feature dimension representing the global features, and They represent the text local feature sequence and image local feature sequence in the i-th image-text pair in the training sample set, respectively, and L t The number of local tokens representing the local feature sequence of the text, L v The number of local tokens representing the local feature sequence of the image, The feature dimension representing the local feature sequence of the text, The feature dimension of the local feature sequence of the image.

[0086] S30. Align the text global features and the image global features to obtain aligned text global features and aligned image global features, embed the learning text semantic features of the text local feature sequence and the image semantic features of the image local feature sequence according to the shared concept, and construct a cross-modal contrast loss term based on the aligned text global features, the aligned image global features, the text semantic features and the image semantic features.

[0087] Specifically, the alignment of text global features is to map the text global features to the same dimensional space representation. The alignment of image global features is to map the image global features to the same dimensional space representation. In this embodiment, the text global features are mapped to the same dimensional space representation. Input one of the residual MLP units, and map the text global features to the same dimensional space representation through the residual multilayer perceptron block (ResMLP) in one of the residual MLP units to obtain the aligned text global features; the image global features The image global features are then input into another residual MLP unit, and the weight-sharing residual multilayer perceptron block (ResMLP) in the other residual MLP is used to map the image global features to the same dimensional space representation to obtain the aligned image global features. The aligned text global features and aligned image global features can be expressed as:

[0088]

[0089]

[0090] in, Represents the global features of aligned text, represents the global features of the aligned image, Represents the global features of the text, Represents the global features of the image, ResMLP represents the mapping function, θ res is the trainable weight coefficient, R d The feature dimension representing the global features of aligned text or aligned image.

[0091] The text semantic feature is based on a concept perspective and converts the local text features in the text local feature sequence into a fine-grained and fixed-length semantic token representation. The image semantic feature is based on a concept perspective and converts the local image features in the image local feature sequence into a fine-grained and fixed-length semantic token representation.

[0092] In the embodiment of the present application, text semantic features and image semantic features are obtained by using local learning units. In other words, the role of the local learning unit is to selectively aggregate low-level, variable-length text local features or image local features to obtain multiple fine-grained fixed-length text semantic features or multiple image semantic features.

[0093] Specifically, by presetting P shared concepts to embed Q∈R P×d , P represents the number of shared concept embeddings, d represents the embedding dimension of the shared concept embedding, and the shared concept embedding Q is input as the key value of the query vector and the text local feature sequence into the cross attention layer of the local learning unit in one way to obtain the text attention map; the shared concept embedding Q is input as the key value of the query vector and the image local feature sequence into the cross attention layer of the local learning unit in the other way to obtain the image attention map.

[0094] The text attention map and the image attention map can be expressed as follows:

[0095]

[0096]

[0097] in, represents the text attention map, represents the image attention map, represents a sequence of local features of the text, represents the local feature sequence of the image, Q represents the shared concept embedding, d represents the embedding dimension of the shared concept embedding Q, represents the learnable matrix, P represents the number of shared concept embeddings, L t The number of local tokens representing the local feature sequence of the text, L v The number of local tokens representing the local feature sequence of the image, and They represent the dimensions of the text attention map and image attention map respectively, and Softmax represents the Softmax activation function.

[0098] Furthermore, based on the text attention map and the text local feature sequence, a text coarse semantic representation can be obtained. Based on the image attention map and the image local feature sequence, an image coarse semantic representation can be obtained. The text coarse semantics and the image coarse semantics can be expressed as:

[0099]

[0100]

[0101] in, Represents the coarse semantic representation of the text, represents the text attention map, represents a sequence of local features of the text, represents the coarse semantic representation of the image, represents the image attention map, represents the local feature sequence of the image, is a learnable matrix, P represents the number of shared concept embeddings, d represents the embedding dimension of shared concept embeddings, and R P×d The dimension representing the coarse semantic representation of text or image.

[0102] The coarse semantic representation of the text is input into the Transformer layer of one of the local learning units to obtain text semantic features, and all the text semantic features are combined into a text semantic feature sequence; at the same time, the coarse semantic representation of the image is input into the Transformer layer of the other local learning unit to obtain image semantic features, and all the image semantic features are combined into an image semantic feature sequence. The Transformer layer of the local learning unit includes a Transformer encoder block, which is used to obtain refined text semantic features by capturing the correlation between coarse semantic tags of the text, and to obtain refined image semantic features by capturing the correlation between coarse semantic tags of the image. The text semantic feature sequence and the image semantic feature sequence can be expressed as:

[0103]

[0104]

[0105] in, Represents a sequence of text semantic features, represents the image semantic feature sequence, Represents the coarse semantic representation of the text, represents the coarse semantic representation of the image, is the trainable weight coefficient, P represents the number of shared concept embeddings, d represents the embedding dimension of shared concept embeddings, and RP×d The dimension representing the sequence of text semantic features or image semantic features.

[0106] This embodiment uses the cross-media comparative alignment module and the residual MLP unit to obtain aligned text global features and aligned image global features, achieve hierarchical alignment of global features between different media, reduce redundant feature representations, and introduce a cross-attention mechanism to retain important contextual information. It selectively aggregates low-level, variable-length text local features and image local features into fine-grained fixed-length text semantic features and image semantic features through the Transformer layer, achieves hierarchical alignment of local semantic tokens between different media, and effectively bridges media heterogeneity to reduce the heterogeneity and semantic gap between different media.

[0107] In one implementation of this embodiment, the cross-modal contrast loss term is constructed based on the aligned text global features, aligned image global features, text semantic features, and image semantic features as follows:

[0108] S31, for each image-text pair in the training batch, determining the aligned text global features of the image-text pair and the global features of each aligned image in the training batch The first similarity of the image-text pair; the global features of the aligned images and the global features of each aligned text in the training batch The second similarity;

[0109] S32. Determine a global contrast loss item based on all the determined first similarities and all the determined second similarities;

[0110] S33, determining each text semantic feature in the text semantic feature sequence of each image-text pair Each image semantic feature in the image semantic feature sequence of the image-text pair The third similarity, and each image semantic feature in the image semantic feature sequence of the image-text pair Each text semantic feature in the text semantic feature sequence of the image-text pair The fourth similarity;

[0111] S34. Determine a local contrast loss item based on all the determined third similarities and all the determined fourth similarities;

[0112] S35. Determine a cross-modal contrast loss term based on the global contrast loss term and the local contrast loss term.

[0113] Specifically, in step S31 , the first similarity and the second similarity are used to determine the similarity of global features of the image and text of each image-text pair in the training batch.

[0114] For ease of explanation, in the embodiment of the present application, the training batch contains M image-text pairs. For the i-th text in each image-text pair in the training batch, the i-th image with similar semantics can be regarded as a positive sample for contrastive learning, and the dissimilar image can be regarded as a negative sample for contrastive learning, that is, images that are semantically related to the text based on text matching. Similarly, for the i-th image, the i-th text with similar semantics can be regarded as a positive sample for contrastive learning, and the dissimilar text can be regarded as a negative sample for contrastive learning, that is, texts that are semantically related to the image based on image matching.

[0115] Based on this, the i-th alignment text global feature of the image-text pair can be and the global features of the cth aligned image in the training batch The first similarity is determined by inputting the global contrast alignment module to perform global contrast alignment, and then the global features of the image-text pair are aligned by the i-th alignment image. and the cth aligned text global feature in the training batch The first and second similarities are input into a global alignment module for global alignment to determine a second similarity. The first similarity and the second similarity can be expressed as:

[0116]

[0117]

[0118] in, represents the first similarity, represents the second similarity, τ1 is the temperature hyperparameter, M represents the number of training samples included in the training batch, represents the i-th aligned text global feature, represents the c-th aligned image global feature, represents the global feature of the i-th aligned image, represents the cth aligned text global feature.

[0119] In step S32, after obtaining the first similarity and the second similarity, a global contrast loss term can be calculated based on the first similarity and the second similarity, wherein the global contrast loss term can be determined by summing the first similarity and the second similarity, or by weighting the first similarity and the second similarity. In the embodiment of the present application, the global contrast loss term can be expressed as:

[0120]

[0121] in, represents the global contrast loss term, M represents the number of training samples included in the training batch, represents the first similarity, Indicates the second similarity.

[0122] In step S33 , the third similarity and the fourth similarity are used to determine the similarity of local features of the image and text of each image-text pair in the training batch.

[0123] Specifically, the positive samples for contrastive learning in the local contrastive alignment module consist of the semantic tags corresponding to the image and text in the image-text pairs in the training batch, and the negative samples consist of the semantic tags of the image and the remaining text in the image-text pairs. For example, the image in the i-th image-text pair in the training batch and the p-th semantic tag of the text in the i-th image-text pair can constitute a positive sample, or the text in the i-th image-text pair in the training batch and the p-th semantic tag of the image in the i-th image-text pair can constitute a positive sample.

[0124] In the embodiment of the present application, the text semantic feature sequence includes a plurality of text semantic features, and the pth text semantic feature in the text semantic feature sequence of the image-text pair is The jth image semantic feature in the image semantic feature sequence of the image-text pair Perform local comparison alignment and determine the third similarity based on comparison learning; the image semantic feature sequence includes several image semantic features, and the pth image semantic feature in the image semantic feature sequence of the image-text pair is The jth text semantic feature in the text semantic feature sequence of the image-text pair Perform local comparison and alignment, and determine a fourth similarity based on comparison learning. The third similarity and the fourth similarity can be expressed as:

[0125]

[0126]

[0127] in, represents the third similarity, represents the fourth similarity, Represents a sequence of text semantic features The pth text semantic feature in Represents a sequence of image semantic features The p-th image semantic feature in Represents a sequence of image semantic features The jth image semantic feature in Represents a sequence of text semantic features where P represents the number of image semantic features or the number of text semantic features, and τ2 is the temperature hyperparameter.

[0128] In step S34, the local contrast loss term can be determined by summing the third similarity and the fourth similarity, or by weighting the third similarity and the fourth similarity. In the embodiment of the present application, the local contrast loss term can be expressed as:

[0129]

[0130] in, Represents the local contrast loss term, M represents the number of training samples included in the training batch, P represents the number of image semantic features or the number of text semantic features, represents the third similarity, Indicates the fourth similarity.

[0131] In step S35, the cross-modal contrast loss term can be determined by summing the global contrast loss term and the local contrast loss term, or by weighting the global contrast loss term and the local contrast loss term. The cross-modal contrast loss term can be expressed as:

[0132]

[0133] in, represents the cross-modal contrast loss term, represents the global contrast loss term, represents the local contrast loss term, and α represents the balancing hyperparameter.

[0134] S40: Determine a text fusion feature based on the aligned text global feature and the text semantic feature, and determine an image fusion feature based on the aligned image global feature and the image semantic feature.

[0135] Specifically, the text fusion feature is a feature that fuses the global features and local features of the text. The image fusion feature is a feature that fuses the global features and local features of the image. In an embodiment of the present application, the text fusion feature can be obtained by inputting the text semantic features into one of the pooling layers in the text media contrast alignment module for pooling operation, and the text semantic features after the pooling operation and the aligned text global features are input into one of the adders in the text media contrast alignment module. Similarly, the image fusion feature can be obtained by inputting the image semantic features into another pooling layer in the image media contrast alignment module for pooling operation, and the image semantic features after the pooling operation and the aligned image global features are input into another adder in the text media contrast alignment module. Wherein, the text fusion feature and the image fusion feature can be respectively expressed as:

[0136]

[0137]

[0138] in, represents the text fusion feature, represents the image fusion feature, Represents the global features of aligned text, represents the global features of the aligned image, Represents a sequence of text semantic features, represents the image semantic feature sequence, and GAP represents global average pooling.

[0139] S50: Determine at least two first hash features based on the text fusion feature, and determine at least two second hash features based on the image fusion feature.

[0140] Specifically, the first hash feature is a text hash feature. The second hash feature is an image hash feature. In an embodiment of the present application, the text fusion feature is input into one of the hash linear projection layers of the multi-hash collaborative learning module, and the text fusion feature is encoded through W hash encoders to obtain W text hash features, i.e., the first hash feature; the image fusion feature is input into another hash linear projection layer, and the image fusion feature is encoded through W hash encoders to obtain W image hash features, i.e., the second hash feature. Wherein, the first hash feature and the second hash feature can be expressed as:

[0141]

[0142]

[0143] in, represents the first hash feature, represents the second hash feature, represents the text fusion feature, Represents image fusion features, Encoder represents encoding function, θ enc,w is the trainable weight parameter, K w Indicates the length of the wth hash code, Represents the dimension of the w-th hash code.

[0144] This embodiment uses a multi-hash collaborative learning module to synchronously learn multiple hash features of different lengths for different media based on text fusion features and image fusion features, so as to simultaneously generate multiple hash codes of different lengths for different media, thereby improving the efficiency and accuracy of cross-media retrieval.

[0145] S60: Construct a hash loss term according to the semantic similarity matrix and at least two first hash features and at least two second hash features of each image-text pair in the training batch.

[0146] Specifically, in order to better maintain the pairwise semantic similarity between images and texts in image-text pairs and learn unified first and second hash features for both media, the multi-hash collaborative learning module jointly considers intra-media similarity preservation, inter-media similarity preservation, and unified hash code learning for text hash codes or image hash codes of different lengths. Intra-media similarity preservation means that semantically similar data within each media should be projected very close in the Hamming space, while semantically dissimilar data should be projected far apart. Inter-media similarity preservation means maintaining semantic similarity between media. The unified hash code learning is intended to learn consistent hash code representations across media.

[0147] In one implementation of this embodiment, constructing a hash loss term based on at least two first hash features, at least two second hash features, and a semantic similarity matrix specifically includes:

[0148] S61: For each image-text pair in the training batch, determine each first hash feature of the image-text pair. and the first hash feature of each image-text pair in the training set The inner product of the first hash feature, each second hash feature of the image-text pair and the second hash feature of each image-text pair in the training set Determine the inner product of the second hash feature of the image-text pair The second hash feature of each image-text pair in the training set The inner product of the third hash feature of the image-text pair and each second hash feature of the image-text pair and the first hash feature of each image-text pair in the training set The inner product of the fourth hash feature;

[0149] S62: Determine a media loss item based on the determined inner products of all first hash features, all second hash features, and the semantic similarity matrix;

[0150] S63: Determine an inter-media loss item based on the determined inner products of all third hash features, all fourth hash features, and the semantic similarity matrix;

[0151] S64, for each image-text pair in the training batch, according to each first hash feature and its corresponding second hash feature Determine hash signature b i,w , and determine all hash features b based on i,w , all first hash features and all second hash features Determine the hash quantization term;

[0152] S65. Determine a hash loss item according to the intra-media loss item, the inter-media loss item, and the hash quantization item.

[0153] Specifically, in step S61, the first hash feature inner product is the hash feature inner product in text media, and the second hash feature inner product is the hash feature inner product in image media.

[0154] In order to achieve similarity preservation within the text media, based on the text hash feature, the hash feature inner product within the text, that is, the first hash feature inner product, can be obtained. In other words, based on the i-th first hash feature of the image-text pair and the jth first hash feature of each image-text pair in the training set The first hash feature inner product can be obtained. In order to achieve similarity preservation within the image medium, the hash feature inner product within the image can be obtained according to the image hash feature, that is, the second hash feature inner product. In other words, according to the i-th second hash feature of the image-text pair and the jth second hash feature of each image-text pair in the training set A second hash feature inner product can be obtained. The first hash feature inner product and the second hash feature inner product can be expressed as:

[0155]

[0156]

[0157] in, represents the inner product of the first hash feature, represents the inner product of the second hash feature, represents the i-th first hash feature, represents the jth first hash feature, represents the i-th second hash feature, Represents the j-th second hash feature.

[0158] The third hash feature inner product and the fourth hash feature inner product are the feature inner products between the image and the text. In the embodiment of the present application, in order to maintain the similarity between the image and the text media, the third hash feature inner product can be calculated based on the i-th first hash feature of the image-text pair. and the jth second hash feature of each image-text pair in the training set Determine; the fourth hash feature inner product can be based on the i-th second hash feature of the image-text pair and the jth first hash feature of each image-text pair in the training set The third hash feature inner product and the fourth hash feature inner product can be expressed as:

[0159]

[0160]

[0161] Among them, Θ i,j,w represents the inner product of the third hash feature, Φ i,j,w represents the inner product of the fourth hash feature, represents the i-th first hash feature, The jth second hash feature, represents the i-th second hash feature, The jth first hash feature.

[0162] In step S62, the intra-media loss items include text media loss items and image media loss items, wherein the text media loss items are obtained based on all first hash feature inner products and a semantic similarity matrix, and the image media loss items are obtained based on all second hash feature inner products and a semantic similarity matrix.

[0163] In the embodiment of the present application, the loss item in the text media is obtained based on the inner product of all first hash features and the semantic similarity matrix, and the acquisition method is specifically as follows:

[0164] Determine an asymmetric pairwise negative log-likelihood loss within the textual media based on the inner products of all first hash features and a semantic similarity matrix;

[0165] A loss term within the textual medium is determined based on an asymmetric pairwise negative log-likelihood loss within the textual medium.

[0166] Specifically, the asymmetric pairwise negative log-likelihood loss within the textual medium can be expressed as:

[0167]

[0168] in, represents the asymmetric pairwise negative log-likelihood loss within the text medium, S ij represents the semantic similarity matrix, represents the inner product of the first hash feature.

[0169] Then, the text media loss term can be expressed as:

[0170]

[0171] in, represents the loss term within the textual media, represents the asymmetric pairwise negative log-likelihood loss within the text medium, M represents the number of training samples included in the training batch, and N represents the number of training samples in the training sample set.

[0172] In this embodiment, the image media loss term is obtained based on the inner product of all second hash features and the semantic similarity matrix, and the acquisition method is specifically as follows:

[0173] determining an asymmetric pairwise negative log-likelihood loss within the image medium based on the inner products of all second hash features and a semantic similarity matrix;

[0174] A loss term within the image medium is determined based on an asymmetric pairwise negative log-likelihood loss within the image medium.

[0175] Specifically, the asymmetric pairwise negative log-likelihood loss in the image medium can be expressed as:

[0176]

[0177] in, represents the asymmetric pairwise negative log-likelihood loss within the image medium, S ij represents the semantic similarity matrix, represents the inner product of the second hash feature.

[0178] Then, the image media loss term can be expressed as:

[0179]

[0180] in, represents the loss term within the image medium, represents the asymmetric pairwise negative log-likelihood loss within the image medium, M represents the number of training samples included in the training batch, and N represents the number of training samples in the training sample set.

[0181] In step S63, the inter-media loss term is determined based on the determined inner products of all third hash features, all fourth hash features, and the semantic similarity matrix, specifically:

[0182] Determine the asymmetric pairwise negative log-likelihood loss between text and image based on the inner product of all third hash features and the semantic similarity matrix;

[0183] Determine the asymmetric pairwise negative log-likelihood loss between image and text based on the inner product of all fourth hash features and the semantic similarity matrix;

[0184] The inter-media loss term is determined based on the asymmetric pairwise negative log-likelihood loss between text and image and the asymmetric pairwise negative log-likelihood loss between image and text.

[0185] Specifically, the asymmetric pairwise negative log-likelihood loss between text and image can be expressed as:

[0186]

[0187] in, represents the asymmetric pairwise negative log-likelihood loss between text and image, Θ i,j,w represents the inner product of the third hash feature, S ij Represents the semantic similarity matrix.

[0188] The asymmetric pairwise negative log-likelihood loss between image and text can be expressed as:

[0189]

[0190] in, represents the asymmetric pairwise negative log-likelihood loss between image and text, Φ i,j,w represents the fourth characteristic inner product, S ij Represents the semantic similarity matrix.

[0191] Then, the inter-media loss term can be expressed as:

[0192]

[0193] in, represents the inter-media loss term, represents the asymmetric pairwise negative log-likelihood loss between text and image, Represents the asymmetric pairwise negative log-likelihood loss between image and text, M represents the number of training samples included in the training batch, and N represents the number of training samples in the training sample set.

[0194] In step S64, in order to achieve unified hash code learning, for the i-th image-text pair in the training batch, according to each first hash feature and its corresponding second hash feature To determine the hash feature b of the wth unified hash code of the i-th image-text pair in the training batch i,w , wherein the hash feature b i,w It can be expressed as:

[0195]

[0196] Among them, b i,w represents the hash feature, denote the first and second hash features of the i-th image-text pair in the training batch, respectively, and sign denotes the element-wise sign function.

[0197] Then, according to determine all the hash features b i,w , all first hash features and all second hash features A hash quantization term may be determined, wherein the hash quantization term may be expressed as:

[0198]

[0199] in, represents the hash quantization term, b i,w represents the hash feature, denote the first and second hash features of the i-th image-text pair in the training batch, respectively, and K w represents the length of the wth hash code, and M represents the number of training samples included in the training batch.

[0200] In step S65, the hash loss term can be determined by summing the text media loss term, the image media loss term, the inter-media loss term, and the hash quantization term. The hash loss term can also be obtained by weighting the text media loss term, the image media loss term, the inter-media loss term, and the hash quantization term. The hash loss term can be expressed as:

[0201]

[0202] in, represents the hash loss term, represents the loss term within the textual media, represents the loss term within the image medium, represents the inter-media loss term, represents the hash quantization term, W represents the number of hash codes, and β and γ represent balancing hyperparameters.

[0203] S70. Based on the cross-modal contrast loss term and the hash loss term, update the parameters of the initial hash model to obtain a multi-bit hash code network model.

[0204] Specifically, the multi-bit hash code network model is used to hierarchically align global and local features between different media, effectively reduce the heterogeneity and semantic differences between different media, and simultaneously generate hash codes of different lengths for multiple different media to improve the efficiency and accuracy of cross-media retrieval.

[0205] In one implementation of this embodiment, before updating the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain a multi-bit hash code network model, the method further includes:

[0206] S71. For each image-text pair in the training batch, determine a text-assisted hash feature based on the text fusion feature, and determine an image-assisted hash feature based on the image fusion feature;

[0207] S72: learning a text hash code according to the text auxiliary hash feature, learning an image hash code according to the image auxiliary hash feature, and determining an auxiliary hash code according to the text hash code and the image hash code;

[0208] S73, mapping the at least two first hash features and the at least two second hash features to the Hamming space where the auxiliary hash features are located to obtain at least two first hash codes and at least two second hash codes;

[0209] S74. Determine an auxiliary hash loss term based on the text auxiliary hash features and the image auxiliary hash features of the image-text pairs in the training batch, and determine a reconstruction loss term based on the auxiliary hash code, the at least two first hash codes, and the at least two second hash codes;

[0210] S75. Determine a modified loss term according to the auxiliary hash loss term and the reconstruction loss term, and modify the hash loss term based on the modified loss term.

[0211] Specifically, to further enhance the discriminability of the hash code, the multi-bit hash code network model may further include an auxiliary hash collaborative learning module, wherein the auxiliary hash collaborative learning module includes an auxiliary hash linear projection layer (AuxHash), which is used to obtain text auxiliary hash features and image auxiliary hash features based on the text fusion features and the image fusion features. The auxiliary hash linear projection layer also includes an online hash learner, which is used to learn the text hash code and the image hash code respectively based on the text auxiliary hash features and the image auxiliary hash features to determine the auxiliary hash code.

[0212] In an embodiment of the present application, the auxiliary hash collaborative learning module includes two auxiliary hash linear projection layers, one of which is used to determine a text hash code based on text fusion features, and the other is used to determine an image hash code based on image fusion features.

[0213] In step S71, the text auxiliary hash feature can be obtained by mapping the text fusion feature through one of the auxiliary hash linear projection layers; the image auxiliary hash feature can be obtained by mapping the image fusion feature through another auxiliary hash linear projection layer. The text auxiliary hash feature and the image auxiliary hash feature can be expressed as:

[0214]

[0215]

[0216] in, represents text-assisted hash features, represents the image-assisted hash feature, represents the text fusion feature, represents image fusion features, AuxHash represents hash function, K aux Indicates the length of the hash code, represents the dimension of auxiliary hash features, θ aux Represents a trainable parameter.

[0217] In step S72, the text auxiliary hash feature The online hash learner input to one of the auxiliary hash linear projection layers can learn a length of K aux The text hash code of the image auxiliary hash feature The online hash learner input to the auxiliary hash linear projection layer of the other path can learn the length K aux The image hash code. According to the text hash code and the image hash code, an auxiliary hash code can be determined, wherein the auxiliary hash code can be expressed as:

[0218]

[0219] Among them, b i,aux represents the auxiliary hash code, represents text-assisted hash features, represents image-assisted hash features, and sign represents the element-wise sign function.

[0220] In step S73, the first hash code is a text hash code. The second hash code is an image hash code. In this embodiment of the present application, the auxiliary hash collaborative learning module further includes an auxiliary Hamming space, which is based on the auxiliary hash feature and is used to obtain at least two text hash codes and at least two image hash codes, that is, to obtain at least two first hash codes and at least two second hash codes.

[0221] Specifically, by using W text hash decoders to decode at least two first hash features Decoding is performed and mapped to the auxiliary Hamming space after decoding to obtain at least two first hash codes; at least two second hash features are decoded by using W image hash decoders. Decoding is performed and mapped to the auxiliary Hamming space after decoding to obtain at least two second hash codes. The first hash code and the second hash code can be expressed as:

[0222]

[0223]

[0224] in, represents the first hash code, represents the second hash code, represents the first hash feature, Represents the second hash feature, Decoder represents the decoding function, θdec ,w is the trainable weight parameter, K aux Indicates the length of the hash code, Represents the dimension of the hash code.

[0225] In step S74, the specific step of determining the auxiliary hash loss term based on the text auxiliary hash features and the image auxiliary hash features of the image-text pairs in the training batch is the same as the specific step of constructing the hash loss term based on the at least two first hash features, the at least two second hash features, and the semantic similarity matrix. The hash loss term can be determined by summing the auxiliary text intra-media loss term, the auxiliary image intra-media loss term, the auxiliary inter-media loss term, and the auxiliary hash quantization term, wherein the specific step of constructing the auxiliary text intra-media loss term is the same as the specific step of constructing the text intra-media loss term, the specific step of constructing the auxiliary image intra-media loss term is the same as the specific step of constructing the image intra-media loss term, the specific step of constructing the auxiliary inter-media loss term is the same as the specific step of constructing the inter-media loss term, and the specific step of constructing the auxiliary hash loss term is the same as the specific step of constructing the hash loss term.

[0226] Then, the auxiliary hash loss term can be expressed as:

[0227]

[0228] in, represents the auxiliary hash loss term, represents the lossy item in the auxiliary text media, represents the loss term within the auxiliary image medium, represents the auxiliary media loss term, represents the auxiliary hash quantization term, β and γ represent the balance hyperparameters;

[0229] The reconstruction loss term includes a text reconstruction loss term and an image reconstruction loss term, wherein the text reconstruction loss term is a text-based reconstruction loss, and the image loss term is an image-based reconstruction loss. The text reconstruction loss term and the image reconstruction loss term are used to determine the correction loss term. Specifically, the text reconstruction loss term can be determined based on the auxiliary hash code and at least two first hash codes, and the image reconstruction loss term is determined based on the auxiliary hash code and at least two second hash codes. The text reconstruction loss term and the image reconstruction loss term can be expressed as follows:

[0230]

[0231]

[0232] in, represents the text reconstruction loss term, represents the image reconstruction loss term, b i,aux represents the auxiliary hash code, represents the first hash code, represents the second hash code, M represents the number of training samples included in the training batch, and W represents the number of hash codes.

[0233] In step S75, the modified loss term is used to modify the hash loss term. The modified loss term can be a combination of the auxiliary hash loss term and the reconstruction loss term. In the embodiment of the present application, the modified loss term can be determined by summing the auxiliary hash loss term and the reconstruction loss term, or by weighting the auxiliary hash loss term and the reconstruction loss term. The modified loss term can be expressed as:

[0234]

[0235] in, represents the corrected loss term, represents the auxiliary hash loss term, represents the text reconstruction loss term, represents the image reconstruction loss term, and δ represents the balancing hyperparameter.

[0236] In this embodiment, based on the correction loss term, the hash loss term, and the cross-modal contrast loss term, a loss function of the multi-bit hash code network model can be constructed. The loss function of the multi-bit hash code network model can be determined by summing the correction loss term, the hash loss term, and the cross-modal contrast loss term, or by weighting the correction loss term, the hash loss term, and the cross-modal contrast loss term. The loss function of the multi-bit hash code network model can be expressed as:

[0237]

[0238] in, represents the loss function of the multi-bit hash code network model, represents the corrected loss term, represents the cross-modal contrast loss term, represents the hash loss term, and λ and μ represent balancing hyperparameters.

[0239] This embodiment obtains a corrected loss term through an auxiliary hash collaborative learning module to correct the hash loss term, which can further improve the quality of generating multiple hash codes of different lengths based on a multi-bit hash code network model, and generate multiple high-quality hash codes of different lengths for different media to improve the accuracy of cross-media retrieval.

[0240] In summary, this embodiment provides a training method for a multi-bit hash code network model, which specifically includes: obtaining a training sample set, and constructing a semantic similarity matrix using the semantic labels of each training sample in the training sample set, wherein the training sample set includes a plurality of training batches, and each training batch includes a plurality of image-text pairs; inputting each image-text pair in the training batch into an initial hash model, and determining the text global features, text local feature sequences, image global features, and image local feature sequences of the training sample through the initial hash model; aligning the text global features and the image global features to obtain aligned text global features and aligned image global features, and embedding the text semantic features of the text local feature sequences and the image semantic features of the image local feature sequences according to the shared concept. The invention relates to a method for extracting the global features of a plurality of media information from a plurality of media, and constructing a cross-modal contrast loss term based on the aligned text global features, the aligned image global features, the text semantic features and the image semantic features; determining a text fusion feature based on the aligned text global features and the text semantic features, and determining an image fusion feature based on the aligned image global features and the image semantic features; determining at least two first hash features based on the text fusion feature, and determining at least two second hash features based on the image fusion feature; constructing a hash loss term based on the semantic similarity matrix and at least two first hash features and at least two second hash features of each image-text pair in the training batch; updating the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain a multi-bit hash code network model. The present invention obtains the global features and local semantic tokens of multiple media information by constructing a multi-bit hash code network model, and effectively reduces the heterogeneity and semantic gap between different media by hierarchically aligning the global features and local semantic tokens of multiple media information respectively. At the same time, multiple high-quality hash codes of different lengths are generated simultaneously through the aligned global features and local semantic tokens of each media information, thereby improving the efficiency and accuracy of cross-media retrieval.

[0241] This embodiment further provides a cross-media retrieval method based on a multi-bit hash code, wherein the cross-media retrieval method based on a multi-bit hash code specifically includes:

[0242] Obtaining a query sample, and determining at least a query hash code sequence of the query sample using the multi-bit hash code network model, wherein the query hash code sequence includes at least two hash codes;

[0243] Obtaining a target data set corresponding to the query sample, and determining a hash code sequence set corresponding to the target data set using the multi-bit hash code network model;

[0244] According to the query hash code sequence and the hash code sequence set, target data corresponding to the query sample is determined in the target data set.

[0245] Specifically, the query sample is an image or text, and the image or text is input into the multi-bit hash code network model to obtain multiple image hash code sequences or text hash code sequences of different lengths. The target data set contains a plurality of target data, wherein the target data is an image-text pair, and the target data in the target data is input into the multi-bit hash code network model to obtain a plurality of target image hash code sequences or target text hash code sequences of different lengths to form a hash code sequence set. The image hash code sequence or text hash code sequence is matched with the hash code sequence set to obtain target data that is semantically similar to the query sample. For example, if the query sample is text, it is necessary to query images that are semantically related to it, and the obtained text hash code sequence is matched with the hash code sequence set to obtain images that are semantically similar to the text.

[0246] Based on the above-mentioned training method of the multi-bit hash code network model, this embodiment provides a training device for the multi-bit hash code network model, such as Figure 3 As shown, the device specifically includes:

[0247] An acquisition module 100 is configured to acquire a training sample set and construct a semantic similarity matrix using the semantic labels of each training sample in the training sample set, wherein the training sample set includes a plurality of training batches, and each training batch includes a plurality of image-text pairs;

[0248] An initial hash model 200 is configured to input each image-text pair in a training batch into the initial hash model, determine the text global features, text local feature sequences, image global features, and image local feature sequences of the training sample through the initial hash model; align the text global features and the image global features to obtain aligned text global features and aligned image global features, learn text semantic features of the text local feature sequences and image semantic features of the image local feature sequences based on shared concept embedding, and construct a cross-modal contrast loss term based on the aligned text global features, aligned image global features, text semantic features, and image semantic features; determine a text fusion feature based on the aligned text global features and the text semantic features, and determine an image fusion feature based on the aligned image global features and the image semantic features; determine at least two first hash features based on the text fusion features, and determine at least two second hash features based on the image fusion features; and construct a hash loss term based on a semantic similarity matrix and the at least two first hash features and the at least two second hash features of each image-text pair in the training batch;

[0249] The parameter updating module 300 is configured to update the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain a multi-bit hash code network model.

[0250] Based on the above-mentioned training method of the multi-bit hash code network model, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the training method of the multi-bit hash code network model as described in the above embodiment.

[0251] Based on the training method of the multi-bit hash code network model, the present application also provides a terminal device, such as Figure 4 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0252] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0253] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0254] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0255] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a multi-bit hash code network model, characterized in that: The training method of the multi-bit hash code network model specifically includes: Obtaining a training sample set, and constructing a semantic similarity matrix using the semantic label of each training sample in the training sample set, wherein the training sample set includes a plurality of training batches, and each training batch includes a plurality of image-text pairs; Each image-text pair in the training batch is input into the initial hash model, and the initial hash model is used to determine the text global features, text local feature sequence, image global features, and image local feature sequence of the training sample; Aligning the text global features and the image global features to obtain aligned text global features and aligned image global features, learning text semantic features of the text local feature sequence and image semantic features of the image local feature sequence based on shared concept embedding, and constructing a cross-modal contrast loss term based on the aligned text global features, the aligned image global features, the text semantic features, and the image semantic features; Determining a text fusion feature based on the aligned text global feature and the text semantic feature, and determining an image fusion feature based on the aligned image global feature and the image semantic feature; Determining at least two first hash features based on the text fusion features, and determining at least two second hash features based on the image fusion features; constructing a hash loss term based on the semantic similarity matrix and at least two first hash features and at least two second hash features of each image-text pair in the training batch; Based on the cross-modal contrast loss term and the hash loss term, the parameters of the initial hash model are updated to obtain a multi-bit hash code network model.

2. The training method of the multi-bit hash code network model according to claim 1, characterized in that The multi-bit hash code network model includes a feature extraction module, a cross-media comparative alignment module and a multi-hash collaborative learning module, wherein the feature extraction module is connected to the cross-media comparative alignment module, and the cross-media comparative alignment module is connected to the multi-hash collaborative learning module, wherein the feature extraction module is used to extract text global features, text local feature sequences, image global features and image local feature sequences; the cross-media comparative alignment module is used to determine text fusion features and image fusion features based on the text global features, the image global features, the text local feature sequences and the image local feature sequences; the multi-hash collaborative learning module is used to determine at least two first hash features based on the text fusion features, and determine at least two first hash codes based on the at least two first hash features, and determine at least two second hash features based on the image fusion features, and determine at least two second hash codes based on the at least two second hash features.

3. The training method of the multi-bit hash code network model according to claim 2, characterized in that: The cross-media comparative alignment module includes a residual MLP unit, a local learning unit, a pooling layer, and an adder. The residual MLP unit and the adder are connected, and the local learning unit is connected to the adder through the pooling layer. The local learning unit includes a cross-attention layer and a Transformer layer connected in sequence, and the query vector of the cross-attention layer is a shared concept embedding.

4. The training method of the multi-bit hash code network model according to claim 1, characterized in that The cross-modal contrast loss term constructed based on aligned text global features, aligned image global features, text semantic features and image semantic features is specifically: For each image-text pair in the training batch, determine the aligned text global features of the image-text pair and the global features of each aligned image in the training batch The first similarity of the image-text pair; the global features of the aligned images and the global features of each aligned text in the training batch The second similarity; Determining a global contrast loss term based on all the determined first similarities and all the determined second similarities; Determine each text semantic feature in the text semantic feature sequence for each image-text pair Each image semantic feature in the image semantic feature sequence of the image-text pair The third similarity, and each image semantic feature in the image semantic feature sequence of the image-text pair Each text semantic feature in the text semantic feature sequence of the image-text pair The fourth similarity; Determining a local contrast loss term based on all the determined third similarities and all the determined fourth similarities; According to the global contrast loss term and the local contrast loss term, a cross-modal contrast loss term is determined.

5. The training method of the multi-bit hash code network model according to claim 1, characterized in that: The constructing of the hash loss term according to the at least two first hash features, the at least two second hash features, and the semantic similarity matrix specifically includes: For each image-text pair in the training batch, determine each first hash feature of the image-text pair and the first hash feature of each image-text pair in the training set The inner product of the first hash feature, each second hash feature of the image-text pair and the second hash feature of each image-text pair in the training set Determine the inner product of the second hash feature of the image-text pair The second hash feature of each image-text pair in the training set The inner product of the third hash feature of the image-text pair and each second hash feature of the image-text pair and the first hash feature of each image-text pair in the training set The inner product of the fourth hash feature; Determining a media loss item based on the determined inner products of all first hash features, the inner products of all second hash features, and a semantic similarity matrix; determining an inter-media loss term based on the determined inner products of all third hash features, the inner products of all fourth hash features, and a semantic similarity matrix; For each image-text pair in the training batch, according to each first hash feature and its corresponding second hash feature Determine hash characteristics , and determine all hash features based on , all first hash features and all second hash features Determine the hash quantization term; A hash loss term is determined according to the intra-media loss term, the inter-media loss term, and the hash quantization term.

6. The training method of the multi-bit hash code network model according to claim 1, characterized in that Before updating the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain a multi-bit hash code network model, the method further includes: For each image-text pair in the training batch, determining a text-assisted hash feature based on the text fusion feature, and determining an image-assisted hash feature based on the image fusion feature; Learning a text hash code according to the text auxiliary hash feature, learning an image hash code according to the image auxiliary hash feature, and determining an auxiliary hash code according to the text hash code and the image hash code; Mapping the at least two first hash features and the at least two second hash features to the Hamming space where the text auxiliary hash feature is located to obtain at least two first hash codes and at least two second hash codes; determining an auxiliary hash loss term based on the text auxiliary hash features and the image auxiliary hash features of the image-text pairs in the training batch, and determining a reconstruction loss term based on the auxiliary hash code, the at least two first hash codes, and the at least two second hash codes; A modified loss term is determined according to the auxiliary hash loss term and the reconstruction loss term, and the hash loss term is modified based on the modified loss term.

7. A cross-media retrieval method based on multi-bit hash codes, characterized in that: Using the multi-bit hash code network model trained by the training method of the multi-bit hash code network model according to any one of claims 1 to 6, the cross-media retrieval method based on the multi-bit hash code specifically includes: Obtaining a query sample, and determining at least a query hash code sequence of the query sample using a multi-bit hash code network model, wherein the query hash code sequence includes at least two hash codes; Obtaining a target data set corresponding to the query sample, and determining a hash code sequence set corresponding to the target data set using the multi-bit hash code network model; According to the query hash code sequence and the hash code sequence set, target data corresponding to the query sample is determined in the target data set.

8. A training device for a multi-bit hash code network model, characterized in that: The training device of the multi-bit hash code network model specifically includes: an acquisition module, configured to acquire a training sample set and construct a semantic similarity matrix using the semantic labels of each training sample in the training sample set, wherein the training sample set includes a plurality of training batches, and each training batch includes a plurality of image-text pairs; An initial hash model is used to input each image-text pair in a training batch into the initial hash model, determine the text global features, text local feature sequences, image global features, and image local feature sequences of the training samples through the initial hash model; align the text global features and the image global features to obtain aligned text global features and aligned image global features, learn text semantic features of the text local feature sequences and image semantic features of the image local feature sequences based on shared concept embedding, and construct a cross-modal contrast loss term based on the aligned text global features, aligned image global features, text semantic features, and image semantic features; determine a text fusion feature based on the aligned text global features and the text semantic features, and determine an image fusion feature based on the aligned image global features and the image semantic features; determine at least two first hash features based on the text fusion features, and determine at least two second hash features based on the image fusion features; and construct a hash loss term based on a semantic similarity matrix and the at least two first hash features and at least two second hash features of each image-text pair in the training batch; A parameter updating module is used to update the parameters of the initial hash model based on the cross-modal contrast loss term and the hash loss term to obtain a multi-bit hash code network model.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the multi-bit hash code network model as described in any one of claims 1 to 6, and / or to implement the steps in the cross-media retrieval method based on the multi-bit hash code as described in claim 7.

10. A terminal device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps of the training method of the multi-bit hash code network model according to any one of claims 1 to 6, and / or implements the steps of the cross-media retrieval method based on the multi-bit hash code according to claim 7.

Images