A cross-modal retrieval method and retrieval system

Abstract

The application provides a cross-modal retrieval method and a retrieval system, the retrieval method comprising: using a CLIP pre-training model to encode features to obtain original modal features comprising original images and texts; performing attention alignment processing on the original modal features to obtain modal alignment data to realize semantic correlation between original modes; passing the modal data formed in the above step through a weight-shared multi-layer perception machine to maintain the invariance of the modes; and distributing the finally obtained feature data to a normalized hypersphere using an Arc4cmr loss function to perform category boundary constraint. The cross-modal retrieval method of the application makes the common representation of pairs of images and texts as close as possible, and realizes simultaneous enhancement of intra-class tightness and inter-class difference.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of semantic maximum correlation and modal alignment cross-modal retrieval, and in particular to a cross-modal retrieval method and a retrieval system. BACKGROUND

[0002] Information resources have shown a mixed situation of multi-modal data (text, image, audio, video, etc.), which are cross-linked and gradually deeply integrated, and these multi-media data show a rapid growth trend. How to mine the hidden semantic correlation between cross-modal data and realize cross-modal information retrieval is an important prerequisite for fully utilizing multi-modal data resources.

[0003] With the continuous increase of data size and model size, deep learning has gradually entered the era of pre-training models, and how to better apply it to downstream tasks has attracted more and more attention, such as CLIP, SimVLM, etc. The existing text-image inference ability of such pre-training models has relatively good migration ability for different downstream tasks such as image description (Image Captioning), visual question answering (Visual Question Answering, VQA), cross-modal retrieval (Cross-Modal Retrieval), etc. Compared with traditional image classification methods, CLIP model no longer allocates a noun label for each image, but a sentence, so the images that have been forced to be classified into the same category have "infinite fine-grained" labels. Although the pre-training model CLIP has obtained rich text-image semantics through 40 billion pairs of image-text pairs through unsupervised contrastive learning method, the early encoding stage of CLIP for two modal data is still independent of each other, and still lacks the interaction of inter-modal information. CLIP uses a contrastive loss to constrain the judgment of matching or not matching of two modalities, and each image (text) modality information has and only one text (image) modality information matched with it, ignoring the rich semantic information and discrimination information contained in the one-to-many approximate matching situation.

[0004] In view of this, the present application is proposed. SUMMARY

[0005] In view of this, the present application discloses a novel cross-modal retrieval method, which first re-represents the feature representation of one modality with another modality through the DecomposableAttention mechanism to obtain more rich semantic information and enhance the semantic correlation of the two modalities, and then distributes the learned multi-modal features to the normalized hypersphere in the label space by using the Arc4cmr loss function, increases the angle edge penalty between the features and the weights to make the class boundary clear, and realizes the simultaneous enhancement of intra-class tightness and inter-class difference.

[0006] Specifically, the present application is realized by the following technical solutions:

[0007] In a first aspect, the present application discloses a novel cross-modal retrieval method, comprising the following steps:

[0008] The CLIP pre-training model is used to encode the features of image and text samples to obtain original modal features including original images and texts.

[0009] The original modal features are subjected to attention alignment processing to obtain modal alignment data to realize semantic correlation between original modalities.

[0010] The modal alignment data formed by the above steps is subjected to weight-shared multi-layer perception to maintain the invariance of modalities.

[0011] The obtained modal data is distributed onto a normalized hypersphere by using an Arc4cmr loss function to perform category boundary constraint.

[0012] In a second aspect, the present application discloses a cross-modal retrieval system, comprising:

[0013] An initial module is used to encode the features of image and text samples by using a CLIP pre-training model to obtain original modal features including original images and texts.

[0014] An alignment module is used to subject the original modal features to attention alignment processing to obtain modal alignment data to realize semantic correlation between original modalities.

[0015] A weight sharing module is used to subject the modal alignment data formed by the above steps to weight-shared multi-layer perception to maintain the invariance of modalities.

[0016] A normalization module is used to distribute the obtained final modal data onto a normalized hypersphere by using an Arc4cmr loss function to perform category boundary constraint.

[0017] In a third aspect, the present application discloses a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to realize the steps of the cross-modal retrieval method according to the first aspect.

[0018] In a fourth aspect, the present application discloses a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the steps of the cross-modal retrieval method according to the first aspect.

[0019] Cross-modal retrieval in the prior art solves the problem of heterogeneity difference by measuring the similarity of different modal data with bottom layer feature heterogeneity and high layer semantic correlation, and can be generally divided into two types: unsupervised retrieval and supervised retrieval.

[0020] Unsupervised cross-modal retrieval: Canonical Correlation Analysis (CCA) is essentially a multivariate statistical analysis that uses the correlation between multiple image and text matching pairs to obtain an unsupervised common subspace with maximum pairwise similarity, and maps image features and text features into the common subspace to obtain unified representation of different modal data, reflecting the overall correlation between the two modalities, thereby realizing cross-modal retrieval. Kernel CCA (KCCA) introduces the kernel trick to improve CCA when there is a nonlinear correlation between two variables. Corr-AE (Correspondence Autoencoders) uses autoencoders to consider reconstruction error and correlation loss in cross-modal retrieval.

[0021] Supervised cross-modal retrieval: JRL (Joint Representation Learning) integrates sparse and semi-supervised regularization of different media types in a unified framework to jointly explore pairwise correlation and semantic information. ACMR (adversarial cross-modal retrieval) attempts to distinguish different modalities by classification through the idea of adversarial learning. CCL (cross-modal correlation learning) mines coarse-grained and fine-grained information of different media types through multi-task learning. DSCMR (Deep Supervised Cross-Modal Retrieval) preserves the semantic discriminability in the public representation space by linear classification of samples, and maintains the invariance of the modal in the public representation space through weight sharing strategy. CLIP4CMR (CLIP for Supervised Cross-Modal Retrieval) adds class-level correlation information to the pre-trained model CLIP, which generates original feature representation of each modality using CLIP as the backbone network, and then sends it to the multi-layer perceptron of each modality to learn the public representation space. To solve the problem of lack of robustness of unknown categories, a set of unified prototypes are assigned as class proxies, and the Nearest-Prototype classification rule is used for reasoning to solve the problem of lack of robustness of unknown categories, and class-level correlation information is added to the pre-trained model CLIP.

[0022] However, the traditional approach to cross-modal retrieval in existing technologies involves embedding text and images into a joint latent space using a dual-tower structure model, and then applying cosine similarity as an equal distance metric to make the model achieve higher similarity between matched text and images. However, the relatively large representational differences between the two modalities make it difficult to directly compare the two modalities themselves.

[0023] To address the aforementioned technical problems, this invention provides a cross-modal retrieval method. First, features are encoded using a pre-trained CLIP model to obtain original image and text representations. To further enhance modal information interaction, the original modal representations are then fed into an attention alignment module. Specifically, for each query in the image (text) modality (within a batch), the system focuses more on the text (image) sample matching the query within a batch-sized text (image) modality database, achieving mutual alignment of individual samples. This simultaneously enhances the semantic association between the two modalities. Finally, a shared multilayer perceptron is used to share weight parameters to process the data processed above. This generates a common representation space for each modality while adding semantic constraints, making the common representations of paired images and text as similar as possible, thus simultaneously enhancing intra-class compactness and inter-class diversity. Attached Figure Description

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0025] Figure 1 This is an overall framework diagram of the cross-modal retrieval method provided in the embodiments of the present invention;

[0026] Figure 2 This is a schematic diagram illustrating the operation of the modal alignment method provided in an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the angle space of Arc4cmr loss provided in an embodiment of the present invention;

[0028] Figure 4 A flowchart illustrating a computer device provided in an embodiment of the present invention;

[0029] Figure 5 The results of the visualization experiment provided in the embodiments of the present invention are shown in the figure. Detailed Implementation

[0030] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to any embodiment of the present disclosure, unless specified otherwise. Accordingly, when the description of the exemplary embodiments is applied to any embodiment of the present disclosure, it should be understood that all the features described and / or illustrated herein are not necessarily essential to the practice of the present disclosure. Furthermore, the terms "a" and "an" are used herein to include one or more of the relevant singular inventive entity or elements and the singular represent "one" unless otherwise specified. The terms "comprising," "including," containing" and "having" are used herein in the inclusive sense of "including" but not "consisting of," unless otherwise specifically noted. The term "consisting essentially of" is used herein to include the elements or steps specified and any additional elements or steps that do not materially affect the desired results.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] It is to be understood that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is to be understood that where the disclosure indicates that certain features are present "in some embodiments," such features can be present in other embodiments.

[0033] A cross-modal retrieval method is disclosed, as shown in Figure 1 comprising the following steps:

[0034] A CLIP pre-training model is used to encode the features of image and text samples to obtain original modal features including original images and texts;

[0035] The original modal features are subjected to attention alignment processing to obtain modal alignment data to realize semantic correlation between the original modal features;

[0036] The modal alignment data formed by the above steps are subjected to weight-shared multi-layer perception to maintain the invariance of the modal;

[0037] An Arc4cmr loss function is used to distribute the obtained final modal data onto a normalized hypersphere to perform class boundary constraint.

[0038] In the scheme of the present application, it is proposed to place the modal alignment on the encoding of the backbone network to increase the matching degree of the same cross-modal data and the separation degree of different modal data. For each sample of modal 1 (image or text) and all samples in the batch of modal 2 (text or image), a decomposable attention adjustment is performed. On the basis of obtaining the original feature representation through the CLIP encoder, the feature representation of one modal is represented by the feature representation of the other modal through the Decomposable Attention mechanism to enhance the semantic association of the two modalities. In the process of modal alignment, the single query of the modal obtains multiple approximate matching information of the other modal, so as to obtain more rich semantic information. In order to prevent the proportion of irrelevant modal in the new feature representation method from being too large to cause information loss, the output feature after modal alignment is added to the original feature, and then subjected to Layer Normalization to ensure the stability of the data feature distribution in the optimization process, so as to accelerate the convergence of the model, that is, the final image representation is The text representation is The modal alignment module adds and normalizes the original image (text) feature representation and the image (text) feature represented by the text (image), promotes the interaction of the two modal information, increases the same class aggregation degree and different class separation degree of the cross-modal data, and enhances the accuracy of the two search modes, so that the accuracy of the two retrieval results can be balanced.

[0039] Specifically, as shown in Figure 2 , in fact, the original features of the images in the batch (left striped grid, with color as the interval to distinguish multiple images in the batch) are taken as the query Q, and the similarity of each image in the batch and all text original features is calculated (one-to-many relationship, the attention weight is obtained by multiplying one Q and multiple K, that is, the blue long bars of different lengths between K and V in the figure, the longer the longer the more similar), and then the attention weight is multiplied by the specific feature value V of the text original feature to obtain the new image feature represented by the text feature, that is, the aligned text feature representation (right striped grid). In order to prevent the proportion of irrelevant semantic original text features in the new aligned text feature representation from being too large to cause information loss of the image represented, the original image feature is added to the aligned text feature and subjected to Layer Normalization.

[0040] The above process refers to the specific decomposition of attention adjustment process when modality 1 is the original image and modality 2 is the original text, that is, the operation of the image (modality 1) retrieving text (modality 2) process. Here, it involves the mutual retrieval between the two modalities. Similarly, the text (modality 1) retrieves the image (modality 2) is to change the specific input of QKV. The core is actually a matrix operation, so the alignment principle is the same, and the attention weight matrix obtained by the image retrieving text process can be used for text retrieving image by transposing.

[0041] In addition, the cross-modal retrieval task requires to increase the similarity and aggregation within the class and the difference and inconsistency between the classes as much as possible. In order to increase the intra-class compactness and inter-class separability while classifying, eliminate the boundary ambiguity problem, additive angular margin loss (ArcFace) is applied to the cross-modal retrieval field, and it is named as Arc4cmr loss. The specific process is: directly in the angle space, between the nearest classes, enforce to maximize the classification boundary, and the feature x i is multiplied by the corresponding weight W yi , and the L2 regularization is performed, so that ||W yi ||=1, the normalized feature is multiplied by a rescale parameter s, so that ||x i ||=s, that is, the embedding feature is distributed on a hypersphere with radius s; on the other hand, a custom additive angular margin m is added between the feature x i and the target weight W yi , which replaces the original cosθ yi with cos(θ yi +m), and the rest remains unchanged. In fact, each weight w here provides a class center, which is changed to θ yi +m by the additional angular interval, so that the original corresponding output is smaller and the angle of the space is larger, thereby increasing the training difficulty and more gathering to the class center. The normalization step of the feature and the weight value makes the prediction only depend on the angle between the feature and the weight value; finally, an angular margin penalty m is added between x i and W yi , which enhances the intra-class compactness and inter-class difference. The specific formula expressions are formula 1 and 2, which represent the constraint conditions.

[0042]

[0043]

[0044] In the above formula, the batch size is N, that is, i=1, 2,..., N, x i is the feature input, and its class label is y i , characteristic x i corresponding weight W yi , m is an angle edge penalty, n is the number of categories, that is, k = 1, 2,..., n, W k is the weight of each category, and θ k is the cosine angle of input characteristic x i misjudged as other k categories of non-y i class (corresponding k class weight W k ). For different retrieval requirements, only the input changes in formulas 1 and 2. For the loss function L I of image retrieval text I2T, the input is The corresponding regularization is That is, For the loss function L T of text retrieval image T2I, the input becomes The corresponding regularization is That is, In summary, the objective function of the proposed SMR-MA model is L Arc4cmr = L I + L T . The angle space diagram of the Arc4cmr loss is shown in Figure 3 , where different colors represent different categories, circles represent image modalities, and triangles represent text modalities.

[0045] In addition, the present application not only provides a cross-modal retrieval method, but also provides a cross-modal retrieval system, which specifically comprises:

[0046] An initial module: used to encode the features of image and text samples by using a CLIP pre-training model to obtain original modal features including original images and texts;

[0047] An alignment module: used to perform attention alignment processing on the original modal features to obtain modal alignment data to realize semantic correlation between the original modalities;

[0048] A weight sharing module: used to pass the modal alignment data formed in the above steps through a weight-shared multi-layer perception machine to maintain the invariance of the modalities;

[0049] A normalization module: used to distribute the obtained final modal data to a normalized hypersphere for category boundary constraint by using an Arc4cmr loss function.

[0050] In specific implementation, the above various modules can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation of the above various units can be referred to the method embodiments above, which will not be repeated here.

[0051] Experimental Example 1

[0052] The cross-modal retrieval method implemented by the embodiment of the present application is compared with eight representative baseline methods in the prior art in overall performance, including four traditional methods, CCA, KCCA, Corr-AE and JRL, and four deep learning-based methods, ACMR, CCL, DSCMR and CLIP4CMR. The mAP (mean Average Precision) of the cross-modal retrieval standard is used as an evaluation index to compare and verify the mAP values of I2T (image-to-text) and T2I (text-to-image).

[0053] Table 1 Comparison of mAP values of SMR-MA and baseline methods on benchmark datasets

[0054]

[0055] The comprehensive analysis experiment on the three benchmark datasets shows that the method of the present application has good performance in the cross-modal retrieval task. Compared with the baseline method that achieves the optimal result on Wikipedia, PascalSentence and NUS-WIDE at present, the proposed SMR-MA improves the mAP by 9.4%, 0.7%, 3.4% and 8.7% respectively, achieving the effect of SOTA (state-of-the-art), and thus has higher application value.

[0056] In order to intuitively observe the effectiveness of the maximum semantic correlation and modal alignment model (SMR-MA), whether the representations of high-dimensional image and text samples in the shared representation space have good separability, the original 1024-dimensional high-dimensional data is projected into a 2-dimensional space for visualization by using a nonlinear dimension reduction algorithm, t-SNE (t-distributed Stochastic Neighbor Embedding). The Wikipedia dataset is selected for the visualization experiment. Figure 5 (d) and Figure 5 (e) respectively represent the original feature distributions of images and texts obtained by CLIP visual encoders and text encoders. As can be seen from the two figures, the class separation degree and the aggregation degree within the class are both low, resulting in low matching accuracy. Figure 5 (a) and Figure 5 (b) respectively show the distribution of image and text representations after SMR-MA. Both of them can effectively distinguish samples of different semantic categories and divide them into corresponding semantic discrimination clusters. Figure 5 (c) shows the overlap degree of the feature embedding distributions of the two modalities in the public representation space, which indicates that the method has obvious effect on eliminating the modal difference.

[0057] Figure 4 Fig. 1 shows a schematic diagram of a computer device according to an embodiment of the present application. The computer device 100 comprises at least a memory 102 and a processor 101. The memory 102 is connected with the processor via a communication bus 103, and is configured to store computer instructions executable by the processor 101. The processor 101 is configured to read the computer instructions from the memory 102 to implement the steps of the method of cross-modal retrieval according to any of the embodiments described above. Figure 4 Fig. 4 shows a schematic diagram of a computer device according to an embodiment of the present application. The computer device 400 comprises at least a memory 402 and a processor 401. The memory 402 is connected with the processor via a communication bus 403, and is configured to store computer instructions executable by the processor 401. The processor 401 is configured to read the computer instructions from the memory 402 to implement the steps of the method of cross-modal retrieval according to any of the embodiments described above.

[0058] For the above-mentioned apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the part of the method embodiments. The apparatus embodiments described above are merely schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present disclosure according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0059] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0060] Finally, it should be noted that while the present disclosure describes various features with respect to specific embodiments, these features can be combined, sub-combined, or removed from the specific embodiments where such features are described. In addition, while features can function in certain combinations as described above, and even initially claimed as such, one or more features from a claimed combination can in some cases be removed from the combination and the claimed combination can be directed to a sub-combination or variation of a sub-combination.

[0061] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order nor limiting it to only that order. One will appreciate that many other operations could be performed or the described operations could be performed in a different order. Furthermore, some operations can be performed simultaneously. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0062] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures need not be performed in the particular order described or in sequential order at all. In certain implementations, multitasking and parallel processing can be advantageous.

[0063] The above-described embodiments are merely possible implementations of the present disclosure, and do not limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the present disclosure.

Claims

1. A cross-modal retrieval method, characterized in that, The method comprises the following steps: The features are encoded using a CLIP pre-training model to obtain original modality features including original images and texts; The original modality features are subjected to attention alignment processing to obtain modality alignment data to realize semantic correlation between the original modalities; The modality alignment data formed by the above steps are subjected to weight-shared multi-layer perception to maintain the invariance of the modalities; The final modality data is distributed onto a normalized hypersphere by using an Arc4cmr loss function to perform category boundary constraint; The attention alignment processing method comprises: Each sample of the original image (text) in modality 1 is decomposed and adjusted in attention by using all samples of the text (image) in modality 2 within a batch, i.e., modality 2 data is used to represent modality 1 data; After the attention alignment processing, the method further comprises: The output features after modal alignment are added to the features of the original modal, and then normalized by LayerNormalization to accelerate the convergence of the model, so as to obtain the image modal feature data of the final features The text modal feature data is The modality alignment method comprises: when modality 1 is an original image and modality 2 is an original text, using the image original features within a batch as a query Q to calculate the similarity between each image and all text original features K within the batch to obtain attention weights, and then multiplying the attention weights by the specific feature values V of the text original features to obtain the output features after the modality alignment; The method for distributing the final modality data onto a normalized hypersphere to perform category boundary constraint comprises: The features x i and the corresponding weights W yi are L2 regularized, so that ||W yi || = 1, and the normalized features are then multiplied by a rescaling parameter s, so that ||x i || = s, i.e. so that the embedding features are distributed on a hypersphere of radius s; Add a custom additive angular margin m between the feature x i and the target weight Replace the original cosθ yi with cos(θ yi +m).

2. The cross-modal retrieval method of claim 1, wherein, The specific formula represented by the method for distributing onto a normalized hypersphere is: In the above formula, the batch size is N, i.e. i = 1, 2,..., N, x i is the feature input, and its category label is y i , is the feature x i , and the cosine angle of its corresponding weight W yi , m is the angle edge penalty, and n is the number of categories, i.e. k = 1, 2,..., n, W k is the weight of each category, and θ k is the input feature x i , and the cosine angle of the corresponding weight W i of the other k categories misjudged as non-y k category, and the cosine angle of the corresponding k category weight is For the loss function L1 of image retrieval text I2T, the input is The corresponding regularization is That is For the loss function L of text retrieval image T2I T , the input becomes The corresponding regularization is Then the target function used by the proposed maximum semantic correlation and modal alignment model is L Are4emp = L I + L T .

3. Retrieval system employing the cross-modal retrieval method according to any one of claims 1-2, characterized in that, It comprises: An initial module for encoding the features of image and text samples using a CLIP pre-training model to obtain original modality features including original images and texts; An alignment module for subjecting the original modality features to attention alignment processing to obtain modality alignment data to realize semantic correlation between the original modalities; A weight sharing module for subjecting the modality alignment data formed by the above steps to weight-shared multi-layer perception to maintain the invariance of the modalities; A normalization module for distributing the final modality data obtained by using an Arc4cmr loss function onto a normalized hypersphere to perform category boundary constraint.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed to realize the steps of the cross-modality retrieval method according to any one of claims 1-2.

5. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor is executed to realize the steps of the cross-modality retrieval method according to any one of claims 1-2.

Images