Semantic segmentation and modal alignment inference learning cross-modal retrieval method and retrieval system

Abstract

The application provides a semantic subdivision and modal alignment inference learning cross-modal retrieval method and a retrieval system. The semantic subdivision and modal alignment inference learning cross-modal retrieval method comprises the following steps: performing modal alignment on original modal features obtained after pre-training based on scaled dot-product attention, so as to realize re-aggregated projection modal alignment features for the original features; after the modal alignment data formed in the above step passes through a weight-shared multi-layer perception machine, a semantic approximate matching and correct matching method is used to realize semantic correct matching and approximate matching mining for the same category label cluster; and an Arc4cmr loss function, a mutual supervision contrast loss function, a contrast loss function between a picture-text feature similarity matrix and a similar label matrix are used to constrain the model. The cross-modal retrieval method further improves the accuracy of cross-modal retrieval.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval based on maximum semantic relevance and modality alignment, and more specifically, to a cross-modal retrieval method and system based on semantic segmentation and modality alignment reasoning learning. Background Technology

[0002] With the rapid development and increasing maturity of multimedia technology, information carriers have gradually evolved from simple text and images to a combination of various media data. These data possess different forms of existence, data types, data distributions, and data representations, showcasing different angles, dimensions, and levels of things; we collectively refer to this as multimodal data. During the rapid growth of multimedia social platforms, data representations have become increasingly diverse, gradually forming a new pattern of content symbiosis and multi-faceted integration. The flourishing of information dissemination methods has also led to a continuous expansion of retrieval dimensions. For example, when searching for an event or concept, we hope to see related images, videos, charts, and other forms of information for better understanding and memorization; therefore, cross-modal retrieval tasks have emerged.

[0003] Cross-modal retrieval aims to address the problem of heterogeneous underlying features but related high-level semantics across different modalities. Based on whether labeling information is used, cross-modal retrieval can be divided into supervised and unsupervised methods. According to its historical development, cross-modal retrieval can also be broadly categorized into traditional methods based on statistical analysis and modern methods based on deep learning.

[0004] Traditional cross-modal retrieval methods based on statistical analysis:

[0005] 1. Unsupervised Methods: Cross-modal Factor Analysis (CFA) proposed by Li et al. is the earliest traditional unsupervised cross-modal method. This method uses the F-norm as a metric and learns the projected subspaces of different modalities by minimizing the distance between sample pairs of different modalities in the transform domain. It then analyzes the potential matching relationships behind the two modalities in the common subspace. Canonical Correlation Analysis (CCA) proposed by Hotelling et al. is an unsupervised common space learning method and a landmark work in image-text content relevance retrieval.

[0006] 2. Supervised Methods: Rasiwasia et al. proposed a cross-modal retrieval model based on Semantic Correlation Matching (SCM). This model abstracts the semantics of images and text and jointly models the cross-correlated data of the two modalities in the shared space to improve the retrieval accuracy. To fully utilize the fact that inter-modal information in real life is not an absolute one-to-one relationship, Ranjan et al., building upon CCA, constructed a multi-label Canonical Correlation Analysis (ml-CCA) model using one-to-many, many-to-one, and many-to-many relationships generated by multi-labels. This model is more closely aligned with real-world scenarios and exhibits superior performance.

[0007] Traditional cross-modal retrieval methods, based on statistical analysis principles, are simpler to implement. However, the models mostly learn nonlinear relationships or shallow mappings of multimodal data, leaving significant room for improvement in advanced semantic modeling. Furthermore, as the scale of data increases, the computational complexity of traditional methods also increases, and their ability to process high-dimensional data decreases sharply.

[0008] Existing deep learning-based cross-modal retrieval methods mainly include:

[0009] 1. Unsupervised methods: Deep Canonical Correlation Analysis (DCCA) proposed by Andrew et al. utilizes neural networks to learn a nonlinear transformation common space, accurately capturing data correlations and solving the problem that CCA is only applicable to learning linear common spaces. Deep Canonically Correlated Autoencoders (DCCAE) proposed by Wang et al. improves DCCA by adding reconstruction errors based on autoencoders.

[0010] 2. Supervised Methods: Zhai et al. proposed the Joint Representation Learning (JRL) method, which combines multiple modal data within a unified framework and performs sparsification and semi-supervised regularization to explore pairwise and semantic relevance information among them. Zhen et al. proposed an end-to-end Deeply Supervised Cross-Modal Retrieval (DSCMR) method, which preserves semantic discriminability by linearly classifying samples in a common space and learns the relevance between cross-modal data through a weight-sharing strategy to maintain modality invariance.

[0011] Compared with traditional statistical analysis methods, the advantages of deep learning network models in terms of large-scale data processing capabilities, nonlinear structure design, and deep semantic information mining capabilities for complex problems have opened up new ideas and technologies for cross-modal retrieval research. In view of this, this invention is proposed based on deep learning methods. Summary of the Invention

[0012] In view of this, this invention discloses a novel cross-modal retrieval method. It enhances the correlation of semantically related modal features through a modal alignment module based on scaled dot product attention, learns the modal alignment between two modalities, and designs semantic approximate matching and correct matching modules to enhance the aggregation of intra-class image-text features. Simultaneously, it refines the distinction between image-text pairs with semantic information differences within a class. A mutually supervised contrastive loss function is used to enhance fine-grained feature alignment, and a contrastive loss function between the image-text feature similarity matrix and the similarity label matrix is ​​used to ensure that the loss of intra-class semantic approximate matching is less than the loss of inter-class incorrect matching.

[0013] Specifically, the present invention is achieved through the following technical solutions:

[0014] In a first aspect, the present invention discloses a novel cross-modal retrieval method, comprising the following steps:

[0015] The original modal features obtained after pre-training are modally aligned based on scaled dot product attention, so as to re-aggregate the projected modal aligned features for the original features;

[0016] The modality alignment data formed in the above steps is passed through a weight-sharing multilayer perceptron, and then semantic approximate matching and correct matching methods are used to achieve semantic correct matching and approximate matching mining for the same category of label clusters.

[0017] The model is constrained by the Arc4cmr loss function, the mutual supervision comparison loss function, and the comparison loss function between the image and text feature similarity matrix and the similar label matrix.

[0018] Secondly, the present invention discloses a cross-modal retrieval system, comprising:

[0019] Modality alignment module: used to align the original modal features obtained after pre-training based on scaled dot product attention, so as to re-aggregate the projected modality aligned features for the original features;

[0020] Matching module: This module is used to pass the modality alignment data formed in the above steps through a weight-sharing multilayer perceptron, and then use semantic approximate matching and correct matching methods to achieve semantic correct matching and approximate matching mining for the same category of label clusters.

[0021] Constraint module: Used to constrain the model using Arc4cmr loss function, mutual supervision comparison loss function, and comparison loss function between image and text feature similarity matrix and similar label matrix.

[0022] Thirdly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cross-modal retrieval method as described in the first aspect.

[0023] Fourthly, the present invention discloses a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the cross-modal retrieval method as described in the first aspect.

[0024] Currently, most supervised model loss function constraints use class label information as the metric. In multi-class classification models, the output dimension is typically set to the number of classes (let's say N) to output the probability score for each class. Simultaneously, to facilitate calculation and evaluation of the model's prediction performance, the true class labels of the samples are converted into an N-dimensional one-hot vector. Specifically, the pre-defined N classes are represented as a vector of length N, where only the corresponding class position is 1, and the other positions are 0. However, this approach hides the problem of forcibly broadening the semantic information into general categories.

[0025] Furthermore, the original intention of the pre-trained model CLIP is to use text sentences containing more fine-grained information as image labels for image category discrimination, which can effectively alleviate the problem of similar images being forcibly classified into the same category. Existing models lack consideration for fine-grained alignment of image and text features. Based on this, this invention uses the modal feature data of a correctly matched image-text pair as the supervision information of another modal feature to perform fine-grained feature alignment processing.

[0026] To enhance the inference ability of the modality alignment module for feature reconstruction, minimize intra-class distances and maximize inter-class distances between different modalities, and enable effective intra-class subdivision based on semantic information to improve fine-grained alignment of modal features, a Semantic Refinement Discrimination and Modal Alignment inference learning (SRD-MA) method is proposed. + Cross-modal retrieval model. Attached Figure Description

[0027] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0028] Figure 1 This is an overall framework diagram of the cross-modal retrieval method provided in the embodiments of the present invention;

[0029] Figure 2 This is a flowchart illustrating the modality alignment workflow based on scaled dot product attention, as provided in an embodiment of the present invention.

[0030] Figure 3 This is a data processing flowchart for the semantic approximation matching and correct matching module provided in an embodiment of the present invention;

[0031] Figure 4 The label provided in this embodiment of the invention utilizes a similarity label matrix. sim Visual visualization of the processed image-text similarity matrix;

[0032] Figure 5 SRD-MA in Experimental Example 1 of this invention + A comparison of the visualization effects of the model and the SMR-MA model;

[0033] Figure 6 This is a flowchart illustrating a computer device provided in an embodiment of the present invention. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0035] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0037] This invention discloses a cross-modal retrieval method, comprising the following steps:

[0038] The original modal features obtained after pre-training are modally aligned based on scaled dot product attention, so as to re-aggregate the projected modal aligned features for the original features;

[0039] The modality alignment data formed in the above steps is passed through a weight-sharing multilayer perceptron, and then semantic approximate matching and correct matching methods are used to achieve semantic correct matching and approximate matching mining for the same category of label clusters.

[0040] The model is constrained by the Arc4cmr loss function, the mutual supervision comparison loss function, and the comparison loss function between the image and text feature similarity matrix and the similar label matrix.

[0041] The overall framework of the cross-modal retrieval model is as follows: Figure 1 As shown, firstly, the pre-trained CLIP model is used to encode features of image and text samples, obtaining original image features and original text features. Secondly, to optimize the modality alignment module and enhance the correlation between semantically related features of the two modalities, a modality alignment module based on scaled dot product attention is used to re-aggregate projected modality alignment features for the original features. Next, a common representation space is learned using a weight-sharing MLP to maintain modality invariance. Then, semantic approximate matching and correct matching modules are used to achieve semantic correct matching and approximate matching mining for label clusters of the same category, improving the accuracy of cross-modality retrieval. Finally, three loss function constraints are combined to improve inter-class separation and intra-class aggregation, enhance fine-grained feature alignment, and make the loss of intra-class semantic approximate matching less than the loss of inter-class incorrect matching, performing semantic subdivision of intra-class image-text pairs.

[0042] SRD-MA + The feature extraction network used in the model employs an improved ResNet-50 network structure from the CLIP pre-trained model as the image encoder φ and a Transformer network structure as the text encoder ψ to obtain the corresponding original image features. and original text feature representation and in, D represents the feature dimension, which is 1024, and M represents the batch size, which is 100.

[0043] The workflow diagram of the modality alignment module based on scaled dot product attention is as follows: Figure 2 As shown, the model includes two data processing steps: image feature alignment and text feature alignment. The input data for each module consists of the original image features and original text features after feature encoding. The output is the fused image features and fused text features. During this process, the model learns a new joint latent space to highlight images (text) that better match the query text (image) and effectively suppress images (text) that are irrelevant to the query text (image). To enhance modal information interaction and eliminate modal differences, the two data processing steps share the same network structure and parameters.

[0044] In the modality alignment module based on scaled dot product attention, for images retrieved from text, an original text feature is... Transform into a single query All original image features within the batch Convert to key Sum The specific conversion is as shown in Formula 1, and so on. For image-to-text retrieval, the specific conversion is as shown in Formula 2.

[0045]

[0046]

[0047] In formulas 1 and 2, D p D represents the size of the projection dimension. p The value is 1024, and LN represents layer normalization. They are projection matrices of the same dimension.

[0048] Single raw text feature Q t With each original image feature K within a batch v Relevance weights Based on the weighting of the relevance weight coefficients, the projected image features V are re-aggregated. v The text feature output obtained after scaling dot product attention is Attention(Q). t ,K v V v ), as shown in Formula 3.

[0049]

[0050] Similarly, the image feature output obtained after scaling dot product attention is Attention(Q). t ,K v V v ), as shown in Formula 4.

[0051]

[0052] Formulas 3 and 4 The scaling factor effectively avoids the need for attention scores QK. T To address the vanishing or exploding gradient problems caused by excessively large or small attention scores, normalize the attention score to improve model performance and stability.

[0053] To embed images into a shared space with text, the aggregated image representation of the attention module is weighted. Projection Back As shown in Formula 5:

[0054] r v∣t =LN(Attention(Q) t ,K v V v W O (5)

[0055] Where, r v∣t This represents aggregated image features conditioned on text t.

[0056] To embed text into a shared space containing images, the aggregated text representation of the attention module is weighted. Projection Back As shown in Formula 6:

[0057] r t∣v =LN(Attention (Q) v ,K t V t W o (6)

[0058] Where, r t∣v It is an aggregated text feature conditioned on image v.

[0059] The final image modal features obtained through modal interaction based on scaling dot product attention, with text as the retrieval condition, are represented as C. v |t, which is the final fusion output of the text feature alignment process, as shown in Equation 7, the text modal features with images as retrieval conditions are represented as C t |v, which is the final fusion output of the image feature alignment process, as shown in Equation 8.

[0060]

[0061]

[0062] in, single element Specifically, it refers to a single raw text within a batch. The features are re-represented as single element Specifically, it refers to the features of a single original image within a batch. Represented as

[0063] The image features obtained after the above processing and text features The input is fed into a weight-sharing multilayer perceptron (MLP). The MLP structure is y(x) = W2(ε(W1x+b1)), where ε represents the GeLU activation function, W1 and W2 represent trainable weight parameters, and b1 and b2 represent bias terms. The image features processed by the weight-sharing MLP are v. i =y(c i I The text features are At this point, all image and text features within the batch are represented as C' t |v=[v1,v2,…,v i ,…,v M ] T C' v |t=[u1,u2,…,u j ,…,u M ] T ,in,

[0064] The data processing principle diagram of the semantic approximate matching and correct matching modules is as follows: Figure 3 As shown, this study simultaneously considers correct matching, semantic approximation matching, and complete mismatch between classes for image-text pairs, accurately distinguishing their positive and negative effects on cross-modal retrieval from different matching dimensions. Two different attention masking mechanisms are used to process image-text pairs with intra-class semantic approximation matching and those with correct matching. First, all image features u within a batch are calculated. i and text features v j The semantic relevance score between them ij Specifically, as shown in Formula 9.

[0065]

[0066] According to Formula 9, the text-to-image (T2I) similarity matrix attn can be obtained. t The formula is 10.

[0067]

[0068] Similarly, the image-to-text (I2T) similarity matrix can be represented as attn v As shown in Formula 11.

[0069]

[0070] The category labels of all images and texts in a batch may be repeated. By performing an XOR operation on the label values ​​at the corresponding positions, i.e., 1 for the same and 0 for different, the similarity label matrix of the approximately matched image-text pairs can be represented by Formula 12.

[0071]

[0072] label with similarity matrix sim Based on this, the text-to-image similarity matrix attn t The similarity weight can be re-expressed as Similarly, the image-to-text similarity matrix attn v The similarity weight can be re-expressed as Specifically, by utilizing the inherent properties of the Softmax function, the similarity of correctly matched image-text pairs is increased. At the same time, the similarity of semantically approximate image-text pairs within a class is made much greater than the similarity of unrelated image-text pairs between classes. This effectively suppresses the negative impact of incorrect matching between classes, enhances the separation between classes, and effectively enhances intra-class cohesion while increasing the proportion of correct matching similarity.

[0073] Using similarity label matrix sim The visual visualization of the processed image-text similarity matrix is ​​available. Figure 4 The diagram is represented as follows: Diagonal elements represent image-text pairs that are correctly matched based on the original semantic information; darker colors indicate greater similarity. Image-text pairs with semantically approximate matches of the same category label have lower similarity than correctly matched pairs, and are represented by lighter colors. Conversely, image-text pairs belonging to different and unrelated categories are represented by lighter colors, indicating they are negative samples to each other.

[0074] When measuring the similarity of image-text pairs, the primary focus is on the shared semantics between the two modalities. For the i-th text, the relevant image features in a batch-sized image library can be aggregated as follows: Formula 13 is shown below:

[0075]

[0076] in, Expanded representation It is an approximate semantic relationship between text and image, as shown in Formula 14.

[0077]

[0078] The relevant text features of the i-th image in a text library of batch size can be aggregated as follows: Formula is defined as 15:

[0079]

[0080] in, Expanded representation It is an approximate semantic relationship between images and text, expressed by a specific formula as shown in Formula 16.

[0081]

[0082] In formulas 14 and 16, λ is the penalty coefficient, and Mask sim (·) represents the masking function. When the input is positive, the output is equal to the input; otherwise, the output is -∞. After processing by the Softmax function, the attention weights of irrelevant samples are reduced to 0, thereby achieving effective attention to relevant samples.

[0083] Based on the semantic information content of the image-text pair, the label discrimination matrix corresponding to the correct matching relationship is the identity matrix, expressed by formula 17.

[0084]

[0085] label matrix eql Based on this, the similarity weight attn of correctly matched image-text pairs is calculated. eql That is, the text-to-image similarity matrix attn t With unit label matrix eql similarity ratio Image-to-text similarity matrix attn v With unit label matrix eql similarity ratio Among them, Mask eql(·) is a masking function; the output is 1 when the input is positive and 0 otherwise. According to the soft attention calculation rules, correctly matched image features can be re-represented as... The correctly matched text features can be re-represented as

[0086] Based on the above analysis, the image features jointly represented by feature aggregation of approximate semantic association and correctly matched features are shown in Equation 18, and the text features are shown in Equation 19.

[0087]

[0088]

[0089] SRD-MA + The objective function of the model mainly consists of three parts: the Arc4cmr (ArcFace loss for Cross-ModalRetrieval) loss function, the mutual supervision comparison loss function, and the comparison loss function between the image and text feature similarity matrix and the similar label matrix.

[0090] The ArcFace loss function is specifically expressed as Equation 20, where x i Represents the feature input, y i It is the corresponding category label. θk Indicates feature x i and corresponding weight W k In the angle space, s is the radius of the hypersphere, n is the number of categories, and M is the batch size. The constraints can be expressed by Equation 21.

[0091]

[0092]

[0093] For text retrieval image tasks, the ArcFace loss function L... simT2I The corresponding input is The corresponding regularization is Specifically, as shown in Formula 22.

[0094]

[0095] For the image retrieval text task, the ArcFace loss function L... simI2T The corresponding input is The corresponding regularization is Specifically, as shown in Formula 23.

[0096]

[0097] Combining formulas 22 and 23, we can see that SRD-MA + The Arc4cmr loss function L of the model Arc4cmr This can be represented as Formula 24.

[0098] L Arc4cmr =L simT2I +L simI2T (twenty four)

[0099] The goal of training the network model is to minimize the distance between positive pairs (where the content of the image and the text description are consistent) and maximize the distance between negative pairs (where the image presentation and the text description contradict each other). Therefore, a contrastive loss function is incorporated into the objective function. In feature x... i Other features x j The similarity between the two is measured using a normalized temperature-scaled Softmax similarity metric, L. contr As shown in Formula 25:

[0100]

[0101] Where τ represents the isothermal parameter, and sim(·) represents the L2 normalization of the input followed by a dot product. Unlike directly using cosine similarity for measurement, normalized Softmax can amplify the similarity of positive pairs and weaken the irrelevance of negative pairs. The contrast loss function is the arithmetic mean of the normalized similarity cross-entropy of all positive pairs in the batch, assuming... It is in terms of content with x i For matched samples, the contrastive loss function L... contr Available public

[0102] Equation 26 shows that B is the size of the sample set.

[0103]

[0104] By using the features of matched image-text pairs as mutual supervision signals, fine-grained feature alignment is achieved. Thus, the mutual supervision contrastive loss function L... contr-MutlSup It can be represented by formula 27.

[0105]

[0106] Traditional classification loss functions do not differentiate between image-text pairs that belong to the same broad category but have different semantic information; instead, they are roughly aggregated within the same category. Our goal is to achieve inter-class separation, intra-class aggregation, and effective subdivision of semantic information differences within categories. We then refine the image-text feature similarity matrix sim...t sim v Similar label matrix sim Compare the loss functions between them The loss for intra-class semantic approximation matching is less than the loss for inter-class incorrect matching, as specifically shown in Formula 28:

[0107]

[0108] Where M is the batch size, sim t The text-image feature similarity matrix, jointly represented using text as the retrieval condition, is shown in Equation 29; v The image-text feature similarity matrix is ​​represented by images as the retrieval criteria, as shown in Formula 30, where ρ is a learnable parameter.

[0109] sim t =ρΣF t F v (29)

[0110] sim v =ρ∑F v F t (30)

[0111] Combining formulas 24, 27, and 28, SRD-MA + The overall objective loss function of the model is shown in Equation 31. μ is a hyperparameter that reflects the contribution of different loss functions to the objective function.

[0112]

[0113] In addition to providing a cross-modal retrieval method, this invention also provides a cross-modal retrieval system, specifically including:

[0114] Modality alignment module: used to align the original modal features obtained after pre-training based on scaled dot product attention, so as to re-aggregate the projected modality aligned features for the original features;

[0115] Matching module: This module is used to pass the modality alignment data formed in the above steps through a weight-sharing multilayer perceptron, and then use semantic approximate matching and correct matching methods to achieve semantic correct matching and approximate matching mining for the same category of label clusters.

[0116] Constraint module: Used to constrain the model using Arc4cmr loss function, mutual supervision comparison loss function, and comparison loss function between image and text feature similarity matrix and similar label matrix.

[0117] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of each unit, please refer to the previous method implementation examples, which will not be repeated here.

[0118] In summary, this invention is an optimization based on a cross-modal retrieval method and system patent (CN 115563316 A). Compared to the SMR-MA model in CN 115563316 A, the two models differ in the network structure of their modality alignment modules. The SMR-MA model's modality alignment module is based on decomposable attention, while the SMR-MA model... + One key difference is that the model's modality alignment module is based on scaled dot product attention. (SRD-MA) + The specific workflow differences of the modality alignment module in the model are as follows: Before being fed into the core computation module, the original feature data obtained from the encoding process is linearly projected; the correlation matrices calculated from the two modalities are scaled and normalized; the aligned modal features are processed through a fully connected layer before being added to the original feature representation. Furthermore, SRD-MA... + The model incorporates additional semantic approximate matching and correct matching modules, which is the second difference. The objective functions of the two models overlap; the SMR-MA model relies solely on the Arc4cmr loss function L... Arc4cmr Apply model constraints, and based on this, SRD-MA + The model also incorporates a mutual supervised comparison loss function L. contr-MutlSup The contrast loss function L between the image / text feature similarity matrix and the similarity label matrix contr-sim Further optimization of the cross-modal retrieval method improved work efficiency and shortened model training time.

[0119] Experimental Example 1

[0120] To verify the model's performance, experimental analyses were conducted on three benchmark datasets: Wikipedia, Pascal-Sentence, and NUS-WIDE. Table 1 shows that SRD-MA... + The model demonstrates significant performance improvements in image-to-text (I2T), text-to-image (T2I), and the average (Avg) of both retrieval tasks. The model's convergence speed is also greatly improved, saving computational costs and shortening training time. Overall, SRD-MA... + The overall performance of the model is better than that of the SMRA-MA model.

[0121] Table 1 SMR-MA and SRD-MA+ Performance comparison

[0122]

[0123] To observe SRD-MA more clearly + For the inter-class separation and intra-class aggregation effects of the model and the SMR-MA model on the two modal features, we used the t-SNE nonlinear dimensionality reduction algorithm for visualization. Figure 5 The visualization results are shown on the Wikipedia dataset. In the figure, circles "○" represent image features and triangles "△" represent text features.

[0124] By comparing the distribution of image and text features in the common representation space of the two models. Figure 5 As seen in (a) and (b), both models demonstrate good separation of feature distributions across different categories. The two modalities with the same semantic meaning show good overlap, proving that both models effectively eliminate modal heterogeneity. The image features and text features of the same category have roughly the same shape, indicating that SRD-MA... + The model effectively eliminates modal discrepancies, allowing features of two modalities within the same category to overlap effectively, while also achieving effective separation for different categories. In comparison, SRD-MA... + The model performs better in the common representation space, with fewer discrete points for samples of the same class, and better intra-class aggregation.

[0125] Analyzing the distribution of image features or text features in isolation, SRD-MA + The model can effectively distinguish samples of different semantic categories and classify them into corresponding semantic category clusters. For image features, such as... Figure 5 (c) and (d), SRD-MA + The model achieved good separation results for different categories, while the SMR-MA model showed instances of different categories being intertwined, which contradicts the purpose of cross-modal retrieval. For example... Figure 5 As shown in (e) and (f), SRD-MA + The model outperforms the SMR-MA model and SRD-MA model in text feature processing. + The model exhibits higher intra-class aggregation and fewer discrete points, achieving better intra-class segmentation, which aligns with the initial design goals of semantic approximate matching and correct matching modules. Furthermore, comparing the performance of the two models on different modal feature distributions, SRD-MA demonstrates superior performance. + The image features and text features of the same category in the model are more similar in shape, while the SMR-MA model shows greater clustering for text features and greater dispersion for image features. A comprehensive analysis of the above visualizations explains the SRD-MA model.+ The direct visual reason why the model outperforms the SMR-MA model.

[0126] Figure 6 This is a schematic diagram of the structure of a computer device disclosed in this invention. (Reference) Figure 6 As shown, the computer device 400 includes at least a memory 402 and a processor 401; the memory 402 is connected to the processor via a communication bus 403 and is used to store computer instructions executable by the processor 401. The processor 401 is used to read computer instructions from the memory 402 to implement the steps of the cross-modal retrieval method described in any of the above embodiments.

[0127] For the above-described apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0128] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0129] Finally, it should be noted that although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe the features of specific embodiments of a particular invention. Certain features described in the various embodiments of this specification may also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0130] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0131] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0132] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A cross-modal retrieval method based on semantic segmentation and modal alignment reasoning learning, characterized in that, Includes the following steps: The original modal features obtained after pre-training are modally aligned based on scaled dot product attention, so as to re-aggregate the projected modal aligned features for the original features; The modality alignment data formed in the above steps is passed through a weight-sharing multilayer perceptron, and then semantic approximate matching and correct matching methods are used to achieve semantic correct matching and approximate matching mining for the same category of label clusters. The model is constrained by the Arc4cmr loss function, the mutual supervision comparison loss function, and the comparison loss function between the image and text feature similarity matrix and the similarity label matrix. Methods for modality alignment based on scaled dot product attention include: Original text features Transform into a single query All original image features within the batch Convert to key Sum The specific conversion method is shown in Formula 1. Similarly, the specific conversion method for image retrieval text tasks is shown in Formula 2. In the above formula, D p D is the size of the projection dimension. p The value is 1024, and LN represents layer normalization. They are projection matrices of the same dimension; Single raw text feature Q t With each original image feature K within a batch v Relevance weights Based on the weighting of the relevance weight coefficients, the projected image features V are re-aggregated. v The text feature output obtained after scaling dot product attention is Attention(Q). t ,K v V v ), as shown in Formula 3; Similarly, the image feature output obtained after scaling dot product attention is Attention(Q). t ,K v V v Specifically, see Formula 4; Formulas 3 and 4 This is the scaling factor; Then the aggregated image representation of the attention module is weighted. Projection Back As in formula 5: r v∣t =LN(Attention(Q t ,K v ,V v )W O ) (5) Where, r v∣t This represents aggregated image features conditioned on text t; The aggregated text representation of the attention module is weighted. Projection Back As in formula 6: r t∣v =LN(Attention(Q v ,K t ,V t )W o ) (6) Where, r t∣v It is an aggregated text feature conditioned on image v; The final image modal features obtained through modal interaction based on scaling dot product attention, with text as the retrieval condition, are represented as C. v |t, which is the final fusion output of text feature alignment processing, as shown in Formula 7, where the text modal features with images as retrieval conditions are represented as C. t |v, which is the final fusion output of the image feature alignment process, as shown in Formula 8; In the above formula, single element Specifically, it refers to a single raw text within a batch. The features are re-represented as single element Specifically, it refers to the features of a single original image within a batch. Represented as 2. The cross-modal retrieval method according to claim 1, characterized in that, The methods for semantic approximate matching and correct matching include the following steps: Calculate all image features u within a batch i and text features v j The semantic relevance score between them ij For details, see Formula 9; According to Formula 9, the text-to-image T2I similarity matrix attn can be obtained. t The formula is 10; The image-to-text I2T similarity matrix can be represented as attn v For example, Formula 11; If all images and texts in a batch have the same category label of 1 and different category labels of 0, then the similarity label matrix of approximately matched image-text pairs can be represented by Formula 12. label with similarity matrix sim Based on this, the text-to-image similarity matrix attn t The similarity weight can be re-expressed as Similarly, the image-to-text similarity matrix attn v The similarity weight can be re-expressed as The relevant image features of the i-th text in the batch-sized image library can be aggregated as follows: Formula 13: in, Expanded representation It is an approximate semantic relationship between text and image, as shown in Formula 14; The relevant text features of the i-th image in a text library of batch size can be aggregated as follows: Formula 15: in, Expanded representation It is an approximate semantic relationship between images and text, and the specific formula is expressed as in Formula 16; In formulas 14 and 16, λ is the penalty coefficient, and Mask sim (·) represents a mask function; when the input is positive, the output is equal to the input, otherwise the output is -∞. The label discrimination matrix corresponding to the correct matching relationship is the identity matrix, and the formula is 17. label matrix eql Based on this, the similarity weight attn of correctly matched image-text pairs is calculated. eql That is, the text-to-image similarity matrix attn t With unit label matrix eql similarity ratio Image-to-text similarity matrix attn v With unit label matrix eql similarity ratio Among them, Mask eql (·) is a mask function; it outputs 1 when the input is positive and 0 otherwise. According to the soft attention calculation rules, the correctly matched image features are re-represented as follows: Correctly matched text features are re-represented as Image features represented by feature aggregation based on approximate semantic association and correctly matched features are shown in Equation 18, and text features are shown in Equation 19, where, 3. The cross-modal retrieval method according to claim 2, characterized in that, The method for using the Arc4cmr loss function to constrain the model is expressed by Equation 21: In the above formula, x i Represents the feature input, y i It is the corresponding category label. θk Indicates feature x i and corresponding weight W k In the angle space, s is the radius of the hypersphere, n is the number of categories, and M is the batch size; For text retrieval image tasks, the ArcFace loss function L... simT2I The corresponding input is The corresponding regularization is As shown in formula 22; For the image retrieval text task, the ArcFace loss function L... simI2T The corresponding input is The corresponding regularization is As shown in formula 23; Combining formulas 22 and 23, we can see that the Arc4cmr loss function L... Arc4cmr Represented as Formula 24; L Arc4cmr L simT2I +L simI2T (24) 4. The cross-modal retrieval method according to claim 2, characterized in that, Methods for constraining models using mutually supervised comparison loss functions include: In feature x i Other features x j The similarity between the two is measured using a normalized temperature-scaled Softmax similarity measure. As in formula 25: In the above formula, τ represents the isothermal parameter, and sim(·) represents the dot product after L2 normalization of the input; The contrast loss function is the arithmetic mean of the normalized similarity cross-entropy of all pairs in the batch, assuming... It is in terms of content with x i For matched samples, the contrastive loss function L... contr This can be expressed by Formula 26, where B is the size of the sample set; By using the features of matched image-text pairs as mutual supervision signals, fine-grained feature alignment is achieved, and the mutual supervision contrastive loss function L... contr-MutlSup Expressed using Formula 27; 5. The cross-modal retrieval method according to claim 2, characterized in that, Methods for constraining models using the comparison loss function between the image-text feature similarity matrix and the similarity label matrix include: The image-text feature similarity matrix sim t sim v Similar label matrix sim Compare the loss functions between them The loss for intra-class semantic approximation matching is less than the loss for inter-class incorrect matching, as specifically shown in Formula 28: Where M is the batch size, sim t The text-image feature similarity matrix, jointly represented by text as the retrieval condition, is shown in Formula 29. sim v The image-text feature similarity matrix is ​​jointly represented using images as retrieval criteria, as shown in Formula 30, where ρ is a learnable parameter. Yes t =ρ∑F t F v (29) Yes v =ρ∑F v F t (30) Combining formulas 24, 27, and 28, the overall target loss function is shown in formula 31, where θ and μ are hyperparameters.

6. A retrieval system employing the cross-modal retrieval method according to any one of claims 1-5, characterized in that, include: Modality alignment module: used to align the original modal features obtained after pre-training based on scaled dot product attention, so as to re-aggregate the projected modality aligned features for the original features; Matching module: This module is used to pass the modality alignment data formed in the above steps through a weight-sharing multilayer perceptron, and then use semantic approximate matching and correct matching methods to achieve semantic correct matching and approximate matching mining for the same category of label clusters. Constraint module: Used to constrain the model using Arc4cmr loss function, mutual supervision comparison loss function, and comparison loss function between image and text feature similarity matrix and similar label matrix.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the steps of the cross-modal retrieval method according to any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the cross-modal retrieval method as described in any one of claims 1-5.

Images