Image and text retrieval method and system based on cross-modal semantic analysis

Abstract

The present invention relates to a method and system for image-text retrieval based on cross-modal semantic parsing, comprising: image representation: understanding a given image and generating feature encodings of salient areas; text representation: understanding a given text query sentence and generating context-related discrete vocabulary encodings; using a self-attention mechanism to perform intra-modal feature fusion on image and text representations; using hash codes and quantization codes generated by aggregated features to respectively calculate the cosine similarity of image-text pairs, screening out top-ranked candidate sets through two rounds of sorting, introducing a cross-modal attention mechanism to calculate the candidate sets to obtain relatively accurate fine-grained matching scores, using similarity re-ranking to fine-tune the internal ranking relationship, and finally achieving high-performance cross-modal image-text retrieval.

Description

Technical Field

[0001] The present invention relates to a picture and text retrieval method and system based on cross-modal semantic analysis, and belongs to the technical field of cross-modal retrieval and artificial intelligence. Background Art

[0002] Image-text retrieval systems, using advanced image recognition and text analysis technologies, have been widely used in e-commerce, social media analysis and other scenarios, becoming an important tool for information acquisition and data analysis. However, most existing image-text retrieval systems rely on keyword matching of entity information, which often fails to fully understand the complex relationship between image content and text description.

[0003] The research on cross-modal image-text matching based on natural language not only deepens the research on traditional image-text retrieval, but also improves the convenience of human-computer interaction. Currently, this research field mainly faces the following two challenges:

[0004] (1) Effective cross-modal semantic alignment. For large-scale image libraries, the query requirements of different users are significantly diverse. For example, user A may want to query images of "children in red clothes playing on the beach", while user B may want to find images of "children in red clothes playing on the grass in the park". These target images have obvious differences in the content presented. Therefore, how to achieve effective cross-modal semantic alignment in a large number of images for different users' text query statements is an important challenge.

[0005] (2) Efficient cross-modal matching mechanism. Existing mainstream image-text matching methods have two directions: global-level matching tends to learn global alignment, that is, representing images or texts as overall features to measure similarity; local-level matching focuses on fine-grained alignment between local fragments, that is, inferring the overall image-text similarity through the correlation of all word-region pairs. Among them, global matching has high retrieval efficiency but poor accuracy; and although local-level matching has good retrieval accuracy, its computational complexity is too high and reasoning is too time-consuming, and its actual implementation ability is greatly limited. Therefore, it is also challenging to explore a cross-modal image-text matching mechanism that has both retrieval accuracy and efficiency. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a picture-text retrieval method based on cross-modal semantic parsing;

[0007] The present invention constructs an efficient cross-modal image-text retrieval framework to perform bidirectional matching of semantically similar images and texts. To this end, the present invention first proposes an end-to-end two-stage filtering strategy, which uses hash codes and quantization codes to filter out a small number of data points with high positive confidence scores from coarse to fine, thereby greatly reducing the number of samples that need to be calculated while covering as many positive samples as possible; secondly, the present invention introduces a cross-modal attention mechanism to perform fine-grained calculations on the screened candidate samples to obtain matching results with higher accuracy; finally, the present invention proposes a bidirectional similarity re-ranking technology to improve the accuracy loss caused by the difficulty of aligning semantic information in cross-modal retrieval.

[0008] The present invention focuses on the research of cross-modal image-text retrieval technology, that is, on the one hand, through the query statement described in natural language, images matching the query semantics are retrieved from a large amount of image data; on the other hand, through a given query image, text descriptions with similar semantics are retrieved in text data. Taking the direction of a text query image as an example, according to the query statement described in natural language ("children wearing red clothes playing in the park"), images matching the query semantics are retrieved from the image database and displayed.

[0009] The present invention also provides a picture-text retrieval system based on cross-modal semantic analysis.

[0010] Terminology explanation:

[0011] 1. ResNet-101: ResNet is a deep neural network architecture proposed by Microsoft Research. ResNet introduces residual modules to train deeper neural networks. In the ResNet series, the number part represents the depth of the model. ResNet-101 is a deep neural network model with 101 layers and a residual learning mechanism.

[0012] 2. Visual Genome Dataset: The Visual Genome Dataset is a large-scale visual reasoning dataset released by the Department of Computer Science and the Intelligent Information Systems Laboratory at Stanford University. It consists of 101,174 images from MSCOCO and 1.7 million question-answer pairs, with an average of 17 questions per image, and provides 108K images with densely annotated objects, attributes, and relationships. It is an important resource for visual scene understanding and reasoning research.

[0013] 3. Faster R-CNN: Faster R-CNN is a classic object detection algorithm proposed by Ross Girshick in 2015. It is an improved version of the R-CNN series of algorithms, which improves the detection speed by introducing the region proposal network. Its core idea is to divide the object detection task into two stages: candidate region generation and object classification. Among them, the region proposal network is used to generate candidate object regions, and the subsequent Fast R-CNN network is used to classify and regress these candidate regions. Compared with the traditional R-CNN algorithm, it has higher detection speed and better accuracy, making the object detection task more efficient in practical applications.

[0014] 4. Bidirectional Gated Recurrent Neural Network: Bidirectional Gated Recurrent Neural Network (Bi-GRU) combines the bidirectional characteristics of Bidirectional Recurrent Neural Network (Bi-RNN) and the gating mechanism of Gated Recurrent Unit (GRU) to process sequence data and capture contextual information. GRU is a variant of recurrent neural network that has update gates and reset gates to control the flow of information and the update of memory. The bidirectional structure enables Bi-GRU to have two hidden states, forward propagation and backward propagation, at each time step. Therefore, the model will take into account the information before and after the current moment at the same time, thereby more comprehensively understanding the sequence data. Bi-GRU is suitable for various sequence modeling tasks, such as text classification, named entity recognition, sentiment analysis, etc. in natural language processing, as well as time series prediction, speech recognition and other fields.

[0015] 5. Mini-Batch K-means clustering algorithm: Mini-Batch K-means is an improved version of the K-means clustering algorithm. Unlike the standard K-means algorithm, Mini-Batch K-means uses small batches of random samples to update the cluster centers, thereby speeding up convergence. The main idea is to randomly select a small number of samples (i.e., small batches) from the dataset for clustering at each iteration, and update the cluster centers based on the average value of these samples. This process is called "one iteration". Compared with the standard K-means algorithm, Mini-Batch K-means can greatly reduce the number of iterations and computing time, especially on large-scale datasets.

[0016] The technical solution of the present invention is:

[0017] The image and text retrieval method based on cross-modal semantic analysis includes:

[0018] The query process of image and text is consistent, so we will take the query from image to text as an example to illustrate.

[0019] Image representation: understand a given image and generate feature encodings of salient areas;

[0020] Text representation: Understand a given text query and generate context-dependent discrete vocabulary encodings;

[0021] Use self-attention mechanism to perform intra-modal feature fusion on image and text representations;

[0022] The hash codes and quantization codes generated by aggregated features are used to calculate the cosine similarity of image-text pairs respectively. After two rounds of sorting, the top candidate sets are screened out. The cross-modal attention mechanism is introduced to calculate the candidate sets to obtain a more accurate fine-grained matching score. The similarity reranking is used to fine-tune the internal ranking relationship, and finally high-performance cross-modal image-text retrieval is achieved.

[0023] As a further preferred solution, the learning process of image or text representation includes:

[0024] For the i-th image, ResNet-101 is used as the backbone, and FasterR-CNN pre-trained on the visual genome dataset extracts the corresponding regional features; the final regional features V = [v1, v2, ..... v n ];

[0025] A bidirectional gated recurrent neural network with d-dimensional hidden layer is used to obtain the information of the forward and reverse hidden layer states of the input text. The average value of the information of the forward and reverse hidden layer states of the text is used as its feature representation to obtain the final vocabulary feature T = [t1, t2, .....t m ];

[0026] For visual feature aggregation, the regional average feature obtained in formula (1) is used As the query, as shown in formula (2), the attention mechanism is used to learn the attention weights of the regional fragments and average features, and the global visual features are obtained by weighting them with the original features.

[0027]

[0028]

[0029] Similarly, according to formulas (3) and (4), we can obtain the aggregated global text features:

[0030]

[0031]

[0032] Among them, W vq, W vk , W vv , W tq , W tk , W tv is the learnable mapping matrix, d k is the size of the hidden layer dimension;

[0033] A multi-layer perceptron is used to incorporate the features of these two modes and Projected into the dimensional space corresponding to the hash code, the tanh function is used as the activation function to limit the value of the vector to between [-1, +1]. The aggregated image features and text features are expressed as shown in equations (5) and (6), respectively:

[0034]

[0035]

[0036] The output vector of the hash layer is discretized into an integer hash code represented by -1 and +1 through the sign function sgn, which can be expressed as equations (7) and (8):

[0037]

[0038]

[0039] Where k represents the generated hash code b v and b t Length;

[0040] The relationship between samples is constrained by the triple ranking loss function shown in formula (9):

[0041]

[0042] in, and represents the second highest order negative text that is the most difficult to distinguish; γ represents the marginal distance between the positive and negative labels, S r represents the cosine similarity of the embedding vector; [a] + represents max(0,a);

[0043] The image modality vector output by the hash layer is and the text modality vector The Mini-Batch K-means clustering algorithm is used to divide each feature vector into vectors with shorter lengths of integer segments. The entire high-dimensional vector space is split into M low-dimensional vector spaces. The clustering process is then performed on each subspace. The resulting clusters are called codebooks C. l, the following equations (10) and (11) use the clusters in the codebook to replace the original features to obtain an approximate refined representation:

[0044]

[0045]

[0046] in, Represent the index of the l-th subspace of image and text respectively; and Represent the quantization codes of images and texts respectively;

[0047] The constraint is performed through the quantization loss of formula (12):

[0048]

[0049] Among them, D. Represent the aggregate vectors of dataset samples, images, and texts respectively, and C represents the codebook; is the Frobenius norm.

[0050] As a further preferred solution, the generated hash codes and quantization codes of the image and text are used to perform two-stage screening to obtain a candidate set, including:

[0051] Hash rough screening: Based on the generated hash code, directly calculate the query image hash code b v and the hash code of the entire text The Hamming similarity of the matching scores is sorted in descending order to obtain L rankings; the first K1 data are selected as the initial candidate set for subsequent detailed retrieval;

[0052] Quantitative Detailed Screening: Generated Quantization Codes Based on Query Images The original vector corresponding to the text in the initial candidate set Perform an asymmetric similarity measurement to obtain a quantitative matching score, and after sorting, select the top K texts with higher priority in the K1 initial rankings as the final candidate set.

[0053] As a further preferred solution, a backbone network based on a cross-modal attention mechanism is introduced to calculate the final candidate set to obtain a more accurate fine-grained matching score; including:

[0054] Given a region feature V of a query image I in the final candidate set, n ] and the vocabulary features T of a text E = [t1, t2, .....t m ], calculate the cosine similarity matrix of all region-word pairs, that is:

[0055] s ij= cosine(v i ,t j ),i∈[1,n],j∈[1,m](13)

[0056] Among them, v i ,t j Represent the i-th image region feature and the j-th text vocabulary feature respectively;

[0057] The similarity matrix of the region-word pair is normalized, and the attention score α of the i-th region feature on the j-th text word feature is further obtained ij , where λ1 is the introduced smoothing coefficient:

[0058]

[0059]

[0060] According to the attention matrix, we get the weighted representation of the i-th region feature with respect to all the lexical features of the text. Calculate v i and The cosine similarity of is used to obtain the similarity semantic matrix of all regions and texts:

[0061]

[0062]

[0063] The similarity matrix and semantic matrix are averagely pooled in the image-to-text direction to obtain the aggregated global matching score of image I and text E:

[0064]

[0065] The matching scores are sorted from large to small, and the initial refinement result is shown in formula (19):

[0066] R(I,K)={E1,E2,...E j ,...,E K}(19)

[0067] Among them, R(I,K) is the initial ranking obtained by the cross-modal attention mechanism: I represents the query image, K represents the number of top candidate sets selected, and E j Represents a text candidate point.

[0068] As a further preferred solution, bidirectional similarity re-ranking is used to fine-tune the internal ranking relationship, and finally high-performance cross-modal image-text retrieval is achieved; including:

[0069] Use the top K2 texts ranked in the initial refinement results for re-ranking;

[0070] Based on the generated representations, the similarity scores of the texts to be rearranged, i.e., the first K2 texts, with respect to the query image are calculated respectively;

[0071] The text is reordered from small to large according to the reallocated sequential index to obtain the final retrieval result.

[0072] Further preferably, the bidirectional similarity reordering includes:

[0073] Calculate the matching scores of the first K2 returned result texts of the query image I and all N images and sort them in descending order:

[0074] R(E j ,N)={I1,I2,....I N},j∈[1,K2](20)

[0075] Redistribute each text E according to formula (21) j Sequential index of:

[0076] p(E j )=s,I s =I(21)

[0077] According to formula (22), the texts are sorted from small to large using the indexes assigned to them, and the final sorting result is obtained:

[0078]

[0079] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a graphic and text retrieval method based on cross-modal semantic parsing when executing the computer program.

[0080] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for image-text retrieval based on cross-modal semantic parsing.

[0081] The image and text retrieval system based on cross-modal semantic analysis includes:

[0082] The multimodal information encoding module is configured to: understand a given image and generate a regional feature encoding; understand a given text query sentence and generate a context-related vocabulary feature encoding;

[0083] The intra-modal feature fusion module is configured to: use a self-attention mechanism to perform intra-modal feature fusion on image and text representations;

[0084] The double filtering module is configured to: use the hash code and quantization code generated by the aggregated features to calculate the cosine similarity of the image-text pair respectively, and select the top-ranked candidate sets through two rounds of sorting;

[0085] The fine-grained computing and re-ranking module is configured as follows: a cross-modal attention mechanism is introduced to calculate a more accurate fine-grained matching score for the candidate set, similarity re-ranking is used to fine-tune the internal ranking relationship, and finally high-performance cross-modal image and text retrieval is achieved.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] 1. This invention proposes an end-to-end two-stage filtering strategy. This framework divides the retrieval process into coarse screening and fine screening: first coarse screening, then fine screening, so as to extract a highly relevant candidate set, greatly reducing the number of samples that need to be calculated, and fully considering the retrieval efficiency problems existing in the actual implementation of existing methods and providing improvements.

[0088] 2. The present invention proposes a method that combines a cross-modal attention mechanism with similarity reranking, performs fine-grained calculations on the screened high-confidence candidate set, and obtains results comparable to those of the backbone attention method; then, a reranking algorithm is designed using the semantic complementarity of bidirectional retrieval, which further effectively improves the retrieval performance.

[0089] 3. The method proposed in the present invention is a lightweight plug-and-play framework that can be embedded into any existing backbone image-text retrieval framework and has the advantage of model parameter portability. A large number of performance comparison experiments conducted on open source benchmark data can prove that the model provided by the present invention can achieve accurate and efficient cross-modal image-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a block diagram of an implementation of the image-text retrieval method based on cross-modal semantic parsing of the present invention;

[0091] Figure 2 is a flow chart of image and text representation generation of the present invention;

[0092] Figure 3 is a two-stage filtration flow chart of the present invention;

[0093] Figure 4 is the attention refinement calculation and reordering flow chart of the present invention; DETAILED DESCRIPTION

[0094] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0095] Example 1

[0096] The image and text retrieval method based on cross-modal semantic analysis includes:

[0097] like Figure 1 As shown in Figure 1, an advanced hierarchical retrieval network (AHR) is constructed to achieve efficient and accurate cross-modal image-text bidirectional retrieval. The image and text bidirectional query has a consistent process, so the image-to-text query is used as an example to illustrate.

[0098] Image representation: understand a given image and generate feature encodings of salient areas;

[0099] Text representation: Understand a given text query and generate context-dependent discrete vocabulary encodings;

[0100] Use self-attention mechanism to perform intra-modal feature fusion on image and text representations;

[0101] The hash codes and quantization codes generated by aggregated features are used to calculate the cosine similarity of image-text pairs respectively. After two rounds of sorting, the top candidate sets are screened out. The cross-modal attention mechanism is introduced to calculate the candidate sets to obtain a more accurate fine-grained matching score. The similarity reranking is used to fine-tune the internal ranking relationship, and finally high-performance cross-modal image-text retrieval is achieved.

[0102] Example 2

[0103] The difference between the image-text retrieval method based on cross-modal semantic parsing described in Example 1 is that:

[0104] The learning process of image or text representation, such as Figure 2 As shown, including:

[0105] For the i-th image, ResNet-101 is used as the backbone, and FasterR-CNN pre-trained on the visual genome dataset extracts the corresponding regional features; the final regional features V = [v1, v2, ..... v n ];

[0106] A bidirectional gated recurrent neural network with d-dimensional hidden layer is used to obtain the information of the forward and reverse hidden layer states of the input text. The average value of the information of the forward and reverse hidden layer states of the text is used as its feature representation to obtain the final vocabulary feature T = [t1, t2, .....t m ];

[0107] For visual feature aggregation, the regional average feature obtained in formula (1) is used As the query, as shown in formula (2), the attention mechanism is used to learn the attention weights of the regional fragments and average features, and the global visual features are obtained by weighting them with the original features.

[0108]

[0109]

[0110] Similarly, according to formulas (3) and (4), we can obtain the aggregated global text features:

[0111]

[0112]

[0113] Among them, W vq , W vk , W vv , W tq , W tk , W tv is the learnable mapping matrix, d k is the size of the hidden layer dimension;

[0114] A multi-layer perceptron (MLP) is used to incorporate the features of these two modes and Projected into the dimensional space corresponding to the hash code, the tanh function is used as the activation function to limit the value of the vector to between [-1, +1]. The aggregated image features and text features are expressed as shown in equations (5) and (6), respectively:

[0115]

[0116]

[0117] The output vector of the hash layer is discretized into an integer hash code represented by -1 and +1 through the sign function sgn, which can be expressed as equations (7) and (8):

[0118]

[0119]

[0120] Where k represents the generated hash code b v and b t Length;

[0121] In order to make the generated hash code maintain the close relationship of the roughly heterogeneous data in the original sample space, the relationship between samples is constrained by the triple ranking loss function shown in formula (9):

[0122]

[0123] in, and represents the second highest order negative text that is the most difficult to distinguish; γ represents the marginal distance between the positive and negative labels, S r represents the cosine similarity of the embedding vector; [a] + represents max(0,a);

[0124] The image modality vector output by the hash layer is and the text modality vector The Mini-Batch K-means clustering algorithm is used to divide each feature vector into vectors with shorter lengths of integer segments. The entire high-dimensional vector space is split into M low-dimensional vector spaces. The clustering process is then performed on each subspace. The resulting clusters are called codebooks C. l , the following equations (10) and (11) use the clusters in the codebook to replace the original features to obtain an approximate refined representation:

[0125]

[0126]

[0127] in, Represent the index of the l-th subspace of image and text respectively; and Represent the quantization codes of images and texts respectively;

[0128] In order to reduce the quantization code as much as possible and continuous eigenvectors The difference between them is constrained by the quantization loss of formula (12):

[0129]

[0130] Among them, D. Represent the aggregate vectors of dataset samples, images, and texts respectively, and C represents the codebook; is the Frobenius norm.

[0131] The generated hash codes and quantization codes of the images and texts are used to perform two-stage screening to obtain the candidate set, such as Figure 3 As shown, including:

[0132] Hash rough screening: In this embodiment, based on the generated hash code, the query image hash code b is directly calculated v and the hash code of the entire text The Hamming similarity of the two models is calculated and the matching scores are arranged in descending order to obtain L rankings. Since the hash code ensures a certain semantic relevance under different modalities, only the first K1 data needs to be selected as the initial candidate set for subsequent detailed retrieval.

[0133] Quantization and detailed screening: In this embodiment, based on the generated query image quantization code The original vector corresponding to the text in the initial candidate set Perform an asymmetric similarity measurement to obtain a quantitative matching score. After sorting, follow a process similar to the hash screening in the first step to select the top K texts with higher priority in the K1 initial rankings as the final candidate set.

[0134] In this method, if Figure 4 As shown in the figure, a backbone network based on the cross-modal attention mechanism is introduced to calculate the final candidate set to obtain a more accurate fine-grained matching score; including:

[0135] Given a region feature V of a query image I in the final candidate set, n ] and the vocabulary features T of a text E = [t1, t2, .....t m ], calculate the cosine similarity matrix of all region-word pairs, that is:

[0136] s ij = cosine(v i ,t j ),i∈[1,n],j∈[1,m](13)

[0137] Among them, v i ,t j Represent the i-th image region feature and the j-th text vocabulary feature respectively;

[0138] The similarity matrix of the region-word pair is normalized, and the attention score α of the i-th region feature on the j-th text word feature is further obtained ij , where λ1 is the introduced smoothing coefficient:

[0139]

[0140]

[0141] According to the attention matrix, we get the weighted representation of the i-th region feature with respect to all the lexical features of the text. Calculate v i and The cosine similarity of is used to obtain the similarity semantic matrix of all regions and texts:

[0142]

[0143]

[0144] The similarity matrix and semantic matrix are averagely pooled in the image-to-text direction to obtain the aggregated global matching score of image I and text E:

[0145]

[0146] The matching scores are sorted from large to small, and the initial refinement result is shown in formula (19):

[0147] R(I,K)={E1,E2,...E j ,...,E K}(19)

[0148] Among them, R(I,K) is the initial ranking obtained by the cross-modal attention mechanism: I represents the query image, K represents the number of top candidate sets selected, and E j Represents a text candidate point.

[0149] In this method, bidirectional similarity re-ranking is used to fine-tune the internal ranking relationship, and finally high-performance cross-modal image-text retrieval is achieved; including:

[0150] Use the top K2 texts ranked in the initial refinement results for re-ranking;

[0151] Based on the generated representations, the similarity scores of the texts to be rearranged, i.e., the first K2 texts, with respect to the query image are calculated respectively;

[0152] The text is reordered from small to large according to the reallocated sequential index to obtain the final retrieval result.

[0153] Bidirectional similarity re-ranking, including:

[0154] Calculate the matching scores of the first K2 returned result texts of the query image I and all N images and sort them in descending order:

[0155] R(E j ,N)={I1,I2,....I N},j∈[1,K2](20)

[0156] Redistribute each text E according to formula (21) j Sequential index of:

[0157] p(E j )=s,I s =I(21)

[0158] According to formula (22), the texts are sorted from small to large using the indexes assigned to them, and the final sorting result is obtained:

[0159]

[0160] In this method, an optimization function is used to solve the parameters in the hash coding network and the cross-modal attention backbone network, and the optimization function is the adam optimizer function in Pytorch.

[0161] Table 1 is a table of comparative models of this embodiment;

[0162] Table 1

[0163]

[0164] Table 2 is a comparison table of retrieval accuracy and efficiency of the present invention;

[0165] Table 2

[0166]

[0167] By comparing the retrieval performance with the internationally leading image-text matching model, as shown in Table 2, the retrieval accuracy and efficiency after applying the AHR framework are significantly superior to those of the original backbone model.

[0168] Example 3

[0169] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the image and text retrieval method based on cross-modal semantic parsing described in Example 1 or 2 are implemented.

[0170] Example 4

[0171] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image-text retrieval method based on cross-modal semantic parsing described in Example 1 or 2.

[0172] Example 5

[0173] The image and text retrieval system based on cross-modal semantic analysis includes:

[0174] The multimodal information encoding module is configured to: understand a given image and generate regional feature encoding; understand a given text query statement and generate context-related vocabulary feature encoding; and be able to perform targeted semantic understanding and feature representation of images and texts respectively, so as to obtain short code aggregation representations of the hash and quantization spaces of the two modal data respectively.

[0175] The intra-modal feature fusion module is configured to: use a self-attention mechanism to perform intra-modal feature fusion on image and text representations;

[0176] The double filtering module is configured as follows: the hash code and quantization code generated by the aggregated features are used to calculate the cosine similarity of the image-text pairs respectively, and the top-ranked candidate sets are screened out after two rounds of sorting; the hash code and quantization code can be used to effectively screen out high-quality candidate sets from redundant original data sets through simple inner product calculations, thereby greatly reducing the amount of calculations, improving the speed of the model, and reducing memory usage.

[0177] The fine-grained calculation and re-ranking module is configured as follows: introduce a cross-modal attention mechanism to calculate a more accurate fine-grained matching score for the candidate set, use similarity re-ranking to fine-tune the internal ranking relationship, and finally achieve high-performance cross-modal image and text retrieval. Based on the backbone of the existing cross-modal attention mechanism, a more accurate matching score of the filtered candidate set is obtained. Considering the data asymmetry problem in unimodal retrieval, the bidirectional query response consistency is used to construct similarity re-ranking to fine-tune the results, which significantly improves the performance.

Claims

1. A graphic and text retrieval method based on cross-modal semantic parsing, characterized in that: include: Image representation: understand a given image and generate feature encodings of salient areas; Text representation: Understand a given text query and generate context-dependent discrete vocabulary encodings; Use self-attention mechanism to perform intra-modal feature fusion on image and text representations; The hash code and quantization code generated by the aggregated features are used to calculate the cosine similarity of the image-text pair. After two rounds of sorting, the top candidate sets are selected. The cross-modal attention mechanism is introduced to calculate the candidate sets to obtain a more accurate fine-grained matching score. The similarity re-ranking is used to fine-tune the internal ranking relationship, and finally high-performance cross-modal image-text retrieval is achieved. The learning process of image or text representation includes: For the i-th image, ResNet-101 is used as the backbone, and Faster R-CNN pre-trained on the visual genome dataset extracts the corresponding regional features; the final regional features V = [v1, v2, ..... v n ]; A bidirectional gated recurrent neural network with d-dimensional hidden layer is used to obtain the information of the forward and reverse hidden layer states of the input text. The average value of the information of the forward and reverse hidden layer states of the text is used as its feature representation to obtain the final vocabulary feature T = [t1, t2, .....t m ]; For visual feature aggregation, the regional average feature obtained in formula (1) is used As the query, as shown in formula (2), the attention mechanism is used to learn the attention weights of the regional fragments and average features, and the global visual features are obtained by weighting them with the original features. Similarly, according to formulas (3) and (4), we can obtain the aggregated global text features: Among them, W vq , W vk , W vv , W tq , W tk , W tv is the learnable mapping matrix, d k is the size of the hidden layer dimension; A multi-layer perceptron is used to incorporate the features of these two modes and Projected into the dimensional space corresponding to the hash code, the tanh function is used as the activation function to limit the value of the vector to between [-1, +1]. The aggregated image features and text features are expressed as shown in equations (5) and (6), respectively: The output vector of the hash layer is discretized into an integer hash code represented by -1 and +1 through the sign function sgn, which can be expressed as equations (7) and (8): Where k represents the generated hash code b v and b t Length; The relationship between samples is constrained by the triple ranking loss function shown in formula (9): in, and represents the second highest order negative text that is the most difficult to distinguish; γ represents the marginal distance between the positive and negative labels, S r represents the cosine similarity of the embedding vector; [a] + represents max(0,a); The image modality vector output by the hash layer is and the text modality vector The Mini-Batch K-means clustering algorithm is used to divide each feature vector into vectors with shorter lengths of integer segments. The entire high-dimensional vector space is split into M low-dimensional vector spaces. The clustering process is then performed on each subspace. The resulting clusters are called codebooks C. l , the following equations (10) and (11) use the clusters in the codebook to replace the original features to obtain an approximate refined representation: in, Represent the index of the l-th subspace of image and text respectively; and Represent the quantization codes of images and texts respectively; The constraint is performed through the quantization loss of formula (12): Among them, D. Represent the aggregate vectors of dataset samples, images, and texts respectively, and C represents the codebook; is the Frobenius norm; A backbone network based on the cross-modal attention mechanism is introduced to calculate the final candidate set to obtain a more accurate fine-grained matching score; including: Given a region feature V of a query image I in the final candidate set, n ] and the vocabulary features T of a text E = [t1, t2, .....t m ], calculate the cosine similarity matrix of all region-word pairs, that is: s ij =cosine(v i ,t j ),i∈[1,n],j∈[1,m](13) Among them, v i ,t j Represent the i-th image region feature and the j-th text vocabulary feature respectively; The similarity matrix of the region-word pair is normalized, and the attention score α of the i-th region feature on the j-th text word feature is further obtained ij , where λ1 is the introduced smoothing coefficient: According to the attention matrix, we get the weighted representation of the i-th region feature with respect to all the lexical features of the text. Calculate v i and The cosine similarity of is used to obtain the similarity semantic matrix of all regions and texts: The similarity matrix and semantic matrix are averagely pooled in the image-to-text direction to obtain the aggregated global matching score of image I and text E: The matching scores are sorted from large to small, and the initial refinement result is shown in formula (19): R(I,K)={E1,E2,...E j ,...,THE K }(19) Among them, R(I,K) is the initial ranking obtained by the cross-modal attention mechanism: I represents the query image, K represents the number of top candidate sets selected, and E j Represents a text candidate point.

2. According to claim 1, the image-text retrieval method based on cross-modal semantic parsing is characterized in that: The generated hash codes and quantization codes of the images and texts are used to perform two-stage screening to obtain the candidate set, including: Hash rough screening: Based on the generated hash code, directly calculate the query image hash code b v and the hash code of the entire text The Hamming similarity of the matching scores is sorted in descending order to obtain L rankings; the first K1 data are selected as the initial candidate set for subsequent detailed retrieval; Quantitative Detailed Screening: Generated Quantization Codes Based on Query Images The original vector corresponding to the text in the initial candidate set Perform an asymmetric similarity measurement to obtain a quantitative matching score, and after sorting, select the top K texts with higher priority in the K1 initial rankings as the final candidate set.

3. The image-text retrieval method based on cross-modal semantic parsing according to claim 1 is characterized in that: Use bidirectional similarity re-ranking to fine-tune the internal ranking relationship, and finally achieve high-performance cross-modal image and text retrieval; including: Use the top K2 texts ranked in the initial refinement results for re-ranking; Based on the generated representations, the similarity scores of the texts to be rearranged, i.e., the first K2 texts, with respect to the query image are calculated respectively; The text is reordered from small to large according to the reallocated sequential index to obtain the final retrieval result.

4. The image-text retrieval method based on cross-modal semantic parsing according to claim 1 is characterized in that: Bidirectional similarity re-ranking, including: Calculate the matching scores of the first K2 returned result texts of the query image I and all N images and sort them in descending order: R(E j ,N)={I1,I2,....I N },j∈[1,K2](20) Redistribute each text E according to formula (21) j Sequential index of: p(E j )=s,I s =I(21) According to formula (22), the texts are sorted from small to large using the indexes assigned to them, and the final sorting result is obtained:

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the image and text retrieval method based on cross-modal semantic parsing described in any one of claims 1-4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image-text retrieval method based on cross-modal semantic parsing described in any one of claims 1 to 4 are implemented.

7. The image-text retrieval system based on cross-modal semantic parsing is characterized by: include: The multimodal information encoding module is configured to: understand a given image and generate a regional feature encoding; Understand the given text query and generate context-related lexical feature encoding; The intra-modal feature fusion module is configured to: use the self-attention mechanism to perform intra-modal feature fusion on image and text representations; The double filtering module is configured to: use the hash code and quantization code generated by the aggregated features to calculate the cosine similarity of the image-text pair respectively, and select the top-ranked candidate sets through two rounds of sorting; The fine-grained calculation and re-ranking module is configured to: introduce a cross-modal attention mechanism to calculate a more accurate fine-grained matching score for the candidate set, use similarity re-ranking to fine-tune the internal ranking relationship, and finally achieve high-performance cross-modal image and text retrieval; The learning process of image or text representation includes: For the i-th image, ResNet-101 is used as the backbone, and Faster R-CNN pre-trained on the visual genome dataset extracts the corresponding regional features; the final regional features V = [v1, v2, ..... v n ]; A bidirectional gated recurrent neural network with d-dimensional hidden layer is used to obtain the information of the forward and reverse hidden layer states of the input text. The average value of the information of the forward and reverse hidden layer states of the text is used as its feature representation to obtain the final vocabulary feature T = [t1, t2, .....t m ]; For visual feature aggregation, the regional average feature obtained in formula (1) is used As the query, as shown in formula (2), the attention mechanism is used to learn the attention weights of the regional fragments and average features, and the global visual features are obtained by weighting them with the original features. Similarly, according to formulas (3) and (4), we can obtain the aggregated global text features: Among them, W vq , W vk , W vv , W tq , W tk , W tv is the learnable mapping matrix, d k is the size of the hidden layer dimension; A multi-layer perceptron is used to incorporate the features of these two modes and Projected into the dimensional space corresponding to the hash code, the tanh function is used as the activation function to limit the value of the vector to between [-1, +1]. The aggregated image features and text features are expressed as shown in equations (5) and (6), respectively: The output vector of the hash layer is discretized into an integer hash code represented by -1 and +1 through the sign function sgn, which can be expressed as equations (7) and (8): Where k represents the generated hash code b v and b t Length; The relationship between samples is constrained by the triple ranking loss function shown in formula (9): in, and represents the second highest order negative text that is the most difficult to distinguish; γ represents the marginal distance between the positive and negative labels, S r represents the cosine similarity of the embedding vector; [a] + represents max(0,a); The image modality vector output by the hash layer is and the text modality vector The Mini-Batch K-means clustering algorithm is used to divide each feature vector into vectors with shorter lengths of integer segments. The entire high-dimensional vector space is split into M low-dimensional vector spaces. The clustering process is then performed on each subspace. The resulting clusters are called codebooks C. l , the following equations (10) and (11) use the clusters in the codebook to replace the original features to obtain an approximate refined representation: in, Represent the index of the l-th subspace of image and text respectively; and Represent the quantization codes of images and texts respectively; The constraint is performed through the quantization loss of formula (12): Among them, D. Represent the aggregate vectors of dataset samples, images, and texts respectively, and C represents the codebook; is the Frobenius norm; A backbone network based on the cross-modal attention mechanism is introduced to calculate the final candidate set to obtain a more accurate fine-grained matching score; including: Given a region feature V of a query image I in the final candidate set, n ] and the vocabulary features T of a text E = [t1, t2, .....t m ], calculate the cosine similarity matrix of all region-word pairs, that is: s ij =cosine(v i ,t j ),i∈[1,n],j∈[1,m](13) Among them, v i ,t j Represent the i-th image region feature and the j-th text vocabulary feature respectively; The similarity matrix of the region-word pair is normalized, and the attention score α of the i-th region feature on the j-th text word feature is further obtained ij , where λ1 is the introduced smoothing coefficient: According to the attention matrix, we get the weighted representation of the i-th region feature with respect to all the lexical features of the text. Calculate v i and The cosine similarity of is used to obtain the similarity semantic matrix of all regions and texts: The similarity matrix and semantic matrix are averagely pooled in the image-to-text direction to obtain the aggregated global matching score of image I and text E: The matching scores are sorted from large to small, and the initial refinement result is shown in formula (19): R(I,K)={E1,E2,...E j ,...,THE K }(19) Among them, R(I,K) is the initial ranking obtained by the cross-modal attention mechanism: I represents the query image, K represents the number of top candidate sets selected, and E j Represents a text candidate point.

Images