Cross-modal image-text retrieval method and system based on semantic correlation similarity learning

Abstract

The application discloses a cross-modal image-text retrieval method based on semantic correlation similarity learning, comprising the following steps: obtaining image data and text data to obtain a training data set; based on adaptive semantic correlation similarity learning, an initial cross-modal image-text retrieval model is constructed; the initial cross-modal image-text retrieval model is trained based on a bidirectional semantic optimization ternary loss function by using the training data set to obtain a cross-modal image-text retrieval model; and the cross-modal image-text retrieval model is used to perform actual cross-modal image-text retrieval. The application further discloses a system for implementing the cross-modal image-text retrieval method based on semantic correlation similarity learning. In the model of the method, an image-text feature dynamic fusion module is used to dynamically fuse image and text features, the accuracy and precision of cross-modal retrieval are improved, and a self-adaptive semantic correlation similarity learning module is used to optimize local similarity, and the retrieval efficiency is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of image and text data processing and analysis, specifically to a cross-modal image and text retrieval method and system based on semantic association similarity learning. Background Technology

[0002] Image and text retrieval has wide applications in information retrieval within power systems, such as identifying the status of electrical equipment, recognizing instructions on operation tickets, identifying safety risks for on-site personnel, and identifying violations of operating procedures. Currently, cross-modal image and text retrieval maps features from both image and text modalities to a common space, performs similarity learning, and achieves cross-modal image and text retrieval through similarity metrics.

[0003] However, in the process of similarity learning, existing cross-modal image-text retrieval methods ignore the semantic associations between different local similarities in the public space, resulting in insufficient accuracy in the semantic similarity measurement between image and text modalities, which affects the accuracy of cross-modal image-text retrieval.

[0004] In feature extraction networks, most existing techniques map global features of images and text into a common space for similarity learning. This approach is prone to feature information loss and insufficient differentiation of key features, resulting in poor robustness of cross-modal image-text retrieval models. Summary of the Invention

[0005] One of the objectives of this invention is to provide a cross-modal image and text retrieval method based on semantic association similarity learning. This method can extract deeper, finer-grained feature representations, improves the local similarity learning method in semantic similarity learning networks, proposes local similarity regions that are independent of overall similarity, and improves the retrieval efficiency of cross-modal image and text retrieval.

[0006] The second objective of this invention is to provide a system for implementing the cross-modal image and text retrieval method based on semantic association similarity learning.

[0007] This invention provides a cross-modal image and text retrieval method based on semantic association similarity learning, comprising the following steps:

[0008] S1. Obtain image and text data to obtain the training dataset;

[0009] S2. Based on adaptive semantic association similarity learning, construct an initial cross-modal image and text retrieval model;

[0010] S3. Using the training dataset obtained in step S1, train the initial cross-modal image and text retrieval model obtained in step S2 based on the bidirectional semantic optimization ternary loss function to obtain the cross-modal image and text retrieval model;

[0011] S4. Using the cross-modal image and text retrieval model obtained in step S3, perform actual cross-modal image and text retrieval.

[0012] The initial cross-modal image-text retrieval model described in step S2 includes an image feature extraction module, a text feature extraction module, an image-text feature dynamic fusion module, an adaptive semantic association similarity learning module, a bidirectional semantic optimization ternary loss function module, and a common space;

[0013] The image feature extraction module includes a global feature extraction module, a local feature extraction module, and a self-attention mechanism fusion module. The global feature extraction module uses a multi-layer self-attention mechanism to extract image features, obtaining the global image features V. g The local feature extraction module uses a graph neural network to model local features and fuses local feature context information to obtain the local image features V. l The inputs to the global feature extraction module and the local feature extraction module are used as the inputs to the self-attention mechanism fusion module. The self-attention mechanism fusion module will process the local features V of the input image. l With image global features V g Internal similarity calculation is performed to obtain the original image features V';

[0014] The text feature extraction module uses the BERT network to extract features, obtaining the original text features U';

[0015] The image and text feature dynamic fusion module includes a first self-attention mechanism module, a second self-attention mechanism module, a first guided attention mechanism module, a second guided attention mechanism module, and a fusion module;

[0016] The first self-attention mechanism module takes the output of the image feature extraction module as input and processes the original image features V' to obtain the image features V' processed by the self-attention mechanism. SA The second self-attention mechanism module uses the output of the text feature extraction module as input, and performs internal similarity calculation on the original text features U' to obtain the text features U' processed by the self-attention mechanism. SA ';

[0017] The first guided attention mechanism module takes the outputs of the first and second self-attention mechanism modules as input. By calculating the correlation between the input image and text feature information, it guides the fusion of the two features based on the correlation to obtain the image feature V processed by the guided attention mechanism. SGA ';

[0018] The second guided attention mechanism module takes the outputs of the first and second self-attention mechanism modules as input. By calculating the correlation between the input image and text feature information, it guides the fusion of the two features based on the correlation to obtain the text feature U processed by the guided attention mechanism. SGA ';

[0019] The fusion module uses the outputs of the first and second guided attention mechanisms as input, allowing text features to correct image features while image features supplement text features, resulting in the fused image feature V. g 'and the fused text features U g ';Then the fused image features V g The original image features V are dynamically fused with the original image features V to obtain the final image features V, and the fused text features U are then combined. g The final text feature U is obtained by dynamically fusing the original text feature V with the image feature V. The image-text feature dynamic fusion module uploads the final image feature V and the final text feature U as an image-text feature pair to the public space.

[0020] The processing procedure of the self-attention mechanism fusion module is represented by the following formula:

[0021] V' = Attention(V g V l )

[0022] Where Attention is used for self-attention similarity calculation; V g For global features of the image; V l V represents local features of the image; V' represents original features of the image.

[0023] The processing procedures of the first self-attention mechanism module and the second self-attention mechanism module are represented by the following formula:

[0024] V SA '=SA(V')

[0025] U SA '=SA(U')

[0026] Among them, SA(·) represents the self-attention mechanism;

[0027] The processing procedures of the first and second guided attention mechanism modules are represented by the following formula:

[0028] V SGA '=GA(V SA ',U SA ')

[0029] USGA '=GA(U SA ',V SA ')

[0030] Among them, GA(·) is the processing of the attention guidance mechanism;

[0031] The processing procedure of the fusion module is represented by the following formula:

[0032]

[0033] Where σ is the sigmoid function; W α and W β Here, x is the weight matrix; softmax(x) is the activation function. Element-wise multiplication represents the operation of multiplying corresponding elements of two matrices: an image and a text matrix. γ1 is the dot product operation, which means that corresponding elements of the image and text are added one by one to fuse them; γ2 is the dynamic weight of the image network; γ1 and γ2 are the dynamic weights of the text network. Using γ1 and γ2, the network can dynamically focus on important content regions in the image and text and filter out some redundant feature information.

[0034] The adaptive semantic association similarity learning module is connected to the public space, which mines the semantic associations between different dimensions in the public space and performs secondary fusion on the local similarities in the public space;

[0035] The bidirectional semantic optimization ternary loss function module is connected to the adaptive semantic association similarity learning module. By training the triplet similarity score, it minimizes the distance between positive sample related images and texts and maximizes the distance between negative sample unrelated images and texts.

[0036] The adaptive semantic association similarity learning module mines dimensions with semantic dependencies based on the latent semantic relationships between different local similarities in the public space, and fuses the semantically related local similarities to obtain the overall similarity Agg(s) of the image-text pair, including the following steps:

[0037] a. Using image-text pairs in public spaces (v i ,u i The cosine distance between an image and text can be calculated using the following formula:

[0038] s i =v i ·u i

[0039] b. Based on the cosine distance between images and text in the common space obtained in step a, calculate the semantic similarity measure of images and text using the following formula:

[0040]

[0041] Where Agg(s) is the semantic similarity measure between the image and the text; Let the different feature vector dimensions represent local similarity in the common space; expanding the above equation according to the dimensional relationship, we get:

[0042]

[0043] Among them, |·| I It is a d×d matrix, where each row indicates whether there is a dependency between dimensions (1 for existence, 0 for non-existence); s i Cosine distance between images and text is used to represent local similarity across different dimensions.

[0044] c. Based on the expanded dimensionality of the image-text semantic similarity metric obtained in step b, construct the learning-to-be-learned dimension dependency-aware aggregation framework, represented by the following formula:

[0045]

[0046] Where i,j∈[1,d]; Local similarity with semantic relationships across different dimensions; |·| w It is a d×d matrix, w ij These are the weighting coefficients corresponding to the local similarity between images and text across different feature dimensions.

[0047] d. Introduce conditional probabilities in each dimension to determine whether semantic relationships exist between local similarities in different dimensions. The conditional probabilities are expressed by the following formula:

[0048]

[0049] in, This indicates the degree of semantic association of local similarities in the public space across different dimensions. The higher the conditional probability value, the stronger the semantic association in that dimension within the public space. Based on the learned conditional probabilities, a normal distribution is used to statistically fit the dependence of different dimensions on latent semantic relationships during the training process. The statistical expression of the conditional probabilities is represented by the following formula:

[0050] t i =μ i +α i ·σ i

[0051] Among them, t i μ is the conditional probability threshold calculated based on the normal distribution. i σ is the mean of the conditional probabilities; i α is the standard deviation of the conditional probability.i The conditional probability is the parameter to be learned and tuned. Then, based on the semantic complexity of different dimensions in the common space, the conditional probability threshold is adaptively adjusted to determine the proportion of selected dimensions. Different dimensions with semantic dependencies are then fused, expressed using the following formula:

[0052]

[0053] Where γ is the parameter to be learned; δ is the tan activation function; according to conditional probability... Guided by the principle of keeping values ​​greater than t i The dimension will be less than t. i The dimensions are compressed and then eliminated, as shown in the following formula:

[0054]

[0055] in, This refers to the local similarity obtained by semantically associating and fusing relevant dimensions;

[0056] e. Based on the results obtained in steps c and d, calculate the local similarity with semantic dependencies, then fuse them to obtain the overall similarity between the image and the text, expressed using the following formula:

[0057]

[0058] Agg(s)=σ(w s Agg uv (s))

[0059] Among them, Agg uv (s) represents local similarity with semantic association; denoted by , where is the conditional probability; ⊙ represents the inner product operation, which multiplies the conditional probability by the corresponding elements of the weight vector, used to calculate the weighted sum of the two vectors; w is the weight vector corresponding to the conditional probability; s is the local similarity of the image and text after semantic association; Agg(s) is the overall similarity of the image and text; σ is the sigmoid activation function; w s Let be the parameter matrix to be learned.

[0060] Step S3 specifically involves training the model based on a bidirectional semantic optimization ternary loss function, introducing an online hard negative sample mining method, and obtaining a cross-modal image and text retrieval model. The loss function is expressed using the following formula:

[0061]

[0062] Where s(I,T) is the distance between image I and text T and s(I,T)=max(Agg(s)); α is the margin parameter; (I,T) are positive samples, the related image-text pairs; and For negative samples, there are irrelevant image-text pairs.

[0063] Step S4 describes using the cross-modal image-text retrieval model obtained in step S3 to perform actual cross-modal image-text retrieval. Specifically, this involves inputting the image or text to be queried; extracting features using the cross-modal image-text retrieval model obtained in step S3; mapping the extracted feature vectors to a common space; learning semantic associations in the common space using an adaptive semantic association similarity learning module to eliminate feature vectors that do not match the query feature vectors; calculating the Euclidean distance between feature vectors using a bidirectional semantic optimization ternary loss function module; sorting the obtained Euclidean distances from smallest to largest; and returning the image or text corresponding to the feature vector with the smallest Euclidean distance as the retrieval result, thus completing the retrieval.

[0064] The present invention also provides a system for implementing the cross-modal image and text retrieval method based on semantic association similarity learning, comprising the following modules: a data acquisition module, a model building module, a model training module, and an image and text retrieval module;

[0065] The data acquisition module acquires image and text data and uploads the data to the model training module;

[0066] The model building module constructs an initial cross-modal image and text retrieval model based on adaptive semantic association similarity learning, and uploads the data to the model training module;

[0067] The model training module trains the initial cross-modal image and text retrieval model based on the received data and the bidirectional semantic optimization ternary loss function to obtain the cross-modal image and text retrieval model, and then uploads the data to the image and text retrieval module;

[0068] The image and text retrieval module performs actual cross-modal image and text retrieval based on the received data.

[0069] The beneficial effects of this invention are:

[0070] (1) In the method of the present invention, the model adopts a dual-branch network architecture in the image feature extraction module, which is used to capture the global features and local features of the image respectively, so that the model can more accurately identify the key feature regions in the image and improve the accuracy of the model.

[0071] (2) In the method of the present invention, the model adopts a guided attention mechanism in the dynamic fusion module of image and text features. During the feature fusion process, key feature information is effectively selected, redundant features are filtered out, and semantic feature alignment between image and text is achieved, thereby improving the robustness and generalization ability of the model.

[0072] In the method of this invention, the model uses conditional probability in the adaptive semantic association similarity learning module to fuse local similarities on different dimensions according to semantic association relationships, making the semantic dependency relationship of the fused local similarities stronger, the overall semantic similarity measurement of image and text more accurate, and improving the retrieval efficiency of the model. Attached Figure Description

[0073] Figure 1 This is a schematic flowchart of the method of the present invention;

[0074] Figure 2 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0075] This invention provides a cross-modal image and text retrieval method based on semantic association similarity learning, the flowchart of which is shown below. Figure 1 As shown, it includes the following steps:

[0076] S1. Obtain image and text data to obtain the training dataset;

[0077] S2. Based on adaptive semantic association similarity learning, construct an initial cross-modal image and text retrieval model;

[0078] The initial cross-modal image-text retrieval model includes an image feature extraction module, a text feature extraction module, an image-text feature dynamic fusion module, an adaptive semantic association similarity learning module, a bidirectional semantic optimization ternary loss function module, and a common space;

[0079] The image feature extraction module includes a global feature extraction module, a local feature extraction module, and a self-attention mechanism fusion module. The global feature extraction module uses a multi-layer self-attention mechanism to extract image features, obtaining the global image features V. g The local feature extraction module uses a graph neural network to model local features and fuses local feature context information to obtain the local image features V. l The inputs to the global feature extraction module and the local feature extraction module are used as the inputs to the self-attention mechanism fusion module. The self-attention mechanism fusion module will process the local features V of the input image. l With image global features V g Internal similarity calculation is performed to obtain the original image features V';

[0080] The text feature extraction module uses the BERT network to extract features, obtaining the original text features U';

[0081] The image and text feature dynamic fusion module includes a first self-attention mechanism module, a second self-attention mechanism module, a first guided attention mechanism module, a second guided attention mechanism module, and a fusion module;

[0082] The first self-attention mechanism module takes the output of the image feature extraction module as input and processes the original image features V' to obtain the image features V' processed by the self-attention mechanism. SA The second self-attention mechanism module uses the output of the text feature extraction module as input, and performs internal similarity calculation on the original text features U' to obtain the text features U' processed by the self-attention mechanism. SA ';

[0083] The first guided attention mechanism module takes the outputs of the first and second self-attention mechanism modules as input. By calculating the correlation between the input image and text feature information, it guides the fusion of the two features based on the correlation to obtain the image feature V processed by the guided attention mechanism. SGA ';

[0084] The second guided attention mechanism module takes the outputs of the first and second self-attention mechanism modules as input. By calculating the correlation between the input image and text feature information, it guides the fusion of the two features based on the correlation to obtain the text feature U processed by the guided attention mechanism. SGA ';

[0085] The fusion module uses the outputs of the first and second guided attention mechanisms as input, allowing text features to correct image features while image features supplement text features, resulting in the fused image feature V. g 'and the fused text features U g ';Then the fused image features V g The original image features V are dynamically fused with the original image features V to obtain the final image features V, and the fused text features U are then combined. g The final text feature U is obtained by dynamically fusing the original text feature V with the image feature V. The image-text feature dynamic fusion module uploads the final image feature V and the final text feature U as an image-text feature pair to the public space.

[0086] The processing procedure of the self-attention mechanism fusion module is represented by the following formula:

[0087] V' = Attention(V g V l )

[0088] Where Attention is used for self-attention similarity calculation; V g For global features of the image; V l V represents local features of the image; V' represents original features of the image.

[0089] The processing procedures of the first self-attention mechanism module and the second self-attention mechanism module are represented by the following formula:

[0090] V SA =SA(V)

[0091] U SA '=SA(U')

[0092] Among them, SA(·) represents the self-attention mechanism;

[0093] The processing procedures of the first and second guided attention mechanism modules are represented by the following formula:

[0094] V SGA '=GA(V SA ',U SA ')

[0095] U SGA '=GA(U SA ',V SA ')

[0096] Among them, GA(·) is the processing of the attention guidance mechanism;

[0097] The processing procedure of the fusion module is represented by the following formula:

[0098]

[0099]

[0100] Where σ is the sigmoid function; W α and W β γ1 and γ2 are the weight matrix; γ1 and γ2 are the dynamic weights; softmax(x) is the activation function; Element-wise multiplication represents the operation of multiplying corresponding elements of two matrices: an image and a text matrix. γ1 is the dot product operation, which means that corresponding elements of the image and text are added one by one to fuse them; γ2 is the dynamic weight of the image network; γ1 and γ2 are the dynamic weights of the text network. Using γ1 and γ2, the network can dynamically focus on important content regions in the image and text and filter out some redundant feature information.

[0101] The adaptive semantic association similarity learning module is connected to the public space, which mines the semantic associations between different dimensions in the public space and performs secondary fusion on the local similarities in the public space;

[0102] The bidirectional semantic optimization ternary loss function module is connected to the adaptive semantic association similarity learning module. By training the triplet similarity score, it minimizes the distance between positive sample related images and texts and maximizes the distance between negative sample unrelated images and texts.

[0103] The adaptive semantic association similarity learning module mines dimensions with semantic dependencies based on the latent semantic relationships between different local similarities in the public space, and fuses the semantically related local similarities to obtain the overall similarity Agg(s) of the image-text pair, including the following steps:

[0104] a. Using image-text pairs in public spaces (v i ,u i The cosine distance between an image and text can be calculated using the following formula:

[0105] s i =v i ·u i

[0106] b. Based on the cosine distance between images and text in the common space obtained in step a, calculate the semantic similarity measure of images and text using the following formula:

[0107]

[0108] Where Agg(s) is the semantic similarity measure between the image and the text; Let the different feature vector dimensions represent local similarity in the common space; expanding the above equation according to the dimensional relationship, we get:

[0109]

[0110] Among them, |·| I It is a d×d matrix, where each row indicates whether there is a dependency between dimensions (1 for existence, 0 for non-existence); s i Cosine distance between images and text is used to represent local similarity across different dimensions.

[0111] c. Based on the expanded dimensionality of the image-text semantic similarity metric obtained in step b, construct the learning-to-be-learned dimension dependency-aware aggregation framework, represented by the following formula:

[0112]

[0113] Where i,j∈[1,d]; Local similarity with semantic relationships across different dimensions; |·| w It is a d×d matrix, w ij These are the weighting coefficients corresponding to the local similarity between images and text across different feature dimensions.

[0114] d. Introduce conditional probabilities in each dimension to determine whether semantic relationships exist between local similarities in different dimensions. The conditional probabilities are expressed by the following formula:

[0115]

[0116] in, This indicates the degree of semantic association of local similarities in the public space across different dimensions. The higher the conditional probability value, the stronger the semantic association in that dimension within the public space. Based on the learned conditional probabilities, a normal distribution is used to statistically fit the dependence of different dimensions on latent semantic relationships during the training process. The statistical expression of the conditional probabilities is represented by the following formula:

[0117] t i =μ i +α i ·σ i

[0118] Among them, t i μ is the conditional probability threshold calculated based on the normal distribution. i σ is the mean of the conditional probabilities; i α is the standard deviation of the conditional probability. i The conditional probability is the parameter to be learned and tuned. Then, based on the semantic complexity of different dimensions in the common space, the conditional probability threshold is adaptively adjusted to determine the proportion of selected dimensions. Different dimensions with semantic dependencies are then fused, expressed using the following formula:

[0119]

[0120] Where γ is the parameter to be learned; δ is the tan activation function; according to conditional probability... Guided by the principle of keeping values ​​greater than t i The dimension will be less than t. i The dimensions are compressed and then eliminated, as shown in the following formula:

[0121]

[0122] in, This refers to the local similarity obtained by semantically associating and fusing relevant dimensions;

[0123] e. Based on the results obtained in steps c and d, calculate the local similarity with semantic dependencies, then fuse them to obtain the overall similarity between the image and the text, expressed using the following formula:

[0124]

[0125] Agg(s)=σ(w s Agg uv (s))

[0126] Among them, Agg uv (s) represents local similarity with semantic association; denoted by , where is the conditional probability; ⊙ represents the inner product operation, which multiplies the conditional probability by the corresponding elements of the weight vector, used to calculate the weighted sum of the two vectors; w is the weight vector corresponding to the conditional probability; s is the local similarity of the image and text after semantic association; Agg(s) is the overall similarity of the image and text; σ is the sigmoid activation function; w s Let be the parameter matrix to be learned.

[0127] S3. Using the training dataset obtained in step S1, train the initial cross-modal image-text retrieval model obtained in step S2 based on the bidirectional semantic optimization ternary loss function to obtain the cross-modal image-text retrieval model, specifically as follows:

[0128] Based on a bidirectional semantic optimization ternary loss function, an online hard negative sample mining method is introduced to train the model, resulting in a cross-modal image and text retrieval model. The loss function is expressed by the following formula:

[0129]

[0130] Where s(I,T) is the distance between image I and text T and s(I,T)=max(Agg(s)); α is the margin parameter; (I,T) are positive samples, the related image-text pairs; and For negative samples, there are irrelevant image-text pairs.

[0131] S4. Using the cross-modal image-text retrieval model obtained in step S3, perform actual cross-modal image-text retrieval, specifically as follows:

[0132] Input the image or text to be queried; extract features using the cross-modal image-text retrieval model obtained in step S3, and map the extracted feature vectors into a public space; learn semantic associations in the public space using the adaptive semantic association similarity learning module, and remove feature vectors that do not match the query feature vectors; calculate the Euclidean distance between feature vectors using the bidirectional semantic optimization ternary loss function module, sort the obtained Euclidean distances from smallest to largest, and return the image or text corresponding to the feature vector with the smallest Euclidean distance as the retrieval result, thus completing the retrieval.

[0133] This invention also provides a system for implementing the cross-modal image and text retrieval method based on semantic association similarity learning, the structural diagram of which is shown below. Figure 2 As shown, it includes the following modules: data acquisition module, model building module, model training module, and image / text retrieval module;

[0134] The data acquisition module acquires image and text data and uploads the data to the model training module;

[0135] The model building module constructs an initial cross-modal image and text retrieval model based on adaptive semantic association similarity learning, and uploads the data to the model training module;

[0136] The model training module trains the initial cross-modal image and text retrieval model based on the received data and the bidirectional semantic optimization ternary loss function to obtain the cross-modal image and text retrieval model, and then uploads the data to the image and text retrieval module;

[0137] The image and text retrieval module performs actual cross-modal image and text retrieval based on the received data.

Claims

1. A cross-modal image and text retrieval method based on semantic association similarity learning, characterized in that, Includes the following steps: S1. Obtain image and text data to obtain the training dataset; S2. Based on adaptive semantic association similarity learning, construct an initial cross-modal image and text retrieval model; S3. Using the training dataset obtained in step S1, train the initial cross-modal image and text retrieval model obtained in step S2 based on the bidirectional semantic optimization ternary loss function to obtain the cross-modal image and text retrieval model; S4. Using the cross-modal image and text retrieval model obtained in step S3, perform actual cross-modal image and text retrieval; The initial cross-modal image-text retrieval model described in step S2 includes an image feature extraction module, a text feature extraction module, an image-text feature dynamic fusion module, an adaptive semantic association similarity learning module, a bidirectional semantic optimization ternary loss function module, and a common space; The image feature extraction module includes a global feature extraction module, a local feature extraction module, and a self-attention mechanism fusion module. The global feature extraction module uses a multi-layer self-attention mechanism to extract image features, obtaining the global image features. The local feature extraction module uses a graph neural network to model local features and fuses local feature context information to obtain local image features. The inputs to the global feature extraction module and the local feature extraction module are used as the inputs to the self-attention mechanism fusion module. The self-attention mechanism fusion module integrates the local features of the input image. With global features of the image Perform internal similarity calculations to obtain the original image features. ; The text feature extraction module uses the BERT network to extract features and obtain the original text features. ; The image and text feature dynamic fusion module includes a first self-attention mechanism module, a second self-attention mechanism module, a first guided attention mechanism module, a second guided attention mechanism module, and a fusion module; The first self-attention mechanism module uses the output of the image feature extraction module as input, and processes the original features of the input image. The image features are processed using a self-attention mechanism. ; The second self-attention mechanism module uses the output of the text feature extraction module as input, and processes the original features of the input text. Internal similarity calculations are performed to obtain text features processed by the self-attention mechanism. ; The first guided attention mechanism module takes the outputs of the first and second self-attention mechanism modules as input. By calculating the correlation between the input image and text feature information, it guides the fusion of the two features based on the correlation to obtain the image features processed by the guided attention mechanism. ; The second guided attention mechanism module takes the outputs of the first and second self-attention mechanism modules as input. By calculating the correlation between the input image and text feature information, it guides the fusion of the two features based on the correlation to obtain the text features processed by the guided attention mechanism. ; The fusion module uses the outputs of the first and second guided attention mechanisms as input, allowing text features to correct image features while image features supplement text features, resulting in fused image features. and the fused text features Then the fused image features Features of the original image Dynamic fusion is performed to obtain the final image features. The merged text features Features of the original text Dynamic fusion is performed to obtain the final text features. ; The image and text feature dynamic fusion module will ultimately fuse image features. With the final text features Uploaded to the public space in the form of image-text feature pairs; The adaptive semantic association similarity learning module is connected to the public space, which mines the semantic associations between different dimensions in the public space and performs secondary fusion on the local similarities in the public space; The bidirectional semantic optimization ternary loss function module is connected to the adaptive semantic association similarity learning module. By training the triplet similarity score, it minimizes the distance between positive sample related images and texts and maximizes the distance between negative sample unrelated images and texts. The adaptive semantic association similarity learning module and the bidirectional semantic optimization ternary loss function module are also used in the model training process.

2. The cross-modal image and text retrieval method based on semantic association similarity learning according to claim 1, characterized in that, The processing procedure of the self-attention mechanism fusion module is represented by the following formula: in, For self-attention similarity calculation; For global features of the image; For local features of the image; Original features of the image; The processing procedures of the first self-attention mechanism module and the second self-attention mechanism module are represented by the following formula: in, This is handled by the self-attention mechanism; The processing procedures of the first and second guided attention mechanism modules are represented by the following formula: in, To guide attention mechanism processing; The processing procedure of the fusion module is represented by the following formula: in, for function; and This is the weight matrix; and Dynamic weights; For activation functions; Element-wise multiplication represents the operation of multiplying corresponding elements of two matrices: an image and a text matrix. This is a dot product operation, which means fusing corresponding elements of the image and text by adding them one by one. For image networks, dynamic weights are used. For the dynamic weights of the text network, utilize and It can enable networks to dynamically focus on important content areas in images and text, filtering out some redundant feature information.

3. The cross-modal image and text retrieval method based on semantic association similarity learning according to claim 1, characterized in that, The adaptive semantic association similarity learning module mines dimensions with semantic dependencies based on the latent semantic relationships between different local similarities in the public space, and fuses semantically related local similarities to obtain the overall similarity of image-text pairs. This includes the following steps: a. Using images and text in public spaces The cosine distance between an image and text can be calculated using the following formula: b. Based on the cosine distance between images and text in the common space obtained in step a, calculate the semantic similarity measure of images and text using the following formula: in, For measuring the semantic similarity between images and text; Let the different feature vector dimensions represent local similarity in the common space; expanding the above equation according to the dimensional relationship, we get: in, It is The matrix, where each row represents whether there is a dependency between dimensions, is 1 if there is a dependency and 0 if there is no dependency. Cosine distance between images and text is used to represent local similarity across different dimensions. c. Based on the expanded dimensionality of the image-text semantic similarity metric obtained in step b, construct the learning-to-be-learned dimension dependency-aware aggregation framework, represented by the following formula: in, ; Local similarity with semantic relationships across different dimensions; It is The matrix, These are the weighting coefficients corresponding to the local similarity between images and text across different feature dimensions. d. Introduce conditional probabilities in each dimension to determine whether semantic relationships exist between local similarities in different dimensions. The conditional probabilities are expressed by the following formula: in, This indicates the degree of semantic association of local similarities in the public space across different dimensions. The higher the conditional probability value, the stronger the semantic association in that dimension within the public space. Based on the learned conditional probabilities, a normal distribution is used to statistically fit the dependence of different dimensions on latent semantic relationships during the training process. The statistical expression of the conditional probabilities is represented by the following formula: in, The conditional probability threshold is calculated based on the normal distribution; The mean of the conditional probabilities; The standard deviation of the conditional probability; The conditional probability is the parameter to be learned and tuned. Then, based on the semantic complexity of different dimensions in the common space, the conditional probability threshold is adaptively adjusted to determine the proportion of selected dimensions. Different dimensions with semantic dependencies are then fused, expressed using the following formula: in, These are the parameters to be learned; The activation function is tan; based on conditional probability. The guidance, retain more than The dimension will be smaller than The dimensions are compressed and then eliminated, as shown in the following formula: in, This refers to the local similarity achieved by semantically fusing related dimensions. e. Based on the results obtained in steps c and d, calculate the local similarity with semantic dependencies, then fuse them to obtain the overall similarity between the image and the text, expressed using the following formula: in, Local similarity with semantic association; For conditional probability; This is the inner product operation, which means multiplying the corresponding elements of the conditional probability and the weight vector, and is used to calculate the weighted sum of the two vectors; This is the weight vector corresponding to the conditional probability; Local similarity after semantic association between images and text; For overall similarity between images and text; for Activation function; Let be the parameter matrix to be learned.

4. The cross-modal image and text retrieval method based on semantic association similarity learning according to claim 3, characterized in that, Step S3 specifically involves training the model based on a bidirectional semantic optimization ternary loss function, introducing an online hard negative sample mining method, and obtaining a cross-modal image and text retrieval model. The loss function is expressed using the following formula: in, For image With text The distance between and ; This refers to the margin parameter; For positive samples, the relevant image-text pairs; and For negative samples, there are irrelevant image-text pairs.

5. The cross-modal image and text retrieval method based on semantic association similarity learning according to claim 1, characterized in that, Step S4 describes using the cross-modal image-text retrieval model obtained in step S3 to perform actual cross-modal image-text retrieval. Specifically, this involves: inputting the image or text to be queried; extracting features using the cross-modal image-text retrieval model obtained in step S3, and mapping the extracted feature vectors to a common space; learning semantic associations in the common space using an adaptive semantic association similarity learning module, and eliminating feature vectors that do not match the query feature vectors; calculating the Euclidean distance between feature vectors using a bidirectional semantic optimization ternary loss function module, sorting the obtained Euclidean distances from smallest to largest, and returning the image or text corresponding to the feature vector with the smallest Euclidean distance as the retrieval result, thus completing the retrieval.

6. A system for implementing the cross-modal image and text retrieval method based on semantic association similarity learning as described in any one of claims 1 to 5, characterized in that, It includes the following modules: data acquisition module, model building module, model training module, and image / text retrieval module; The data acquisition module acquires image and text data and uploads the data to the model training module; The model building module constructs an initial cross-modal image and text retrieval model based on adaptive semantic association similarity learning, and uploads the data to the model training module; The model training module trains the initial cross-modal image and text retrieval model based on the received data and the bidirectional semantic optimization ternary loss function to obtain the cross-modal image and text retrieval model, and then uploads the data to the image and text retrieval module; The image and text retrieval module performs actual cross-modal image and text retrieval based on the received data.

Images