An image-text matching method of a position attention knowledge embedding network

Abstract

The application discloses a kind of position attention knowledge embedding network's image-text matching method, comprising:1, by the image position feature extraction network based on Fast R-CNN model obtains image feature representation;2, by the text feature extraction network based on BERT model obtains text feature representation with context information;3, construct the knowledge feature extraction network based on graph convolutional neural network extracts consensus knowledge feature representation;4, by feature fusion network obtains image knowledge feature and text knowledge feature;5, by the feature distance between positive sample and negative sample is constrained by bidirectional triplets ranking loss function, to train to obtain optimal matching model.The application obtains image feature containing position information and text feature containing context information by position feature attention module and BERT pre-training model respectively, greatly improves the utilization efficiency of consensus knowledge, thereby greatly improves the accuracy of image-text matching.

Description

TECHNICAL FIELD

[0001] The present application relates to a position attention module, a graph convolutional neural network, and a method for solving a graph-text matching problem, and belongs to the field of natural language processing (NLP), in particular to a graph-text matching method based on a position attention knowledge embedding network of a consensus knowledge graph. BACKGROUND

[0002] The goal of cross-modal image-text matching is to retrieve the best text that describes the image for a given image, or to retrieve the image that describes a given text snippet. The main difficulty of cross-modal image-text matching is that the features of different modalities are located in different feature spaces, and they are heterogeneous in the underlying data structure and related in the top-level semantics. For example, text encoding features and image encoding features representing the same topic are in completely different feature spaces. Although they both represent the same topic, their feature vectors are completely different.

[0003] With the development of graph neural networks and knowledge graphs, existing methods introduce external knowledge, which can increase an additional consensus knowledge feature space outside the image feature space and the text feature space. By performing auxiliary matching in a unified knowledge space, the semantic difference between the image and the text can be overcome to a certain extent, and the matching effect can be improved.

[0004] However, the existing method simply determines the existence of objects in the picture through the knowledge graph, which may cause images that do not match the text to also match the text successfully, resulting in a decline in the effect of image-text matching. Since the model simply fuses the knowledge features and the image-text features, the utilization rate of the knowledge features is reduced, which leads to poor final matching effect. SUMMARY

[0005] To overcome the problems of the prior art, the present application provides a position attention knowledge embedding network for image-text matching, which can fuse image target position features and extracted image features to enhance the expression of the region, use a BERT model to obtain text features containing context information, and strengthen the guiding effect of consensus knowledge on the relationship between objects that should exist, so as to improve the efficiency of the network in utilizing consensus knowledge and thus improve the accuracy of image-text matching.

[0006] To achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0007] The image-text matching method of the position attention knowledge embedding network of the present application has the following characteristics:

[0008] Step 1: Build an image position feature extraction network based on a FastR-CNN model, which includes an image feature extraction module, a position feature extraction module, and a self-attention processing module.

[0009] Step 1.1: The image feature extraction module acquires the image dataset and its corresponding text dataset, and uses the Fast R-CNN model to extract features from any image pic, obtaining a set of features V = {v1,…,v...} for image pic. i ,…,v q}, v i Let v represent the i-th feature of image pic. i ∈R d ; d represents the dimension of the feature, q represents the total number of features; i = 1, ..., q;

[0010] Step 1.2: The location feature extraction module first uses the Fast R-CNN model to process any image pic in the image dataset to obtain q bounding boxes;

[0011] Then the image pic is divided into K×K equal blocks, and the position of each block is determined by the index e∈[1,K]. 2 ] indicates that the K segments will be divided. 2 Each block is used as the positional vocabulary B, and any e-th block b is represented by a one-hot vector. e b e ∈B;

[0012] Calculate the number of intersecting pixels between the i-th object box and the e-th block. The location vocabulary B is then sorted in descending order based on the number of intersecting pixels, and the L highest-ranking blocks are selected for calculating v. i Position representation Thus, a set of positional features P = {p1, ..., p2} extracted from the image pic by the positional feature extraction module is obtained. i , ..., p q}, p i p represents the i-th positional feature of image pic. i ∈R τ τ represents the dimension of the location feature;

[0013] Step 1.3: Concatenate the feature V of image pic and the location feature P together to obtain a region-level image feature set V containing location information. P ={(v1, p1), ..., (v i p i ), ..., (v q p q )},(v i p i ) represents the i-th region-level image feature, (v i p i )∈R d V P ∈Rq×d ;

[0014] Step 1.4, the region-level image feature V P = {(v1, p1), …, (v i , p i ), …, (v q , p q )} is taken as the key K and the value V in the attention mechanism, the global image feature is taken as the query Q in the attention mechanism, so that the region-level image feature set V p is merged into a single image feature v I ∈R d through self-attention operation;

[0015] Step 2, a text feature extraction network based on the BERT model is built, including a text feature extraction module and a text self-attention processing module;

[0016] Step 2.1, the text feature extraction module uses the BERT model to perform word-level feature extraction on the text corresponding to the image pic, to obtain a group of word-level features T = {t1, …, t j , …, t s}, t j represents the jth word-level feature, t j ∈R d , s represents the total number of word-level features, and j = 1, …, s;

[0017] Step 2.2, the text self-attention module takes the word-level feature T = {t1, …, t j , …, t s} as the key K and the value V in the attention mechanism, takes the global word-level feature as the query Q in the attention mechanism, so that the word-level feature T is merged into a single sentence-level feature t I ∈R d through self-attention operation;

[0018] Step 3, a knowledge feature extraction network based on a graph convolutional neural network is built, including a knowledge concept extraction module, a concept correlation graph construction module, and a knowledge feature representation module;

[0019] Step 3.1, the knowledge concept extraction module extracts knowledge concepts from the image caption corpus corresponding to the text data set, and uses the Glove method to vectorize all the knowledge concepts, to obtain instantiated concept representations C = {c1, …, c k , …, c n}, c k represents the kth concept representation, n represents the total number of concept representations, and ck ∈R d k = 1, ..., n;

[0020] Step 3.2: The concept association graph construction module constructs the concept association graph G using equation (1):

[0021]

[0022] In equation (1), O αβ The α-th concept is represented by c. α And the βth concept represents c β The number of times a text appears simultaneously, G=1 represents c. α and c β The co-occurrence of c, G=0 indicates that c α and c β The missing value, θ represents the threshold for the number of occurrences; c α ∈C, c β ∈C;

[0023] Step 3.3: The knowledge feature representation module uses a graph convolutional neural network to learn the concept representation C and the concept-related graph G, thereby obtaining the knowledge feature representation Z = {z1, ..., z2}. k ,…,z n}, z k Let z represent the k-th knowledge feature. k ∈R d k = 1, ..., n;

[0024] Step 4: Build a feature fusion network, including: an image feature fusion module, a text feature fusion module, and an image-text fusion module;

[0025] Step 4.1: The image feature fusion module obtains the image knowledge feature v using equation (2). C :

[0026]

[0027] In equation (2), Represents the k-th knowledge feature z k The importance score is obtained from equation (3);

[0028]

[0029] In equation (3), W v ∈R d×d It is the parameter matrix to be learned. Represents the k-th knowledge feature z k The transpose of λ controls the smoothness of the softmax function;

[0030] Step 4.2, the text feature fusion module obtains the text knowledge feature t by formula (4) C :

[0031]

[0032] In formula (4), represents the importance score of the kth knowledge feature z k , and is obtained by formula (5);

[0033]

[0034] In formula (5), W t ∈R d×d is the parameter matrix to be learned;

[0035] Step 4.3, the image-text fusion module combines the image feature v I and the image knowledge feature v C respectively by formula (6) to obtain the comprehensive representation image information v U ; combines the sentence-level feature t I and the text knowledge feature t C by formula (7) to obtain the comprehensive representation text information t U ;

[0036] v U =αv C +(1-α)v I (6)

[0037] t U =αt C +(1-α)t I (7)

[0038] In formula (6) and formula (7), alpha is a parameter for determining the relative proportion between the two types of features;

[0039] Step 5, constructing an image-text matching network composed of an image position feature extraction network, a text feature extraction network, a knowledge feature extraction network, and a feature fusion network, and establishing a loss function for training the network;

[0040] Step 5.1, constructing the loss functions Loss(v U , t U ), Loss(v C , t C ) and Loss(v I , t I ) of image-text fusion, image-text knowledge and image-text feature respectively by formula (8);

[0041]

[0042] In formula (8), (v, t) represents any one of (v C , t C ), (v I , t I ), (v U , t U ) matched image-text pair; δ is a predefined margin parameter, and s() is a cosine distance function; (v - , t) and (v, t - ) represent any one of (v C , t C ), (v I , t I ), (v U , t U ) unmatched image-text pair;

[0043] Step 5.2, constructing the total loss function L of the network by using formula (9);

[0044] L=Loss(v U , t U )+Loss(v C , t C )+Loss(v I , t I ) (9)

[0045] Step 5.3, inputting the image data set and its corresponding text data set into the image-text matching network, and constantly optimizing the total loss function L by the Adam optimizer to adjust the network parameters, so as to obtain the optimal image-text matching network for matching the image and the text.

[0046] The electronic device comprises a memory and a processor, and is characterized in that the memory is used for storing a program supporting the processor to execute the image-text matching method, and the processor is configured to execute the program stored in the memory.

[0047] The computer readable storage medium stores a computer program, and the computer program is characterized in that when the computer program is executed by a processor, the steps of the image-text matching method are executed.

[0048] Compared with the prior art, the present application has the following advantages:

[0049] 1、The present application uses an external corpus to obtain concepts, uses a scene knowledge graph to capture co-occurrence concepts as common knowledge, learns common sense perception for further expression of common-sense knowledge concept representation, and then performs image-text matching based on common knowledge, thereby strengthening the semantic relationship between the image and the text.

[0050] 2、The application proposes a location feature extraction module (LFE), which infers the relative position of the region in the image by dividing the image into blocks, and then proposes a location attention mechanism to model the relationship between the image region and the block, and generates valuable location features, the obtained location features focus on the significant region in the image, which will strengthen the guiding consensus knowledge to act on the relationship between the objects that should exist in the image, thereby improving the efficiency of the network using consensus knowledge.

[0051] 3、The application uses the BERT model which can better describe the meaning of the word to extract the text features, the model focuses on exploring the relationship between words, can integrate left and right context information and produce context-aware word representation. The consensus knowledge is fused with the text features extracted from the BERT model, and the consensus knowledge is guided to act on each word, which improves the effect of consensus knowledge in text, thereby improving the accuracy of image-text matching. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The structural diagram of the location attention knowledge embedding network of the application;

[0053] Figure 2 The flowchart of the location feature extraction module (LFE) of the application;

[0054] Figure 3 The structural diagram of the graph convolutional neural network (GCN) of the application;

[0055] Figure 4 The structural diagram of the self-attention mechanism of the application;

[0056] Figure 5 The visualization result diagram of the text retrieval of the application;

[0057] Figure 6 The visualization result diagram of the image retrieval of the application. DETAILED DESCRIPTION

[0058] In this embodiment, an image-text matching method of a location attention knowledge embedding network proposes a LAKE model, as shown in Figure 1 First, the image features are extracted by the image location feature extraction network based on the Fast R-CNN model, which includes an image feature extraction module and a location feature extraction module in turn, as shown in Figure 2As shown, the method includes a self-attention processing module; then, text features are extracted through a BERT-based text feature extraction network, including a text feature extraction module and a text self-attention processing module; next, knowledge features are extracted through a graph convolutional neural network-based knowledge feature extraction network, including a knowledge concept extraction module, a concept association graph construction module, and a knowledge feature representation module; then, image knowledge features and text knowledge features are obtained through a feature fusion network, including an image feature fusion module, a text feature fusion module, and an image-text fusion module; finally, the model is trained using a total loss function to obtain the final result. Specifically, the method includes the following steps:

[0059] Step 1: Build an image location feature extraction network based on the Fast R-CNN model, including: an image feature extraction module, a location feature extraction module, and a self-attention processing module;

[0060] Step 1.1: The image feature extraction module obtains the image dataset and its corresponding text dataset, and uses the Fast R-CNN model to extract features from any image pic, obtaining a set of features V = {v1, ..., v1} for image pic. i , ..., v q}, v i Let v represent the i-th feature of image pic. i ∈R d ; d represents the dimension of the feature, q represents the total number of features; i = 1, ..., q;

[0061] In this embodiment, feature extraction is performed on images from two datasets: Flickr30k and MSCOCO. Flickr30k is an image caption dataset containing 31,783 images, each annotated with five sentences. This dataset is divided into 29,783 training images, 1,000 validation images, and 1,000 test images. MSCOCO is another image caption dataset, containing a total of 123,287 images, each annotated with approximately five sentence-level descriptions. This dataset is divided into 113,287 training images, 1,000 validation images, and 5,000 test images.

[0062] Step 1.2: The location feature extraction module first uses the Fast R-CNN model to process any image pic in the image dataset to obtain q bounding boxes;

[0063] Then the image pic is divided into K×K equal blocks, and the position of each block is determined by the index e∈[1,K]. 2 ] indicates that the K segments will be divided. 2 Each block is used as the positional vocabulary B, and any e-th block b is represented by a one-hot vector. e b e∈B; Calculate the number of intersecting pixels between the i-th object box and the e-th block using equation (1).

[0064]

[0065] Among them, v i b is the number of pixels in the i-th object frame. e It is the number of pixels in the e-th block. This represents the number of intersecting pixels between the i-th object frame and the e-th block. The position vocabulary B is then sorted in descending order according to the number of intersecting pixels, and the L highest-ranking blocks are selected using equation (11).

[0066]

[0067] To obtain a more accurate description of the location, this invention embeds the block index into a dense representation. The segmented K... 2 Each block is a location vocabulary B. i ∈B is represented by a one-hot vector, which represents the index in the positional vocabulary. Next, the invention applies an embedding layer to project the one-hot representation onto a τ-dimensional vector. This is used to compute v. i Position representation Thus, a set of location features P = {p1, ..., p2} extracted from the image pic by the location feature extraction module is obtained. i , ..., p q}, p i p represents the i-th positional feature of image pic. i ∈R τ τ represents the dimension of the location feature;

[0068] Step 1.3: Concatenate the feature V of image pic and the location feature P together to obtain a region-level image feature set V containing location information. P ={(v1, p1), ..., (v i p i ), ..., (v q p q )},(v i p i ) represents the i-th region-level image feature, (v i p i )∈R d V P ∈R q×d ;

[0069] Step 1.4, Self-attention processing module, as follows Figure 4 As shown, the regional image features V P ={(v1, p1), ..., (vi p i ), ..., (v q p q )} as the key K and value V in the attention mechanism, which will integrate global image features As a query Q in the attention mechanism, the region-level image feature set V is then processed through self-attention operations. p Merged into a single image feature v I ∈R d ;

[0070] Step 2: Build a text feature extraction network based on the BERT model, including: a text feature extraction module and a text self-attention processing module;

[0071] Step 2.1: The text feature extraction module uses the BERT model to extract word-level features from the text corresponding to the image pic, obtaining a set of word-level features T = {t1, ..., t2}. j , ..., t s}, t j Let t represent the j-th word-level feature. j ∈R d s represents the total number of word-level features, j = 1, ..., s;

[0072] Step 2.2: The text self-attention module will process the word-level features T = {t1, ..., t2}. j , ..., t s} as the key K and value V in the attention mechanism, global word-level features The query Q serves as the focus of the attention mechanism, thereby merging word-level features T into a single sentence-level feature t through self-attention operations. I ∈R d ;

[0073] Step 3: Construct a knowledge feature extraction network based on graph convolutional neural networks, including: a knowledge concept extraction module, a concept association graph construction module, and a knowledge feature representation module;

[0074] Step 3.1: In this example, knowledge concepts are extracted from the image caption corpus of natural sentences in the dataset; word frequencies are calculated using a probability-based method; and then the concepts with the top n frequencies are selected as nodes in the knowledge graph. Knowledge concepts are divided into three categories: objects; movements; and attributes. In most cases, nouns are classified as objects, verbs as movements, and adjectives as attributes. In this module, part-of-speech tagging (POS) is used to classify knowledge concepts and determine their category labels; and these concepts are vectorized using the GloVe technique, represented as C = {c1, ..., c...} i c n}, c i ∈Rd Let i = 1, ..., n. And obtain the instantiated conceptual representation C = {c1, ..., cn}. k c n}, c k Let c represent the k-th concept representation, n represent the total number of concept representations, and c represent the k-th concept representation. k ∈R d k = 1, ..., n;

[0075] Step 3.2: The concept association graph construction module constructs the concept association graph G using equation (3):

[0076]

[0077] In equation (3), O αβ The α-th concept is represented by c. α And the βth concept represents c β The number of times a text appears simultaneously, G=1 represents c. α and c β The co-occurrence of c, G=0 indicates that c α and c β The missing value, θ represents the threshold for the number of occurrences; c α ∈C, c β ∈C;

[0078] Step 3.3: The knowledge feature representation module uses a graph convolutional neural network (GCN) to learn concept representations, such as... Figure 3 This indicates that it introduces higher-order neighborhood information between concepts to model their interdependencies. Given an instantiated concept representation C and a concept correlation graph G, the embedding features of the l-th layer are calculated using equation (4):

[0079]

[0080] in, W represents a normalized symmetric matrix. (0) ∈R d×d Let represent the learnable parameter matrix. The embedding features of the second layer are calculated using equation (5):

[0081]

[0082] in, W (1) ∈R d×d This represents the learnable parameter matrix. The output H of the second layer. (2) This is the final knowledge feature representation Z = {z1, ..., z2}. k ,…,z n}, z k Let z represent the k-th knowledge feature. k ∈R d, k = 1, …, n;

[0083] Step 4, build a feature fusion network, including: an image feature fusion module, a text feature fusion module and a graph-text fusion module;

[0084] Step 4.1, the image feature fusion module obtains image knowledge features v C by formula (6);

[0085]

[0086] In formula (6), represents the importance score of the kth knowledge feature z k , and is obtained by formula (7);

[0087]

[0088] In formula (7), W v ∈R d×d is a parameter matrix to be learned, represents the transpose of the kth knowledge feature z k , and λ controls the smoothness of the Softmax function;

[0089] Step 4.2, the text feature fusion module obtains text knowledge features t C by formula (8);

[0090]

[0091] In formula (8), represents the importance score of the kth knowledge feature z k , and is obtained by formula (9);

[0092]

[0093] In formula (9), W t ∈R d×d is a parameter matrix to be learned;

[0094] Step 4.3, the graph-text fusion module combines the image features v I and the image knowledge features v C by formula (10) respectively to obtain comprehensive representation image information v U ; combines the sentence-level features t I and the text knowledge features t C by formula (11) to obtain comprehensive representation text information t U ;

[0095] v U = αv C + (1-α)v I (10)

[0096] t U =αt C +(1-α)t I (11)

[0097] In equations (10) and (11), α is a parameter that determines the relative proportion between the two types of features;

[0098] Step 5: Construct an image-text matching network consisting of an image location feature extraction network, a text feature extraction network, a knowledge feature extraction network, and a feature fusion network, and establish a loss function for training the network:

[0099] Step 5.1: Construct the loss function Loss(v) for image-text fusion using equation (12). U , t U Loss function for text and image knowledge (v) C , t C Loss function for image and text features (v) I , t I );

[0100]

[0101] In equation (12), (v, t) represents (v C , t C ), (v I , t I ), (v U , t U Any matching image-text pair in ); δ is a predefined margin parameter, s() is the cosine distance function; (v - (v,t) and (v,t) - ) indicates (v C , t C ), (v I , t I ), (v U , t U Any set of mismatched image-text pairs in )

[0102] Step 5.2: Construct the total loss function L of the network using equation (13);

[0103] L = Loss(v U , t U )+Loss(v C , t C )+Loss(v I , t I (13)

[0104] Step 5.3, input the image data set and its corresponding text data set into the image-text matching network, and constantly optimize the total loss function L by the Adam optimizer to adjust the network parameters, so as to obtain the optimal image-text matching network for matching the image and the text.

[0105] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0106] In this embodiment, a computer readable storage medium stores a computer program on the computer readable storage medium, and the computer program is run by a processor to execute the steps of the above method.

[0107] To further verify the effectiveness of the present application, in Figure 5 , the present application and the baseline model are compared on the Flickr30K dataset, and the text retrieval results are shown. For each picture, the present application inputs the top 3 sentences sorted from top to bottom. The sentences marked with a green check √ indicate correct matching, and the sentences marked with a red cross × indicate incorrect matching. It can be seen that the present application model can retrieve more correct sentences than the baseline model, and the correct results are ranked first. In Figure 6 , the model of the present application and the baseline model are compared on the Flickr30K dataset, and the image retrieval results are shown. For each sentence, the present application inputs the top 3 picture sentences sorted from left to right. The green box indicates correct matching, and the red box indicates incorrect matching. It can be seen that the present application model can rank the correct results higher than the baseline model.

[0108] In summary, the present application can strengthen the guiding consensus knowledge to act on the relationship between the objects existing in the image and the text by mining the position relationship of the objects in the image and the information of the text context, and improve the accuracy of image-text matching by improving the efficiency of the network using consensus knowledge.

Claims

1. A method for image-text matching using a location attention knowledge embedding network, characterized in that, Includes the following steps: Step 1: Build an image location feature extraction network based on the Fast R-CNN model, including: an image feature extraction module, a location feature extraction module, and a self-attention processing module; Step 1.1: The image feature extraction module acquires the image dataset and its corresponding text dataset, and uses the Fast R-CNN model to extract features from any one of the images. pic Perform feature extraction to obtain the image pic A set of features , Representing an image pic The i One characteristic, ; d Dimensions representing features q Represents the total number of features; ; Step 1.2: The location feature extraction module first uses the Fast R-CNN model to extract the location feature from any image in the image dataset. pic Process and obtain One object box; Then the image pic Divide the block into K×K equal parts, with the position of each block determined by an index. [1, ] indicates that the split Each block is used as the positional vocabulary B, and any e-th block is represented by a one-hot vector. , B; Calculate the first The first object box and the first Number of intersecting pixels between blocks The location vocabulary B is sorted in descending order according to the number of intersecting pixels, and the L highest-ranking blocks are selected for calculation. Position representation Thus, the location feature extraction module obtains the image... pic A set of extracted location features , Representing an image pic The i Location features, ; Dimensions representing location features; Step 1.3: Transfer the image pic The feature V and the location feature P are concatenated together to obtain a region-level image feature set with location information. , Indicates the first i Regional-level image features, , ; Step 1.4: The self-attention processing module processes region-level image features. As the key K and value V in the attention mechanism, global image features are... = As a query Q in the attention mechanism, the region-level image feature set is then processed through self-attention operations. Merged into a single image feature ; Step 2: Build a text feature extraction network based on the BERT model, including: a text feature extraction module and a text self-attention processing module; Step 2.1: The text feature extraction module uses the BERT model to process the image. pic Word-level features are extracted from the corresponding text to obtain a set of word-level features. , Indicates the first j Word-level features , This represents the total number of word-level features. ; Step 2.2: The text self-attention processing module processes word-level features. As the key K and value V in the attention mechanism, global word-level features are... = The query Q in the attention mechanism is used to merge word-level features T into a single sentence-level feature through self-attention operations. ; Step 3: Construct a knowledge feature extraction network based on graph convolutional neural networks, including: a knowledge concept extraction module, a concept association graph construction module, and a knowledge feature representation module; Step 3.1: The knowledge concept extraction module extracts knowledge concepts from the image caption corpus corresponding to the text dataset, and uses the GloVe method to vectorize all knowledge concepts to obtain instantiated concept representations. , This represents the k-th concept. This represents the total number of concepts represented. , ; Step 3.2: The concept association graph construction module constructs the concept association graph using equation (1). : (1) In equation (1), It is the αth concept representation and the βth concept representation The number of times a text appears simultaneously, G=1 indicates that... and Co-occurrence, G=0 indicates and The absence of A threshold indicating the number of occurrences; , ; Step 3.3: The knowledge feature representation module uses a graph convolutional neural network to learn the concept representation C and the concept-related graph G, thereby obtaining the knowledge feature representation. , This represents the k-th knowledge feature. , ; Step 4: Build a feature fusion network, including: an image feature fusion module, a text feature fusion module, and an image-text fusion module; Step 4.1: The image feature fusion module obtains image knowledge features through equation (2). : (2) In equation (2), Represents the k-th knowledge feature The importance score is obtained from equation (3); (3) In equation (3), It is the parameter matrix to be learned. Represents the k-th knowledge feature transpose, Controlling the smoothness of the softmax function; Step 4.2: The text feature fusion module obtains text knowledge features through equation (4). : (4) In equation (4), Represents the k-th knowledge feature The importance score is obtained from equation (5); (5) In equation (5), It is the parameter matrix to be learned; Step 4.3: The image-text fusion module uses equation (6) to combine image features. Image knowledge features Combined to obtain a comprehensive representation of image information Sentence-level features are obtained through formula (7). With textual knowledge features Combined, to obtain a comprehensive representation of textual information. ; (6) (7) In equations (6) and (7), It is a parameter that determines the relative proportion between two types of features; Step 5: Construct an image-text matching network consisting of an image location feature extraction network, a text feature extraction network, a knowledge feature extraction network, and a feature fusion network, and establish a loss function for training the network: Step 5.1: Construct the loss function for image-text fusion using equation (8). Loss function of text and image knowledge Loss function for image and text features ; (8) In equation (8), express , , Any matching image-text pair in the dataset; It is a predefined margin parameter. It is a cosine distance function; and express , , Any set of mismatched image-text pairs; Step 5.2: Construct the total loss function L of the network using equation (9); (9) Step 5.3: Input the image dataset and its corresponding text dataset into the image-text matching network, and continuously optimize the total loss function L through the Adam optimizer to adjust the network parameters, thereby obtaining the optimal image-text matching network for matching images and text.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the image-text matching method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the image-text matching method of claim 1.

Images