Unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion

Abstract

The present invention discloses an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion, comprising: extracting entity image features and entity text features from an entity dataset; obtaining an entity public semantic space based on entity image features and entity text features through negative sample selection and intra-modal neighbor relationship constraints; obtaining a high-confidence image-text pseudo-pairing set based on the entity public semantic space through a pseudo-pairing generation and screening method; capturing and aligning relationship information based on the high-confidence image-text pseudo-pairing set through a relationship fusion module to obtain a fused public semantic space; calculating the similarity between images and texts based on the fused public semantic space, sorting according to the similarity, and outputting cross-modal retrieval results. Compared with other unpaired cross-modal retrieval methods, the present invention has significantly improved performance. The learned space can accurately express data from a deeper level, and the overall method has better interpretability.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and natural language generation, and in particular to an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion. Background Art

[0002] With the development of artificial intelligence technology and the explosive growth of multimedia data in daily life, cross-modal retrieval has become an important research topic. Cross-modal retrieval aims to explore the connections between data of different modalities, and retrieves data with similar semantics in another modal database by inputting data of one modality. There is a semantic gap between data of different modalities, and how to accurately measure the semantic similarity of cross-modal data has become one of the most important challenges of cross-modal retrieval. As a multimodal cross-topic, cross-modal retrieval technology promotes the model's understanding of multimodal information, provides matching functions for various types of data, and provides new ideas and technologies for the development of other modal tasks. It has important theoretical research value. At the same time, cross-modal retrieval has broad application prospects. For example, information retrieval, product retrieval on e-commerce platforms, and personalized recommendations on social networking platforms can all be applied to cross-modal retrieval technology.

[0003] Current mainstream cross-modal retrieval methods are trained on large-scale paired annotated datasets. By aligning cross-modal data and constructing a common semantic space, they aim to bridge the cross-modal semantic gap. However, in reality, although sufficient data can be collected from the internet, labeling this data with paired information requires a significant amount of manpower and time resources. Therefore, some work has investigated cross-modal retrieval in the absence of paired information. Current unpaired cross-modal retrieval methods guide model training by introducing external general databases. However, most of these methods replace text features with image features, resulting in a loss of information in the text modality. In retrieval tasks, each modality often has its own information that promotes understanding. This lack of information can lead to biased retrieval results. Secondly, these methods treat each region in the data in isolation, ignoring the relationships between regions. The relative positions of different regions in the image or text reveal the degree of association between regions. The lack of relationship information can also lead to inaccurate retrieval results. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion. The method introduces a general entity dataset, preprocesses the dataset to construct entity image and entity text pairs, and constructs an entity public semantic space by aligning the entity data with the neighbor relationships within the constrained modality. In this space, high-confidence image-text pseudo-pairings are generated and screened for the unpaired image-text dataset. The relationship information in the data is captured through a distance-weighted relationship fusion module, and the final fused public semantic space is obtained by fine-tuning through alignment, which improves the shortcomings of the existing methods and realizes reliable cross-modal retrieval.

[0005] To achieve the above objectives, the present invention provides a technical solution: an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion, comprising the following steps:

[0006] 1) Extract entity image features and entity text features from the entity dataset;

[0007] 2) Based on entity image features and entity text features, the distance between features is calculated. The correct negative sample is selected for each feature based on the distance relationship. Through the method of embedding space alignment and intra-modal neighbor relationship constraints, the negative sample is moved away from the feature to obtain the entity common semantic space. Among them, the intra-modal neighbor relationship constraint first calculates the neighbors of the feature in the unimodal space, and then constrains the distance relationship between the neighbors before and after alignment to minimize the change, thereby avoiding the damage of the unimodal neighbor structure during the alignment process.

[0008] 3) Based on the entity public semantic space, a pseudo-pairing generation method is used to construct pseudo-pairings from an unpaired image-text dataset. A pseudo-pairing screening method is used to obtain a high-confidence image-text pseudo-pairing set. The unpaired image-text dataset is a dataset lacking pairing information annotations given by the task. The pseudo-pairing generation method selects the samples with the highest similarity based on the similarity ranking of the image and text in the entity public semantic space to construct pseudo-pairings. The pseudo-pairing screening method samples the difference between the highest and second-highest similarity based on the similarity ranking of the image and text in the entity public semantic space. This difference is called the advantage score, which is used as an indicator to measure the reliability of the pseudo-pairing. The advantage score is fitted using a two-component Gaussian mixture model, and a probability density function is obtained by optimization based on the expectation-maximization algorithm. The probability density function is used to calculate and screen pseudo-pairings to obtain a high-confidence image-text pseudo-pairing set.

[0009] 4) Based on a set of high-confidence pseudo-image-text pairs, the Faster R-CNN entity recognition network is used to segment the images in the set into multiple entity images. The text in the set is segmented into multiple entity texts using a word-by-word segmentation method. A relational fusion module is then used to extract fused features containing information relationships between the entity images and entity texts. The fused features are then aligned to obtain a fused common semantic space. The relational fusion module is a similarity-weighted feature fusion method that first calculates weights based on the similarity between the entity images and entity texts, and then uses the weights to exchange information between the entities.

[0010] 5) Embed cross-modal data into the fused common semantic space to calculate cross-modal similarity, that is, calculate the similarity between images and texts based on the fused common semantic space, sort them according to the similarity, and select the top n instances as the final cross-modal retrieval results.

[0011] Furthermore, in step 1), the convolutional neural network ResNet50 is used to extract entity image features, and the time recurrent neural network LSTM is used to extract entity text features. The specific process is: given an entity dataset, each image in the entity dataset is scaled to a size of 256×256, and then the entity image features are extracted by the convolutional neural network ResNet50 to obtain the entity image feature set. Total N E samples, v Ei is the i-th entity image feature; extract the text label corresponding to the image from the entity dataset, encode the text label into a word vector through the word embedding method, and input the word vector into the time recurrent neural network LSTM to extract the entity text feature, and obtain the entity text feature set Total N E samples, t Ei is the text feature of the i-th entity.

[0012] Furthermore, in step 2), an image unimodal encoder composed of a multi-layer neural network is used. Embed the entity image features into the image unimodal space to obtain the image unimodal features A text unimodal encoder composed of a multi-layer neural network Embed entity text features into text unimodal space to obtain text unimodal features Then, negative samples are selected based on the distance relationship between the image unimodal features and the text unimodal features;

[0013] for The negative sample is obtained by the following calculation method

[0014]

[0015]

[0016] Where, and is the text unimodal feature, agrmin is the function of the independent variable when taking the minimum value, j is the independent variable of the argmin function, p represents the pth sample in the data set, ||·|| is the L2 norm, ε is the distance boundary parameter, is with The unimodal features of the text corresponding to the subscript;

[0017] by As an anchor point, the corresponding text unimodal features As a positive sample, Get triples as negative samples

[0018] for The negative sample is obtained by the following calculation method

[0019]

[0020] Where, and is the image unimodal feature, q represents the qth sample in the dataset, is with The subscript corresponds to the unimodal feature of the image;

[0021] by As an anchor point, the corresponding image unimodal feature As a positive sample, Get triples as negative samples

[0022] Through an image cross-modal encoder composed of a multi-layer neural network With the text cross-modal encoder f t C The image unimodal features and text unimodal features in the triplet are embedded into the common subspace respectively to obtain the cross-modal feature expression of the triplet in the common subspace. and and They are and The corresponding cross-modal feature expression is then expressed through the entity feature alignment loss L based on the triplet loss entity The distance between the anchor point and the positive sample is shortened, and the distance between the anchor point and the negative sample is increased. The calculation method is as follows:

[0023]

[0024] Where S E (·) is the cross-modal feature similarity metric function, α is the marginal hyperparameter, and in [·] + Negative values ​​are set to 0, and positive values ​​remain the original value;

[0025] Then, the intra-modal neighbor relationship constraint is added based on the distance relationship to avoid the entity feature alignment loss L entity Destroy the potential single-mode neighbor structure, take τ as the distance boundary parameter, and calculate it according to the following formula Neighbor sample set V Ei+ and non-neighbor sample set V Ei- as well as Neighbor sample set T Ei+ With non-neighbor sample set T Ei- :

[0026]

[0027] pass With f t C Will And their neighbor sample sets and non-neighbor sample sets are embedded into the common subspace to obtain the common features of the image and its neighbor sample set With non-neighbor sample set Common features of text and its neighbor sample set With non-neighbor sample set The features of the neighbor sample set are aggregated, while the features of the non-neighbor sample set are alienated. The calculation method is as follows:

[0028]

[0029] Where, and is the cross-modal feature expression of the image in the common subspace, and is the cross-modal feature expression of the text in the common subspace, s(·) is the intra-modal feature similarity measurement function, is the image modality neighbor relation constraint, is the text modality neighbor relation constraint, L neigh is the cross-modal neighbor relationship constraint, μ is the weight coefficient;

[0030] In the entity feature alignment loss L entity and cross-modal neighbor relation constraints L neigh The entity public semantic space is obtained through training under the joint action.

[0031] Furthermore, in step 3), the image samples in the unpaired image-text dataset are first divided into k r entities and embed them into the entity public semantic space to obtain the entity image feature matrix j indicates that the image sample is the jth image sample in the unpaired image-text dataset. is the image sample after segmentation k r The image unimodal features of entities; the text samples in the unpaired image-text dataset are divided into k w entity words and embed them into the entity public semantic space to obtain the entity text feature matrix i indicates that the text sample is the i-th text sample in the unpaired image-text dataset. is the k after the text sample is segmented w The text unimodal features of entity words; in the entity public semantic space, calculate T i U and The similarity matrix C between ij , calculated as follows:

[0032]

[0033] Where, T i U ' indicates T i U The transpose of

[0034] Then for C ij Perform maximum pooling in the row direction and mean pooling in the column direction, and define the image-text similarity between the jth image sample and the i-th text sample as The calculation is as follows:

[0035]

[0036] C m =max{C mi ,i=1,...,k r}

[0037] Where C m Indicates C ij The result of the maximum pooling on the mth row, k r With k w They are With T i U length;

[0038] In the unpaired image-text dataset, for all images, similarity is calculated with all texts according to the image-text similarity calculation method, and the text with the highest similarity is selected as the pseudo-pairing sample of the image to form a pseudo-pairing set Define the image at the same time Advantage score l of the retrieved text j For images Similarity to the most similar text and image The difference in similarity with the second most similar text is used to sample the advantage scores of all images. The two-component Gaussian mixture model is fitted and optimized using the expectation maximization algorithm to obtain the probability density function p(l). The formula for p(l) is as follows:

[0039]

[0040] Where, represents the probability density of the kth component, and the component with a higher average value is set as the component of reliable pseudo-pairing. k = 1 is defined as a reliable pseudo-pairing component, k = 2 is defined as an unreliable pseudo-pairing component, l is the advantage score of image retrieval text, β k represents the probability density coefficient of component k;

[0041] According to the probability density function according to the advantage score l j calculate The posterior probability p(k=1|l j ), calculated as follows:

[0042]

[0043] Where, p(l j ) is the total probability density calculated by the probability density function p(l), p(k=1) is the weight of the first component in p(l), namely β1, p(l j |k=1) is the probability density calculated from the first component in p(l);

[0044] From the pseudo-pairing set Select pseudo-pairings with posterior probability greater than 0.5 to form a high-confidence image-text pseudo-pairing set Where V Pi Represents the image, T Pi Represents text, N P D P The number of pseudo-pairs included.

[0045] Furthermore, in step 4), the high confidence image-text pseudo pairing set D is aligned by embedding P , and use the relation fusion module to capture the relationship between entities in the sample, as follows:

[0046] First, D is transformed into P Image V in Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an MLP network to obtain the entity image hidden layer feature matrix Indicates the k contained in the image r' The image hidden layer features of the entity are obtained through the relationship fusion module to obtain the image fusion feature matrix The calculation is as follows:

[0047]

[0048] Where, is the k contained in the image r' The image fusion features of entities, W HV is the entity image similarity weight matrix, the element in the jth row and kth column of the matrix Represents the similarity weight between the j-th entity image and the k-th entity image, which is represented by Given, and is the hidden layer feature of the entity image;

[0049] By word-by-word segmentation method, D P Text in T Pi The entity text is divided into multiple entities and embedded into the entity common semantic space, and then input into an LSTM network to obtain the entity text hidden layer feature matrix Indicates the k contained in the text i' The text hidden layer features of entities are obtained through the relation fusion module to obtain the text fusion feature matrix The calculation is as follows:

[0050]

[0051] Where, is the k contained in the text i' The text fusion feature of entities, W HT is the entity text similarity weight matrix, the element in the jth row and kth column of the matrix The similarity weight between the j-th entity text and the k-th entity text is given by Given, and is the text hidden layer feature of the entity;

[0052] Finally, the fusion feature alignment loss L is calculated based on the triplet loss. fusionThe image fusion feature and the text fusion feature are aligned by making the pseudo-paired samples close together and the non-pseudo-paired samples far apart. The calculation method is as follows:

[0053]

[0054] Where, and They are and The hardest negative sample, β is the marginal hyperparameter, S F (·) is the cross-modal feature similarity measurement function, which is the result of row-wise maximum pooling and column-wise mean pooling of the similarity matrix between image and text;

[0055] Through L fusion The optimization finally obtains the fusion public semantic space.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] 1. This paper proposes a basic framework for an unpaired cross-modal retrieval method. First, a general entity dataset is introduced. The entity public semantic space is learned based on the entity dataset. The entity public semantic space is used to generate a picture-text pseudo-pairing set for the unpaired picture-text dataset. A high-confidence picture-text pseudo-pairing set is obtained through a pseudo-pairing screening algorithm. Finally, a fused public semantic space that can express data relationship information is trained based on the high-confidence picture-text pseudo-pairing set.

[0058] 2. The present invention proposes a public semantic space learning method guided by entity semantics, introduces an entity-level public dataset, and ensures that the intra-modal neighbor information is not destroyed during the alignment process by embedding a spatial alignment method and intra-modal neighbor relationship constraints. The obtained entity public semantic space can be used for image and text retrieval based on the similarity between entities, improving the problem of modal information loss in existing methods.

[0059] 3. The present invention proposes a relationship fusion module, which aims to deeply mine the relationship information between graphic and text data, train and integrate the common semantic space, and enable it to have the ability to understand the relationship between regions.

[0060] 4. This paper proposes a pseudo-pairing generation method and a pseudo-pairing screening method, respectively. Based on the entity public semantic space, a high-confidence pseudo-pairing set is searched. Subsequently, a similarity-weighted feature fusion method and neural network training such as LSTM are used to ultimately obtain a fused public semantic space. This method can measure cross-modal semantic similarity at both the entity and entity relationship levels, enabling accurate unpaired cross-modal retrieval. This overcomes the problem that existing methods often overlook entity relationship information in the data.

[0061] 5. Compared with other unpaired cross-modal retrieval methods, the performance of this invention is significantly improved. The learned space can accurately express the data from a deeper level, and the overall method is more interpretable. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a logic flow chart of the method of the present invention.

[0063] Figure 2 It is a logic flow chart of the pseudo-pair generation and screening method proposed by the method of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0065] like Figure 1 As shown, this embodiment discloses a pair-free cross-modal retrieval method based on entity semantic guidance and relationship fusion, the details of which are as follows:

[0066] 1) Introducing a general entity dataset, first use the label information of the entity image as the paired entity text, and then extract features from the entity image and entity text. Use the convolutional neural network ResNet50 to extract entity image features, and use the time-recurrent neural network LSTM to extract entity text features. The specific process is: given an entity dataset, scale each image in the entity dataset to a size of 256×256, and then use the convolutional neural network ResNet50 to extract entity image features to obtain the entity image feature set. Total N E samples, v Ei is the i-th entity image feature; extract the text label corresponding to the image from the entity dataset, encode the text label into a word vector through the word embedding method, and input the word vector into the time recurrent neural network LSTM to extract the entity text feature to obtain the entity text feature set Total N E samples, t Ei is the text feature of the i-th entity.

[0067] 2) Based on the entity image features and entity text features extracted in the previous step, the correct negative sample is selected for each feature according to the distance relationship between the features, and then the entity public semantic space is obtained based on the embedding space alignment method and the intra-modal neighbor relationship constraint. First, the image unimodal encoder composed of a multi-layer neural network is used to Embed the entity image features into the image unimodal space to obtain the image unimodal features Through a text unimodal encoder f composed of a multi-layer neural network t MEmbed entity text features into text unimodal space to obtain text unimodal features Then, negative samples are selected based on the distance relationship between the image unimodal features and the text unimodal features. Specifically, for The negative sample is obtained by the following calculation method

[0068]

[0069] Where, and is the text unimodal feature, agrmin is the function of the independent variable when taking the minimum value, j is the independent variable of the argmin function, p represents the pth sample in the data set, ||·|| is the L2 norm, ε is the distance boundary parameter, is with The unimodal features of the text corresponding to the subscript;

[0070] by As an anchor point, the corresponding text unimodal features As a positive sample, Get triples as negative samples

[0071] for The negative sample is obtained by the following calculation method

[0072]

[0073] Where, and is the image unimodal feature, q represents the qth sample in the dataset, is with The subscripts correspond to the unimodal features of the image.

[0074] by As an anchor point, the corresponding image modality features As a positive sample, Get triples as negative samples Then, the image cross-modal encoder composed of a multi-layer neural network With the text cross-modal encoder f t C The image unimodal features and text unimodal features in the triplet are embedded into the common subspace respectively to obtain the cross-modal feature expression of the triplet in the common subspace. and and They are and The corresponding cross-modal feature expression is then expressed through the entity feature alignment loss L based on the triplet loss entity The distance between the anchor point and the positive sample is shortened, and the distance between the anchor point and the negative sample is increased. The calculation method is as follows:

[0075]

[0076] Where S E (·) is the cross-modal feature similarity metric function, α is the marginal hyperparameter, and in [·] + Negative values ​​are set to 0, and positive values ​​remain the same.

[0077] Then, the intra-modal neighbor relationship constraint is added based on the distance relationship to avoid the entity feature alignment loss L entity Destroy the potential unimodal neighbor structure. Specifically, take τ as the distance boundary parameter and calculate it according to the following formula Neighbor sample set V Ei+ and non-neighbor sample set V Ei- , Neighbor sample set T Ei+ With non-neighbor sample set T Ei- :

[0078]

[0079] pass With f t C Will And their neighbor sample sets and non-neighbor sample sets are embedded into the common subspace to obtain the common features of the image and its neighbor sample set With non-neighbor sample set Common features of text and its neighbor sample set With non-neighbor sample set The features of the neighbor sample set are aggregated, while the features of the non-neighbor sample set are alienated. The calculation method is as follows:

[0080]

[0081]

[0082] Where, and is the cross-modal feature expression of the image in the common subspace, and is the cross-modal feature expression of the text in the common subspace, s(·) is the intra-modal feature similarity measurement function, specifically cosine similarity is used here, is the image modality neighbor relation constraint, is the text modality neighbor relation constraint, L neigh is the cross-modal neighbor relationship constraint, and μ is the weight coefficient.

[0083] In the entity feature alignment loss L entity and cross-modal neighbor relation constraints L neigh The entity public semantic space is obtained through training under the joint action.

[0084] 3) Based on the entity public semantic space obtained in the previous step, a pseudo pairing generation method is used to construct a pseudo pairing for the unpaired image-text dataset, and then a pseudo pairing screening method is used to obtain a high-confidence complex image-text pseudo pairing set. The process is as follows: Figure 2 First, the image samples in the unpaired image-text dataset are divided into k r entities and embed them into the entity public semantic space to obtain the entity image feature matrix j indicates that the image sample is the jth image sample in the unpaired image-text dataset. is the image sample after segmentation k r The image unimodal features of entities; the text samples in the unpaired image-text dataset are divided into k w entity words and embed them into the entity public semantic space to obtain the entity text feature matrix i indicates that the text sample is the i-th text sample in the unpaired image-text dataset. is the k after the text sample is segmented w The text unimodal features of entity words; in the entity public semantic space, calculate T i U and The similarity matrix C between ij , calculated as follows:

[0085]

[0086] Where, T i U ' indicates T i U The transpose of .

[0087] Then for C ij Perform maximum pooling in the row direction and mean pooling in the column direction, and define the complex image-text similarity between the jth image sample and the ith text sample as The calculation is as follows:

[0088]

[0089] C m =max{Cmi ,i=1,...,k r}

[0090] Where C m Indicates C ij The result of the maximum pooling on the mth row, k r With k w They are With T i U length.

[0091] In the unpaired image-text dataset, for all images, similarity is calculated with all texts according to the image-text similarity calculation method, and the text with the highest similarity is selected as the pseudo-pairing sample of the image to form a pseudo-pairing set Define the image at the same time Advantage score l of the retrieved text j For images Similarity to the most similar text and image The difference in similarity with the second most similar text is used to sample the advantage scores of all images. The two-component Gaussian mixture model is fitted and optimized using the expectation maximization algorithm to obtain the probability density function p(l). The formula for p(l) is as follows:

[0092]

[0093] Where, represents the probability density of the kth component, and the component with a higher average value is set as the component of reliable pseudo-pairing. k = 1 is defined as a reliable pseudo-pairing component, k = 2 is defined as an unreliable pseudo-pairing component, l is the advantage score of image retrieval text, β k represents the probability density coefficient of component k.

[0094] According to the probability density function according to the advantage score l j calculate The posterior probability p(k=1|l j ), calculated as follows:

[0095]

[0096] Where, p(l j ) is the total probability density calculated by the probability density function p(l), p(k=1) is the weight of the first component in p(l), namely β1, p(l j |k=1) is the probability density calculated from the first component in p(l).

[0097] From the pseudo-pairing set Select pseudo-pairings with posterior probability greater than 0.5 to form a high-confidence image-text pseudo-pairing set Where V Pi Represents the image, T Pi Represents text, N P D P The number of pseudo-pairs included.

[0098] 4) Based on the high-confidence complex image-text pseudo-pairing set obtained in the previous step, the relationship fusion module is used to extract and align the fusion features containing the information relationship to obtain the fusion public semantic space.

[0099] First, align the high-confidence image-text pseudo-pairing set D by embedding P , and use the relationship fusion module (feature fusion method based on similarity weighting) to capture the relationship between entities in the sample. Specifically, first, D P Image V in Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an MLP network to obtain the entity image hidden layer feature matrix Indicates the k contained in the image r' The image hidden layer features of the entity are obtained through the relationship fusion module to obtain the image fusion feature matrix The calculation is as follows:

[0100]

[0101] Where, is the k contained in the image r' The image fusion features of entities, W HV is the entity image similarity weight matrix, the element in the jth row and kth column of the matrix Represents the similarity weight between the j-th entity image and the k-th entity image, which is represented by Given, and is the hidden layer feature of the entity image.

[0102] By word-by-word segmentation method, D P Text in T Pi The entity text is divided into multiple entities and embedded into the entity common semantic space, and then input into an LSTM network to obtain the entity text hidden layer feature matrix Indicates the k contained in the text i' The text hidden layer features of entities are obtained through the relation fusion module to obtain the text fusion feature matrix The calculation is as follows:

[0103]

[0104] Where, is the k contained in the text i' The text fusion feature of entities, W HT is the entity text similarity weight matrix, the element in the jth row and kth column of the matrix The similarity weight between the j-th entity text and the k-th entity text is given by Given, and is the textual hidden layer feature of the entity.

[0105] Finally, the fusion feature alignment loss L is calculated based on the triplet loss. fusion The image fusion feature and the text fusion feature are aligned by making the pseudo-paired samples close together and the non-pseudo-paired samples far apart. The calculation method is as follows:

[0106]

[0107] Where, and They are and The most difficult negative sample, the selection method of the most difficult negative sample is the same as that in general cross-modal retrieval, β is the marginal hyperparameter, S F (·) is a cross-modal feature similarity measurement function, which is the result of performing row-wise maximum pooling and column-wise mean pooling on the similarity matrix between image and text.

[0108] Through L fusion The optimization finally obtains the fusion public semantic space.

[0109] 5) By embedding cross-modal data into the fused common semantic space, the cross-modal similarity is calculated, and based on the similarity ranking, the top n instances are selected as the final cross-modal retrieval results.

[0110] The above embodiments are preferred implementations of the present invention, but the implementations of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. Unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion, characterized by: The following steps are involved: 1) Extract entity image features and entity text features from the entity dataset; 2) Based on entity image features and entity text features, the distance between features is calculated. The correct negative sample is selected for each feature based on the distance relationship. Through the method of embedding space alignment and intra-modal neighbor relationship constraints, the negative sample is moved away from the feature to obtain the entity common semantic space. Among them, the intra-modal neighbor relationship constraint first calculates the neighbors of the feature in the unimodal space, and then constrains the distance relationship between the neighbors before and after alignment to minimize the change, thereby avoiding the damage of the unimodal neighbor structure during the alignment process. 3) Based on the entity public semantic space, a pseudo-pairing generation method is used to construct pseudo-pairings from an unpaired image-text dataset. A pseudo-pairing screening method is used to obtain a high-confidence image-text pseudo-pairing set. The unpaired image-text dataset is a dataset lacking pairing information annotations given by the task. The pseudo-pairing generation method selects the samples with the highest similarity based on the similarity ranking of the image and text in the entity public semantic space to construct pseudo-pairings. The pseudo-pairing screening method samples the difference between the highest and second-highest similarity based on the similarity ranking of the image and text in the entity public semantic space. This difference is called the advantage score, which is used as an indicator to measure the reliability of the pseudo-pairing. The advantage score is fitted using a two-component Gaussian mixture model, and a probability density function is obtained by optimization based on the expectation-maximization algorithm. The probability density function is used to calculate and screen pseudo-pairings to obtain a high-confidence image-text pseudo-pairing set. 4) Based on a set of high-confidence pseudo-image-text pairs, the Faster R-CNN entity recognition network is used to segment the images in the set into multiple entity images. The text in the set is segmented into multiple entity texts using a word-by-word segmentation method. A relational fusion module is then used to extract fused features containing information relationships between the entity images and entity texts. The fused features are then aligned to obtain a fused common semantic space. The relational fusion module is a similarity-weighted feature fusion method that first calculates weights based on the similarity between the entity images and entity texts, and then uses the weights to exchange information between the entities. 5) Embed cross-modal data into the fused common semantic space to calculate cross-modal similarity, that is, calculate the similarity between images and texts based on the fused common semantic space, sort them according to the similarity, and select the top n instances as the final cross-modal retrieval results.

2. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 1 is characterized in that: In step 1), the convolutional neural network ResNet50 is used to extract entity image features, and the time recurrent neural network LSTM is used to extract entity text features. The specific process is: given an entity dataset, each image in the entity dataset is scaled to a size of 256×256, and then the convolutional neural network ResNet50 is used to extract entity image features to obtain the entity image feature set. Total N E samples, v Ei is the i-th entity image feature; extract the text label corresponding to the image from the entity dataset, encode the text label into a word vector through the word embedding method, and input the word vector into the time recurrent neural network LSTM to extract the entity text feature, and obtain the entity text feature set Total N E samples, t Ei is the text feature of the i-th entity.

3. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 2 is characterized in that: In step 2), the image is encoded using a unimodal encoder consisting of a multi-layer neural network. Embed the entity image features into the image unimodal space to obtain the image unimodal features Through a text unimodal encoder f composed of a multi-layer neural network t M Embed entity text features into text unimodal space to obtain text unimodal features Then, negative samples are selected based on the distance relationship between the image unimodal features and the text unimodal features; for The negative sample is obtained by the following calculation method Where, and is the text unimodal feature, agrmin is the function of the independent variable when taking the minimum value, j is the independent variable of the argmin function, p represents the pth sample in the data set, ||·|| is the L2 norm, ε is the distance boundary parameter, is with The unimodal features of the text corresponding to the subscript; by As an anchor point, the corresponding text unimodal features As a positive sample, Get triples as negative samples for The negative sample is obtained by the following calculation method Where, and is the image unimodal feature, q represents the qth sample in the dataset, is with The subscript corresponds to the unimodal feature of the image; by As an anchor point, the corresponding image unimodal feature As a positive sample, Get triples as negative samples Through an image cross-modal encoder composed of a multi-layer neural network With the text cross-modal encoder f t C The image unimodal features and text unimodal features in the triplet are embedded into the common subspace respectively to obtain the cross-modal feature expression of the triplet in the common subspace. and and They are and The corresponding cross-modal feature expression is then expressed through the entity feature alignment loss L based on the triplet loss entity The distance between the anchor point and the positive sample is shortened, and the distance between the anchor point and the negative sample is increased. The calculation method is as follows: Where S E (·) is the cross-modal feature similarity metric function, α is the marginal hyperparameter, and in [·] + Negative values ​​are set to 0, and positive values ​​remain the original value; Then, the intra-modal neighbor relationship constraint is added based on the distance relationship to avoid the entity feature alignment loss L entity Destroy the potential single-mode neighbor structure, take τ as the distance boundary parameter, and calculate it according to the following formula Neighbor sample set V Ei+ and non-neighbor sample set V Ei- as well as Neighbor sample set T Ei+ With non-neighbor sample set T Ei- : pass With f t C Will And their neighbor sample sets and non-neighbor sample sets are embedded into the common subspace to obtain the common features of the image and its neighbor sample set With non-neighbor sample set Common features of text and its neighbor sample set With non-neighbor sample set The features of the neighbor sample set are aggregated, while the features of the non-neighbor sample set are alienated. The calculation method is as follows: Where, and is the cross-modal feature expression of the image in the common subspace, and is the cross-modal feature expression of the text in the common subspace, s(·) is the intra-modal feature similarity measurement function, is the image modality neighbor relation constraint, is the text modality neighbor relation constraint, L neigh is the cross-modal neighbor relationship constraint, μ is the weight coefficient; In the entity feature alignment loss L entity and cross-modal neighbor relation constraints L neigh The entity public semantic space is obtained through training under the joint action.

4. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 3 is characterized in that: In step 3), the image samples in the unpaired image-text dataset are first divided into k r entities and embed them into the entity public semantic space to obtain the entity image feature matrix j indicates that the image sample is the jth image sample in the unpaired image-text dataset. is the image sample after segmentation k r The image unimodal features of entities; the text samples in the unpaired image-text dataset are divided into k w entity words and embed them into the entity public semantic space to obtain the entity text feature matrix i indicates that the text sample is the i-th text sample in the unpaired image-text dataset. is the k after the text sample is segmented w The text unimodal features of entity words; in the entity public semantic space, calculate T i U and The similarity matrix C between ij , calculated as follows: Where, T i U' Indicates T i U The transpose of Then for C ij Perform maximum pooling in the row direction and mean pooling in the column direction, and define the image-text similarity between the jth image sample and the ith text sample as The calculation is as follows: C m =max{C mi ,i=1,...,k r } Where C m Indicates C ij The result of the maximum pooling on the mth row, k r With k w They are With T i U length; In the unpaired image-text dataset, for all images, similarity is calculated with all texts according to the image-text similarity calculation method, and the text with the highest similarity is selected as the pseudo-pairing sample of the image to form a pseudo-pairing set Define the image at the same time Advantage score l of the retrieved text j For images Similarity to the most similar text and image The difference in similarity with the second most similar text is obtained by sampling the advantage scores of all images in sequence, fitting them through a two-component Gaussian mixture model, and optimizing the two-component Gaussian mixture model through the expectation maximization algorithm to obtain the probability density function p(l). The formula of p(l) is as follows: Where, represents the probability density of the kth component, and the component with a higher average value is set as the component of reliable pseudo-pairing. k = 1 is defined as a reliable pseudo-pairing component, k = 2 is defined as an unreliable pseudo-pairing component, l is the advantage score of image retrieval text, β k represents the probability density coefficient of component k; According to the probability density function according to the advantage score l j calculate The posterior probability p(k=1|l j ), calculated as follows: Where, p(l j ) is the total probability density calculated by the probability density function p(l), p(k=1) is the weight of the first component in p(l), namely β1, p(l j |k=1) is the probability density calculated from the first component in p(l); From the pseudo-pairing set Select pseudo-pairings with posterior probability greater than 0.5 to form a high-confidence image-text pseudo-pairing set Where V Pi Represents the image, T Pi Represents text, N P D P The number of pseudo-pairs included.

5. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 4 is characterized in that: In step 4), the high confidence image-text pseudo pairing set D is aligned by embedding P , and use the relation fusion module to capture the relationship between entities in the sample, as follows: First, D is transformed into P Image V in Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an MLP network to obtain the entity image hidden layer feature matrix Indicates the k contained in the image r' The image hidden layer features of the entity are obtained through the relationship fusion module to obtain the image fusion feature matrix The calculation is as follows: Where, is the k contained in the image r' The image fusion features of entities, W HV is the entity image similarity weight matrix, the element in the jth row and kth column of the entity image similarity weight matrix Represents the similarity weight between the j-th entity image and the k-th entity image, which is represented by Given, and is the hidden layer feature of the entity image; By word-by-word segmentation method, D P Text in T Pi The entity text is divided into multiple entities and embedded into the entity common semantic space, and then input into an LSTM network to obtain the entity text hidden layer feature matrix Indicates the k contained in the text i' The text hidden layer features of entities are obtained through the relation fusion module to obtain the text fusion feature matrix The calculation is as follows: Where, is the k contained in the text i' The text fusion feature of entities, W HT is the entity text similarity weight matrix, the element in the jth row and kth column of the entity text similarity weight matrix The similarity weight between the j-th entity text and the k-th entity text is given by Given, and is the textual hidden layer feature of the entity; Finally, the fusion feature alignment loss L is calculated based on the triplet loss. fusion The image fusion feature and the text fusion feature are aligned by making the pseudo-paired samples close together and the non-pseudo-paired samples far apart. The calculation method is as follows: Where, and They are and The hardest negative sample, β is the marginal hyperparameter, S F (·) is the cross-modal feature similarity measurement function, which is the result of row-wise maximum pooling and column-wise mean pooling of the similarity matrix between image and text; Through L fusion The optimization finally obtains the fusion public semantic space.

Images