Non-pairing cross-modal retrieval method based on entity semantic guidance and relation fusion

Abstract

The invention discloses a non-pairing cross-modal retrieval method based on entity semantic guidance and relation fusion. The method comprises the following steps: extracting entity image features and entity text features from an entity data set; obtaining an entity public semantic space through negative sample selection and intra-modal neighbor relation constraint based on the entity image features and the entity text features; based on the entity public semantic space, a high-confidence image-text pseudo-pairing set is obtained through a pseudo-pairing generation and screening method; on the basis of the high-confidence image-text pseudo-pairing set, relation information is captured and aligned through a relation fusion module, and a fused public semantic space is obtained; and calculating the similarity between the images and texts based on the fused public semantic space, sorting according to the similarity, and outputting a cross-modal retrieval result. Compared with other unpaired cross-modal retrieval methods, the method has the advantages that the performance is obviously improved, a space obtained by learning can accurately express data from a deeper level, and the whole method is better in interpretability.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and natural language generation, and in particular to an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion. Background Art

[0002] With the development of artificial intelligence technology and the explosive growth of multimedia data in daily life, cross-modal retrieval has become an important research topic. Cross-modal retrieval aims to explore the connection between data of different modalities, and retrieve data with similar semantics in another modal database by inputting data of one modality. There is a semantic gap between data of different modalities. How to accurately measure the semantic similarity of cross-modal data has become one of the main challenges of cross-modal retrieval. As a multimodal cross-topic, cross-modal retrieval technology promotes the model's understanding of multimodal information, provides matching functions for various types of data, and provides new ideas and technologies for the development of other modal tasks. It has important theoretical research value; at the same time, cross-modal retrieval has broad application prospects. For example, information retrieval, product retrieval on e-commerce platforms, and personalized recommendations on social networking platforms can all be applied to cross-modal retrieval technology.

[0003] The current mainstream cross-modal retrieval methods are trained on large-scale paired annotated datasets. By aligning cross-modal data and constructing a common semantic space, the cross-modal semantic gap is eliminated. However, in reality, although enough data can be collected from the Internet, it takes a lot of manpower and time to annotate paired information for these data. Therefore, there is work studying cross-modal retrieval in the absence of paired information. Current unpaired cross-modal retrieval methods guide model training by introducing external general databases, but most of these methods replace text features with image features, resulting in the loss of information on the text modality. Often, each modality has its own information that promotes understanding in the retrieval task. The lack of this information will lead to biased retrieval results. Secondly, these methods treat each region in the data in isolation and ignore the relationship between regions. The relative positions of different regions in the image or text reveal the degree of association between regions. The lack of relationship information will also lead to inaccurate retrieval results. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and proposes an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion. The method introduces a general entity dataset, preprocesses the dataset to construct entity image and entity text pairs, and constructs an entity public semantic space by aligning the entity data and the neighbor relationship within the constrained modality. In this space, high-confidence image-text pseudo-pairings are generated and screened for the unpaired image-text dataset, and the relationship information in the data is captured through a distance-weighted relationship fusion module. The final fused public semantic space is obtained by fine-tuning through alignment, which improves the deficiencies of the existing methods and realizes reliable cross-modal retrieval.

[0005] To achieve the above object, the technical solution provided by the present invention is: an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion, comprising the following steps:

[0006] 1) Extract entity image features and entity text features from entity datasets;

[0007] 2) Based on the entity image features and entity text features, the distance between the features is calculated, and the correct negative sample is selected for each feature according to the distance relationship. The negative sample is kept away from the feature by embedding the space alignment method and the intra-modal neighbor relationship constraint to obtain the entity common semantic space; wherein, the intra-modal neighbor relationship constraint first calculates the neighbors of the feature in the unimodal space, and then constrains the change in the distance relationship between the neighbors before and after alignment to be minimized, thereby avoiding the destruction of the unimodal neighbor structure during the alignment process;

[0008] 3) Based on the entity public semantic space, a pseudo-pairing generation method is used to construct a pseudo-pairing from an unpaired image-text dataset, and a pseudo-pairing screening method is used to obtain a high-confidence image-text pseudo-pairing set for the pseudo-pairing; wherein, the unpaired image-text dataset is a dataset lacking pairing information annotations given by the task, the pseudo-pairing generation method is to select the sample with the highest similarity based on the similarity ranking of the image and text in the entity public semantic space to construct a pseudo-pairing, and the pseudo-pairing screening method is to sample the difference between the highest similarity and the second highest similarity based on the similarity ranking of the image and text in the entity public semantic space, and the difference is called the advantage score, which is used as an indicator to measure the reliability of the pseudo-pairing, and the advantage score is fitted by a two-component Gaussian mixture model, and a probability density function is obtained based on the expectation maximization algorithm optimization, and the pseudo-pairing is calculated and screened by the probability density function to obtain a high-confidence image-text pseudo-pairing set;

[0009] 4) Based on the high-confidence image-text pseudo-pairing set, the image in the high-confidence image-text pseudo-pairing set is segmented into multiple entity images through the entity recognition network Faster R-CNN, and the text in the high-confidence image-text pseudo-pairing set is segmented into multiple entity texts through the word-by-word segmentation method. Then, the fusion features containing the information relationship between the entity image and the entity text are extracted through the relationship fusion module, and the fusion features are aligned to obtain the fusion common semantic space; wherein, the relationship fusion module is a feature fusion method based on similarity weighting, firstly, the weight is calculated by the similarity between the entity image and the entity text, and then the information between the entities is interacted according to the weight;

[0010] 5) Embed the cross-modal data into the fused public semantic space to calculate the cross-modal similarity, that is, calculate the similarity between images and texts based on the fused public semantic space, sort them according to the similarity, and select the top n instances as the final cross-modal retrieval results.

[0011] Furthermore, in step 1), the convolutional neural network ResNet50 is used to extract entity image features, and the time recurrent neural network LSTM is used to extract entity text features. The specific process is: given an entity data set, each image in the entity data set is scaled to a size of 256×256, and then the convolutional neural network ResNet50 is used to extract entity image features to obtain the entity image feature set Total N E samples, v Ei is the i-th entity image feature; extract the text label corresponding to the image from the entity dataset, encode the text label into a word vector through the word embedding method, and input the word vector into the time recurrent neural network LSTM to extract the entity text feature, and obtain the entity text feature set Total N E samples, t Ei is the i-th entity text feature.

[0012] Further, in step 2), an image unimodal encoder composed of a multi-layer neural network is used. Embed entity image features into image unimodal space to obtain image unimodal features Through a text unimodal encoder composed of a multi-layer neural network Embed entity text features into text unimodal space to obtain text unimodal features Then, negative samples are selected based on the distance relationship between the image unimodal features and the text unimodal features;

[0013] for The negative sample is obtained by the following calculation method.

[0014]

[0015]

[0016] In the formula, and is the unimodal feature of the text, agrmin is the function of the independent variable when taking the minimum value, j is the independent variable of the argmin function, p represents the pth sample in the data set, ||·|| is the L2 norm, ε is the distance boundary parameter, is with The unimodal features of the text corresponding to the subscript;

[0017] by As an anchor point, the corresponding text unimodal feature As a positive sample, Get triples as negative samples

[0018] for The negative sample is obtained by the following calculation method.

[0019]

[0020] In the formula, and is the unimodal feature of the image, q represents the qth sample in the data set, is with The subscripts correspond to the unimodal features of the image;

[0021] by As an anchor point, the corresponding image unimodal feature As a positive sample, Get triples as negative samples

[0022] Through an image cross-modal encoder composed of a multi-layer neural network With the text cross-modal encoder f t C The image unimodal features and text unimodal features in the triplet are respectively embedded into the common subspace to obtain the cross-modal feature expression of the triplet in the common subspace. and and They are and The corresponding cross-modal feature expression is then expressed through the entity feature alignment loss L based on the triplet loss entity The distance between the anchor point and the positive sample is shortened, and the distance between the anchor point and the negative sample is increased. The calculation method is as follows:

[0023]

[0024] In the formula, S E (·) is the cross-modal feature similarity metric function, α is the marginal hyperparameter, and in [·] + Negative values ​​are taken as 0, and positive values ​​remain the original value;

[0025] Then, the intra-modal neighbor relationship constraint is added according to the distance relationship to avoid the entity feature alignment loss L entity Destroy the potential unimodal neighbor structure, take τ as the distance boundary parameter, and calculate it according to the following formula Neighbor sample set V Ei+ With non-neighbor sample set V Ei- as well as Neighbor sample set T Ei+ With non-neighbor sample set T Ei- :

[0026]

[0027] pass With f t C Will And their neighbor sample sets and non-neighbor sample sets are embedded into the common subspace to obtain the common features of the image and its neighbor sample set With non-neighbor sample set Common features of text and its neighbor sample set With non-neighbor sample set The features of the neighbor sample set are clustered, while the features of the non-neighbor sample set are alienated. The calculation method is as follows:

[0028]

[0029] In the formula, and is the cross-modal feature expression of the image in the common subspace, and is the cross-modal feature expression of the text in the common subspace, s(·) is the intra-modal feature similarity measurement function, is the image modality neighbor relation constraint, is the text modality neighbor relation constraint, L neigh is the cross-modal neighbor relationship constraint, μ is the weight coefficient;

[0030] In the entity feature alignment loss L entity and the cross-modal neighbor relation constraint L neigh The entity common semantic space is obtained through joint training.

[0031] Further, in step 3), the image samples in the unpaired image-text dataset are first segmented into k r entities and embed them into the entity common semantic space to obtain the entity image feature matrix j indicates that the image sample is the jth image sample in the unpaired image-text dataset. is the image sample after segmentation k r The image unimodal features of entities; the text samples in the unpaired image-text dataset are divided into k w entity words and embed them into the entity common semantic space to obtain the entity text feature matrix i means that the text sample is the i-th text sample in the unpaired image-text dataset. is the k after the text sample is segmented w The text unimodal features of entity words; in the entity common semantic space, calculate T i U and The similarity matrix C between ij , calculated as follows:

[0032]

[0033] Where, T i U ' indicates T i U The transpose of

[0034] Then for C ij Perform maximum pooling in the row direction and mean pooling in the column direction, and define the image-text similarity between the jth image sample and the i-th text sample as The calculation is as follows:

[0035]

[0036] C m =max{C mi ,i=1,...,k r}

[0037] In the formula, C m Represents C ij The result of the maximum pooling on the mth row, k r With k w They are With T i U Length;

[0038] In the unpaired image-text dataset, for all images, similarity is calculated with all texts according to the image-text similarity calculation method, and the text with the highest similarity is selected as the pseudo-pairing sample of the image to form a pseudo-pairing set Define the image at the same time Advantage score l of the retrieved text j For images Similarity and image to the most similar text The difference in similarity with the second highest similarity text is used to sample the advantage scores of all images, fit them through a two-component Gaussian mixture model, and optimize the two-component Gaussian mixture model through the expectation maximization algorithm to obtain the probability density function p(l). The formula of p(l) is as follows:

[0039]

[0040] In the formula, represents the probability density of the kth component, and the component with a higher average value is set as the component of reliable pseudo-pairing. k = 1 is defined as a reliable pseudo-pairing component, k = 2 is defined as an unreliable pseudo-pairing component, l is the advantage score of the image retrieval text, β k represents the probability density coefficient of component k;

[0041] According to the probability density function according to the advantage score l j calculate The posterior probability p(k=1|l j ), calculated as follows:

[0042]

[0043] In the formula, p(l j ) is the total probability density calculated by the probability density function p(l), p(k=1) is the weight of the first component in p(l), namely β1, p(l j |k=1) is the probability density calculated from the first component in p(l);

[0044] From the pseudo-pairing set Select pseudo-pairings with posterior probability greater than 0.5 to form a high-confidence image-text pseudo-pairing set Where V Pi Represents the image, T Pi Represents text, N P D P The number of pseudo-pairs included.

[0045] Further, in step 4), the high confidence image-text pseudo pairing set D is aligned by embedding P , and use the relation fusion module to capture the relationship between entities in the sample, as follows:

[0046] First, D is transformed into P Image V in Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an MLP network to obtain the entity image hidden layer feature matrix Indicates the k contained in the image r' The image hidden layer features of entities are obtained through the relationship fusion module to obtain the image fusion feature matrix The calculation is as follows:

[0047]

[0048] In the formula, is the number of k contained in the image r' The image fusion features of entities, W HV is the entity image similarity weight matrix, the element in the jth row and kth column of the matrix Represents the similarity weight between the j-th entity image and the k-th entity image, Given, and is the image hidden layer feature of the entity;

[0049] By using the word-by-word segmentation method, D P Text in T Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an LSTM network to obtain the entity text hidden layer feature matrix Indicates the k contained in the text i' The text hidden layer features of entities are obtained through the relation fusion module to obtain the text fusion feature matrix The calculation is as follows:

[0050]

[0051] In the formula, is the k contained in the text i' The text fusion feature of entities, W HT is the entity text similarity weight matrix, the element in the jth row and kth column of the matrix The similarity weight between the j-th entity text and the k-th entity text is given by Given, and is the textual hidden layer feature of the entity;

[0052] Finally, the fusion feature alignment loss L is calculated based on the triplet loss. fusionThe pseudo-paired samples are brought closer together and the non-pseudo-paired samples are moved away from each other to achieve the alignment of the image fusion features and the text fusion features. The calculation method is as follows:

[0053]

[0054] In the formula, and They are and is the hardest negative sample, β is the marginal hyperparameter, S F (·) is the cross-modal feature similarity measurement function, which is the result of row maximum pooling and column mean pooling of the similarity matrix between image and text;

[0055] Through L fusion Finally, the fused public semantic space is obtained.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] 1. The present invention proposes a basic framework of an unpaired cross-modal retrieval method. First, a general entity dataset is introduced, and the entity public semantic space is learned based on the entity dataset. The entity public semantic space is used to generate a picture-text pseudo-pairing set for the unpaired picture-text dataset, and a high-confidence picture-text pseudo-pairing set is obtained through a pseudo-pairing screening algorithm. Finally, a fused public semantic space that can express data relationship information is trained based on the high-confidence picture-text pseudo-pairing set.

[0058] 2. The present invention proposes a public semantic space learning method guided by entity semantics, introduces an entity-level public data set, and ensures that the intra-modal neighbor information is not destroyed during the alignment process by embedding a spatial alignment method and intra-modal neighbor relationship constraints. The obtained entity public semantic space can be used for image and text retrieval based on the similarity between entities, which improves the problem of modal information loss in existing methods.

[0059] 3. The present invention proposes a relationship fusion module, which aims to deeply mine the relationship information between graphic and text data, train and integrate the public semantic space, so that it has the ability to understand the relationship between regions.

[0060] 4. The present invention proposes a pseudo-pairing generation method and a pseudo-pairing screening method, respectively, to search for high-confidence pseudo-pairing sets based on the entity public semantic space, and then obtains a fused public semantic space based on similarity-weighted feature fusion methods and neural network training such as LSTM, which can simultaneously measure cross-modal semantic similarity from the entity and entity relationship levels, and realize accurate unpaired cross-modal retrieval. This improves the problem that existing methods ignore entity relationship information in the data.

[0061] 5. Compared with other unpaired cross-modal retrieval methods, the performance of this invention is significantly improved. The learned space can accurately express the data at a deeper level, and the overall method has better interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a logic flow chart of the method of the present invention.

[0063] Figure 2 It is a logic flow chart of the pseudo-pairing generation and screening method proposed by the method of the present invention. DETAILED DESCRIPTION

[0064] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0065] like Figure 1 As shown, this embodiment discloses an unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion, and its specific situation is as follows:

[0066] 1) Introduce a general entity dataset. First, use the label information of the entity image as the paired entity text, and then extract features from the entity image and entity text. Use the convolutional neural network ResNet50 to extract entity image features, and use the time recurrent neural network LSTM to extract entity text features. The specific process is: given an entity dataset, scale each image in the entity dataset to a size of 256×256, and then use the convolutional neural network ResNet50 to extract entity image features to obtain an entity image feature set. Total N E samples, v Ei is the i-th entity image feature; extract the text label corresponding to the image from the entity data set, encode the text label into a word vector through the word embedding method, and input the word vector into the time recurrent neural network LSTM to extract the entity text feature to obtain the entity text feature set Total N E samples, t Ei is the i-th entity text feature.

[0067] 2) Based on the entity image features and entity text features extracted in the previous step, select the correct negative sample for each feature according to the distance relationship between the features, and then obtain the entity common semantic space based on the embedding space alignment method and the intra-modal neighbor relationship constraints. First, the image unimodal encoder composed of a multi-layer neural network is used Embed entity image features into image unimodal space to obtain image unimodal features Through a text unimodal encoder composed of a multi-layer neural network f t MEmbed entity text features into text unimodal space to obtain text unimodal features Then, negative samples are selected based on the distance relationship between the image unimodal features and the text unimodal features. Specifically, for The negative sample is obtained by the following calculation method.

[0068]

[0069] In the formula, and is the unimodal feature of the text, agrmin is the function of the independent variable when taking the minimum value, j is the independent variable of the argmin function, p represents the pth sample in the data set, ||·|| is the L2 norm, ε is the distance boundary parameter, is with The unimodal features of the text corresponding to the subscript;

[0070] by As an anchor point, the corresponding text unimodal feature As a positive sample, Get triples as negative samples

[0071] for The negative sample is obtained by the following calculation method.

[0072]

[0073] In the formula, and is the unimodal feature of the image, q represents the qth sample in the dataset, is with The subscripts correspond to the unimodal features of the image.

[0074] by As an anchor point, the corresponding image modality feature As a positive sample, Get triples as negative samples Then, the image cross-modal encoder composed of a multi-layer neural network With the text cross-modal encoder f t C The image unimodal features and text unimodal features in the triplet are respectively embedded into the common subspace to obtain the cross-modal feature expression of the triplet in the common subspace. and and They are and The corresponding cross-modal feature expression is then expressed through the entity feature alignment loss L based on the triplet loss entity The distance between the anchor point and the positive sample is shortened, and the distance between the anchor point and the negative sample is increased. The calculation method is as follows:

[0075]

[0076] In the formula, S E (·) is the cross-modal feature similarity metric function, α is the marginal hyperparameter, and in [·] + Negative values ​​are set to 0, and positive values ​​remain the same.

[0077] Then, the intra-modal neighbor relationship constraint is added according to the distance relationship to avoid the entity feature alignment loss L entity Destroy the potential unimodal neighbor structure. Specifically, take τ as the distance boundary parameter and calculate it according to the following formula Neighbor sample set V Ei+ With non-neighbor sample set V Ei- , Neighbor sample set T Ei+ With non-neighbor sample set T Ei- :

[0078]

[0079] pass With f t C Will And their neighbor sample sets and non-neighbor sample sets are embedded into the common subspace to obtain the common features of the image and its neighbor sample set With non-neighbor sample set Common features of text and its neighbor sample set With non-neighbor sample set The features of the neighbor sample set are clustered, while the features of the non-neighbor sample set are alienated. The calculation method is as follows:

[0080]

[0081]

[0082] In the formula, and is the cross-modal feature expression of the image in the common subspace, and is the cross-modal feature expression of the text in the common subspace, s(·) is the intra-modal feature similarity metric function, specifically the cosine similarity is used here, is the image modality neighbor relation constraint, is the text modality neighbor relation constraint, L neigh is the cross-modal neighbor relationship constraint, and μ is the weight coefficient.

[0083] In the entity feature alignment loss L entity and the cross-modal neighbor relation constraint L neigh The entity common semantic space is obtained through joint training.

[0084] 3) Based on the entity common semantic space obtained in the previous step, a pseudo pairing generation method is used to construct a pseudo pairing for the unpaired image-text dataset, and then a pseudo pairing screening method is used to obtain a high-confidence complex image-text pseudo pairing set. The process is as follows: Figure 2 First, the image samples in the unpaired image and text dataset are divided into k r entities and embed them into the entity common semantic space to obtain the entity image feature matrix j indicates that the image sample is the jth image sample in the unpaired image-text dataset. is the image sample after segmentation k r The image unimodal features of entities; the text samples in the unpaired image-text dataset are divided into k w entity words and embed them into the entity common semantic space to obtain the entity text feature matrix i means that the text sample is the i-th text sample in the unpaired image-text dataset. is the k after the text sample is segmented w The text unimodal features of entity words; in the entity common semantic space, calculate T i U and The similarity matrix C between ij , calculated as follows:

[0085]

[0086] Where, T i U ' indicates T i U The transpose of .

[0087] Then for C ij The maximum pooling is performed in the row direction and the mean pooling is performed in the column direction. The complex image-text similarity between the jth image sample and the i-th text sample is defined as The calculation is as follows:

[0088]

[0089] C m =max{Cmi ,i=1,...,k r}

[0090] In the formula, C m Represents C ij The result of the maximum pooling on the mth row, k r With k w They are With T i U Length.

[0091] In the unpaired image-text dataset, for all images, similarity is calculated with all texts according to the image-text similarity calculation method, and the text with the highest similarity is selected as the pseudo-pairing sample of the image to form a pseudo-pairing set Define the image at the same time Advantage score l of the retrieved text j For images Similarity and image to the most similar text The difference in similarity with the second highest similarity text is used to sample the advantage scores of all images, fit them through a two-component Gaussian mixture model, and optimize the two-component Gaussian mixture model through the expectation maximization algorithm to obtain the probability density function p(l). The formula of p(l) is as follows:

[0092]

[0093] In the formula, represents the probability density of the kth component, and the component with a higher average value is set as the component of reliable pseudo-pairing. k = 1 is defined as a reliable pseudo-pairing component, k = 2 is defined as an unreliable pseudo-pairing component, l is the advantage score of the image retrieval text, β k Represents the probability density coefficient of component k.

[0094] According to the probability density function according to the advantage score l j calculate The posterior probability p(k=1|l j ), calculated as follows:

[0095]

[0096] In the formula, p(l j ) is the total probability density calculated by the probability density function p(l), p(k=1) is the weight of the first component in p(l), namely β1, p(l j |k=1) is the probability density calculated from the first component in p(l).

[0097] From the pseudo-pairing set Select pseudo-pairings with posterior probability greater than 0.5 to form a high-confidence image-text pseudo-pairing set Where V Pi Represents the image, T Pi Represents text, N P D P The number of pseudo-pairs included.

[0098] 4) Based on the high-confidence complex image-text pseudo-pairing set obtained in the previous step, the relationship fusion module is used to extract and align the fusion features containing the information relationship to obtain the fused common semantic space.

[0099] First, align the high-confidence image-text pseudo pairing set D by embedding P , and use the relation fusion module (feature fusion method based on similarity weighting) to capture the relationship between entities in the sample. Specifically, firstly, D P Image V in Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an MLP network to obtain the entity image hidden layer feature matrix Indicates the k contained in the image r' The image hidden layer features of entities are obtained through the relationship fusion module to obtain the image fusion feature matrix The calculation is as follows:

[0100]

[0101] In the formula, is the number of k contained in the image r' The image fusion features of entities, W HV is the entity image similarity weight matrix, the element in the jth row and kth column of the matrix Represents the similarity weight between the j-th entity image and the k-th entity image, Given, and It is the hidden layer feature of the entity image.

[0102] By using the word-by-word segmentation method, D P Text in T Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an LSTM network to obtain the entity text hidden layer feature matrix Indicates the k contained in the text i' The text hidden layer features of entities are obtained through the relation fusion module to obtain the text fusion feature matrix The calculation is as follows:

[0103]

[0104] In the formula, is the k contained in the text i' The text fusion feature of entities, W HT is the entity text similarity weight matrix, the element in the jth row and kth column of the matrix The similarity weight between the j-th entity text and the k-th entity text is given by Given, and is the textual hidden layer feature of the entity.

[0105] Finally, the fusion feature alignment loss L is calculated based on the triplet loss. fusion The pseudo-paired samples are brought closer together and the non-pseudo-paired samples are moved away from each other to achieve the alignment of the image fusion features and the text fusion features. The calculation method is as follows:

[0106]

[0107] In the formula, and They are and The most difficult negative sample, the selection method of the most difficult negative sample is the same as that in general cross-modal retrieval, β is the marginal hyperparameter, S F (·) is the cross-modal feature similarity measurement function, which is the result of row maximum pooling and column mean pooling of the similarity matrix between image and text.

[0108] Through L fusion Finally, the fused public semantic space is obtained.

[0109] 5) The cross-modal similarity is calculated by embedding the cross-modal data into the fused common semantic space, and the top n instances are selected as the final cross-modal retrieval results based on the similarity sorting.

[0110] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An unpaired cross-modal retrieval method based on entity semantics guidance and relationship fusion, characterized in that: The following steps are involved: 1) Extract entity image features and entity text features from entity datasets; 2) Based on the entity image features and entity text features, the distance between the features is calculated, and the correct negative sample is selected for each feature according to the distance relationship. The negative sample is kept away from the feature by embedding the space alignment method and the intra-modal neighbor relationship constraint to obtain the entity common semantic space; wherein, the intra-modal neighbor relationship constraint first calculates the neighbors of the feature in the unimodal space, and then constrains the change in the distance relationship between the neighbors before and after alignment to be minimized, thereby avoiding the destruction of the unimodal neighbor structure during the alignment process; 3) Based on the entity public semantic space, a pseudo-pairing generation method is used to construct a pseudo-pairing from an unpaired image-text dataset, and a pseudo-pairing screening method is used to obtain a high-confidence image-text pseudo-pairing set for the pseudo-pairing; wherein, the unpaired image-text dataset is a dataset lacking pairing information annotations given by the task, the pseudo-pairing generation method is to select the sample with the highest similarity based on the similarity ranking of the image and text in the entity public semantic space to construct a pseudo-pairing, and the pseudo-pairing screening method is to sample the difference between the highest similarity and the second highest similarity based on the similarity ranking of the image and text in the entity public semantic space, and the difference is called the advantage score, which is used as an indicator to measure the reliability of the pseudo-pairing, and the advantage score is fitted by a two-component Gaussian mixture model, and a probability density function is obtained based on the expectation maximization algorithm optimization, and the pseudo-pairing is calculated and screened by the probability density function to obtain a high-confidence image-text pseudo-pairing set; 4) Based on the high-confidence image-text pseudo-pairing set, the image in the high-confidence image-text pseudo-pairing set is segmented into multiple entity images through the entity recognition network Faster R-CNN, and the text in the high-confidence image-text pseudo-pairing set is segmented into multiple entity texts through the word-by-word segmentation method. Then, the fusion features containing the information relationship between the entity image and the entity text are extracted through the relationship fusion module, and the fusion features are aligned to obtain the fusion common semantic space; wherein, the relationship fusion module is a feature fusion method based on similarity weighting, firstly, the weight is calculated by the similarity between the entity image and the entity text, and then the information between the entities is interacted according to the weight; 5) Embed the cross-modal data into the fused public semantic space to calculate the cross-modal similarity, that is, calculate the similarity between images and texts based on the fused public semantic space, sort them according to the similarity, and select the top n instances as the final cross-modal retrieval results.

2. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 1 is characterized in that: In step 1), the convolutional neural network ResNet50 is used to extract entity image features, and the time recurrent neural network LSTM is used to extract entity text features. The specific process is: given an entity data set, each image in the entity data set is scaled to a size of 256×256, and then the convolutional neural network ResNet50 is used to extract entity image features to obtain the entity image feature set. Total N E samples, v Ei is the i-th entity image feature; extract the text label corresponding to the image from the entity dataset, encode the text label into a word vector through the word embedding method, and input the word vector into the time recurrent neural network LSTM to extract the entity text feature, and obtain the entity text feature set Total N E samples, t Ei is the i-th entity text feature.

3. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 2 is characterized in that: In step 2), the image is encoded by a unimodal encoder consisting of a multi-layer neural network. Embed entity image features into image unimodal space to obtain image unimodal features Through a text unimodal encoder composed of a multi-layer neural network f t M Embed entity text features into text unimodal space to obtain text unimodal features Then, negative samples are selected based on the distance relationship between the image unimodal features and the text unimodal features; for The negative sample is obtained by the following calculation method. In the formula, and is the unimodal feature of the text, agrmin is the function of the independent variable when taking the minimum value, j is the independent variable of the argmin function, p represents the pth sample in the data set, ||·|| is the L2 norm, ε is the distance boundary parameter, is with The unimodal features of the text corresponding to the subscript; by As an anchor point, the corresponding text unimodal feature As a positive sample, Get triples as negative samples for The negative sample is obtained by the following calculation method. In the formula, and is the unimodal feature of the image, q represents the qth sample in the data set, is with The subscripts correspond to the unimodal features of the image; by As an anchor point, the corresponding image unimodal feature As a positive sample, Get triples as negative samples Through an image cross-modal encoder composed of a multi-layer neural network With the text cross-modal encoder f t C The image unimodal features and text unimodal features in the triplet are respectively embedded into the common subspace to obtain the cross-modal feature expression of the triplet in the common subspace. and and They are and The corresponding cross-modal feature expression is then expressed through the entity feature alignment loss L based on the triplet loss entity The distance between the anchor point and the positive sample is shortened, and the distance between the anchor point and the negative sample is increased. The calculation method is as follows: In the formula, S E (·) is the cross-modal feature similarity metric function, α is the marginal hyperparameter, and in [·] + Negative values ​​are taken as 0, and positive values ​​remain the original value; Then, the intra-modal neighbor relationship constraint is added according to the distance relationship to avoid the entity feature alignment loss L entity Destroy the potential unimodal neighbor structure, take τ as the distance boundary parameter, and calculate it according to the following formula Neighbor sample set V Ei+ With non-neighbor sample set V Ei- as well as Neighbor sample set T Ei+ With non-neighbor sample set T Ei- : pass With f t C Will And their neighbor sample sets and non-neighbor sample sets are embedded into the common subspace to obtain the common features of the image and its neighbor sample set With non-neighbor sample set Common features of text and its neighbor sample set With non-neighbor sample set The features of the neighbor sample set are clustered, while the features of the non-neighbor sample set are alienated. The calculation method is as follows: In the formula, and is the cross-modal feature expression of the image in the common subspace, and is the cross-modal feature expression of the text in the common subspace, s(·) is the intra-modal feature similarity measurement function, is the image modality neighbor relation constraint, is the text modality neighbor relation constraint, L neigh is the cross-modal neighbor relationship constraint, μ is the weight coefficient; In the entity feature alignment loss L entity and the cross-modal neighbor relation constraint L neigh The entity common semantic space is obtained through joint training.

4. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 3 is characterized in that: In step 3), the image samples in the unpaired image-text dataset are first segmented into k r entities and embed them into the entity common semantic space to obtain the entity image feature matrix j indicates that the image sample is the jth image sample in the unpaired image-text dataset. is the image sample after segmentation k r The image unimodal features of entities; the text samples in the unpaired image-text dataset are divided into k w entity words and embed them into the entity common semantic space to obtain the entity text feature matrix i means that the text sample is the i-th text sample in the unpaired image-text dataset. is the k after the text sample is segmented w The text unimodal features of entity words; in the entity common semantic space, calculate T i U and The similarity matrix C between ij , calculated as follows: Where, T i U ' indicates T i U The transpose of Then for C ij Perform maximum pooling in the row direction and mean pooling in the column direction, and define the image-text similarity between the jth image sample and the i-th text sample as The calculation is as follows: C m =max{C mi ,i=1,...,k r } In the formula, C m Represents C ij The result of the maximum pooling on the mth row, k r With k w They are With T i U Length; In the unpaired image-text dataset, for all images, similarity is calculated with all texts according to the image-text similarity calculation method, and the text with the highest similarity is selected as the pseudo-pairing sample of the image to form a pseudo-pairing set Define the image at the same time Advantage score l of the retrieved text j For images Similarity and image to the most similar text The difference in similarity with the second highest similarity text is used to sample the advantage scores of all images, fit them through a two-component Gaussian mixture model, and optimize the two-component Gaussian mixture model through the expectation maximization algorithm to obtain the probability density function p(l). The formula of p(l) is as follows: In the formula, represents the probability density of the kth component, and the component with a higher average value is set as the component of reliable pseudo-pairing. k = 1 is defined as a reliable pseudo-pairing component, k = 2 is defined as an unreliable pseudo-pairing component, l is the advantage score of the image retrieval text, β k represents the probability density coefficient of component k; According to the probability density function according to the advantage score l j calculate The posterior probability p(k=1|l j ), calculated as follows: In the formula, p(l j ) is the total probability density calculated by the probability density function p(l), p(k=1) is the weight of the first component in p(l), namely β1, p(l j |k=1) is the probability density calculated from the first component in p(l); From the pseudo-pairing set Select pseudo-pairings with posterior probability greater than 0.5 to form a high-confidence image-text pseudo-pairing set Where V Pi Represents the image, T Pi Represents text, N P D P The number of pseudo-pairs included.

5. The unpaired cross-modal retrieval method based on entity semantic guidance and relationship fusion according to claim 4 is characterized in that: In step 4), the high-confidence image-text pseudo-pairing set D is aligned by embedding P , and use the relation fusion module to capture the relationship between entities in the sample, as follows: First, D is transformed into P Image V in Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an MLP network to obtain the entity image hidden layer feature matrix Indicates the k contained in the image r' The image hidden layer features of entities are obtained through the relationship fusion module to obtain the image fusion feature matrix The calculation is as follows: In the formula, is the number of k contained in the image r' The image fusion features of entities, W HV is the entity image similarity weight matrix, the element in the jth row and kth column of the matrix Represents the similarity weight between the j-th entity image and the k-th entity image, Given, and is the image hidden layer feature of the entity; By using the word-by-word segmentation method, D P Text in T Pi It is divided into multiple entities and embedded into the entity common semantic space, and then input into an LSTM network to obtain the entity text hidden layer feature matrix Indicates the k contained in the text i' The text hidden layer features of entities are obtained through the relation fusion module to obtain the text fusion feature matrix The calculation is as follows: In the formula, is the k contained in the text i' The text fusion feature of entities, W HT is the entity text similarity weight matrix, the element in the jth row and kth column of the matrix The similarity weight between the j-th entity text and the k-th entity text is given by Given, and is the textual hidden layer feature of the entity; Finally, the fusion feature alignment loss L is calculated based on the triplet loss. fusion The pseudo-paired samples are brought closer together and the non-pseudo-paired samples are moved away from each other to achieve the alignment of the image fusion features and the text fusion features. The calculation method is as follows: In the formula, and They are and is the hardest negative sample, β is the marginal hyperparameter, S F (·) is the cross-modal feature similarity measurement function, which is the result of row-wise maximum pooling and column-wise mean pooling of the similarity matrix between image and text; Through L fusion Finally, the fused public semantic space is obtained.

Images