A method and device for image-text retrieval based on location information and confidence perception

Abstract

The present invention discloses a method and device for image-text retrieval based on position information and confidence perception, comprising the following steps: obtaining an object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved; when the object to be retrieved is an image to be retrieved, extracting image region features of the image to be retrieved; extracting text word features from each text information in a preset corpus; generating sentence-level text features using the text word features; calculating a first confidence level between the image region features and the sentence-level text features; determining text information corresponding to the image to be retrieved based on the first confidence level; when the object to be retrieved is a text to be retrieved, extracting word features of the text to be retrieved; extracting local image features corresponding to the word features from each image in a preset image library; generating global image features using the local image features; calculating a second confidence level between the text to be retrieved and the global image features; and determining the image corresponding to the text to be retrieved based on the second confidence level. The method improves the performance of cross-modal retrieval.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval technology, and in particular to a method and device for image and text retrieval based on location information and confidence perception. Background Art

[0002] With the continuous development of smart devices and social networks, multimedia materials on the Internet are experiencing explosive growth in fields such as digital libraries, intellectual property, healthcare, fashion design, e-commerce, environmental monitoring, geo-information systems, communications systems, and military systems. These multimedia materials include text, images, videos, and audio in various modalities. Although they have different formats, they are semantically related to each other.

[0003] Traditional information retrieval methods rely on unimodal retrieval, meaning that the query set for each retrieval is of the same modality, such as text retrieval for text, image retrieval for images, and video retrieval for videos. In image retrieval, unimodal retrieval techniques primarily include methods such as keyword-based retrieval, retrieval based on underlying image features, and retrieval based on semantic models. While these methods can achieve good retrieval results, the information they retrieve is limited to data from a single modality. However, with the growth and increasing complexity of multimedia data, unimodal retrieval can no longer meet the demand for efficient, comprehensive, and accurate information retrieval.

[0004] When faced with a vast amount of interconnected multimedia data, people urgently need to find related auxiliary materials in other modalities within this modal data, such as extracting relevant text information from images or extracting relevant images from text. Because text and images belong to different modalities, the process of searching this multimodal data is called cross-modal retrieval.

[0005] However, images and text have different underlying features: color, texture, and shape for images, while words, phrases, and sentences for text. It's easy to see that images and text have different representations and distributions, and there's a significant heterogeneity gap between the data features of the two modalities, making it impossible to directly compare the similarities between the two modalities.

[0006] Current work addressing these challenges primarily involves global, coarse-grained retrieval methods and local, fine-grained retrieval methods. Global, coarse-grained retrieval primarily extracts holistic representations from the entire image and complete sentence, then projects these representations end-to-end into a constructed shared subspace. The similarity between visual and text embeddings can be directly calculated in the subspace using a similarity function. Early benchmarks for learning common spaces were canonical correlation analysis (CCA), which encodes cross-modal data into a highly correlated common subspace through linear projection. DCCA, on the other hand, learns to maximize the correlation between image and text representations by stacking multiple nonlinear transformation layers. Subsequently, many researchers introduced DNNs into the mapping process, combining DNNs with deep canonical correlation analysis (CCA), or proposed encoding images and text using CNNs and LSTMs, respectively. Because CNNs and LSTMs both have excellent representational capabilities, they can provide stronger feature representations for images and text, thereby improving the performance of related models. Subsequently, VSE++ introduced the concept of hard negatives, which served as the foundation for much subsequent research.

[0007] While global, coarse-grained retrieval methods can calculate similarity through mapping, they cannot extract extensive information from images and text. To better address visual semantic differences, subsequent research has introduced local matching algorithms. Compared to traditional CNNs, regional image-text matching algorithms are more suitable for detecting objects in images using object detection. Furthermore, the text encoder outputs a word feature matrix rather than a global sentence vector, enabling more accurate matching of details between images and sentences based on local matching algorithms. A method was proposed to detect objects in images and encode them into subspaces, where pairwise image-text similarity is calculated by summing the similarities of all region-word pairs. Furthermore, SCAN introduced a bottom-up attention scheme and used a pre-trained Faster R-CNN to encode images into region-level features and text into word-level features.

[0008] However, existing attention-based methods only focus on regional relationships, and pay less attention to the relationship between regional objects and global concepts. When the salient regions of an image are viewed separately, their visual semantics are fragmented, so that the local region-word may be inconsistent with the global image-text semantics. In addition, most existing methods ignore the relative position information of the salient regions in the image. For example, the target in the center of the image is often more important than the target at the edge of the image. Therefore, it is difficult to model the complex correspondence between data of different modalities well, which limits the performance of cross-modal retrieval models to a certain extent. Summary of the Invention

[0009] The present invention provides a method and device for image and text retrieval based on location information and confidence perception, which are used to solve the technical problem of poor performance of existing cross-modal retrieval methods.

[0010] The present invention provides a method for image and text retrieval based on location information and confidence perception, comprising:

[0011] Obtaining an object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved;

[0012] When the object to be retrieved is an image to be retrieved, extracting image region features of the image to be retrieved;

[0013] Extracting text word features from each text information in a preset corpus;

[0014] Generating sentence-level text features using the text word features;

[0015] Calculating a first confidence level of the image region feature and the sentence-level text feature;

[0016] Determining text information corresponding to the image to be retrieved according to the first confidence level;

[0017] When the object to be retrieved is a text to be retrieved, extracting word features of the text to be retrieved;

[0018] Extracting local image features corresponding to the word features from each image in a preset image library;

[0019] Generating global image features using the local image features;

[0020] Calculating a second confidence level between the text to be retrieved and the global image feature;

[0021] An image corresponding to the text to be retrieved is determined according to the second confidence level.

[0022] Optionally, the step of extracting image region features of the image to be retrieved includes:

[0023] Dividing the image to be retrieved into multiple regions and extracting visual features of each region;

[0024] Obtaining a location index for each area, and using the location index to generate location information for each area;

[0025] The visual features and position information corresponding to each region are spliced ​​together to obtain the image region features of each region.

[0026] Optionally, the step of extracting text word features from each text information in the preset corpus includes:

[0027] Split each text information of the preset corpus into several words;

[0028] Get the forward word vector and backward word vector for each word;

[0029] Obtaining a first hidden state of the forward word vector and a second hidden state of the backward word vector;

[0030] An average of the first hidden state and the second hidden state is calculated to obtain a text word feature of the word.

[0031] Optionally, the step of generating sentence-level text features using the text word features includes:

[0032] Obtaining an attention score for the text word feature;

[0033] The attention score is used as a weighting coefficient to perform weighted averaging on the text word features to obtain sentence-level text features.

[0034] Optionally, the step of calculating a first confidence level of the image region feature and the sentence-level text feature includes:

[0035] Calculating a similarity matrix between each of the image region features and each of the text word features;

[0036] Normalizing the similarity matrix to obtain a normalized similarity matrix;

[0037] Generating a similarity score between each of the image region features and each of the text word features according to the normalized similarity matrix;

[0038] Taking the similarity scores as weights, performing weighted averaging on the text word features to obtain semantically relevant text that matches the image region features;

[0039] Calculating a local semantic similarity vector between the image region feature and the semantically related text;

[0040] Calculating a global image feature vector of the image to be retrieved and a global text feature vector of the text information;

[0041] Calculating a global semantic similarity vector between the global image feature vector and the global text feature vector;

[0042] The local semantic similarity vector and the global semantic similarity vector are used to calculate a first confidence of the image region feature and the sentence-level text feature.

[0043] The present invention also provides a graphic and text retrieval device based on location information and confidence perception, comprising:

[0044] The object to be retrieved acquisition module is used to acquire the object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved;

[0045] An image region feature extraction module, configured to extract image region features of an image to be retrieved when the object to be retrieved is an image to be retrieved;

[0046] A text word feature extraction module is used to extract text word features from each text information in a preset corpus;

[0047] A sentence-level text feature generation module, configured to generate sentence-level text features using the text word features;

[0048] a first confidence calculation module, configured to calculate a first confidence of the image region feature and the sentence-level text feature;

[0049] a text information determination module, configured to determine text information corresponding to the image to be retrieved according to the first confidence level;

[0050] A word feature extraction module is used to extract word features of the text to be retrieved when the object to be retrieved is the text to be retrieved;

[0051] A local image feature extraction module is used to extract local image features corresponding to the word features from each image in a preset image library;

[0052] A global image feature generation module, configured to generate global image features using the local image features;

[0053] a second confidence calculation module, configured to calculate a second confidence between the text to be retrieved and the global image feature;

[0054] An image retrieval module is used to determine the image corresponding to the text to be retrieved according to the second confidence level.

[0055] Optionally, the image region feature extraction module includes:

[0056] A visual feature extraction submodule, configured to divide the image to be retrieved into a plurality of regions and extract visual features of each of the regions;

[0057] A location information generation submodule, configured to obtain a location index for each area and generate location information for each area using the location index;

[0058] The stitching submodule is used to stitch the visual features and position information corresponding to each region to obtain the image region features of each region.

[0059] Optionally, the text word feature extraction module includes:

[0060] The splitting submodule is used to split each text information of the preset corpus into several words;

[0061] The word vector acquisition submodule is used to obtain the forward word vector and backward word vector of each word;

[0062] A hidden state acquisition submodule, configured to acquire a first hidden state of the forward word vector and a second hidden state of the backward word vector;

[0063] The text word feature calculation submodule is used to calculate the average value of the first hidden state and the second hidden state to obtain the text word feature of the word.

[0064] The present invention also provides a device, comprising a processor and a memory:

[0065] The memory is used to store program code and transmit the program code to the processor;

[0066] The processor is used to execute the image-text retrieval method based on location information and confidence perception as described in any one of the above items according to the instructions in the program code.

[0067] The present invention also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the image-text retrieval method based on location information and confidence perception as described above.

[0068] It can be seen from the above technical solution that the present invention has the following advantages: the present invention improves the performance of cross-modal retrieval by obtaining an object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved; when the object to be retrieved is an image to be retrieved, extracting image region features of the image to be retrieved; extracting text word features from each text information in a preset corpus; generating sentence-level text features using text word features; calculating a first confidence level of image region features and sentence-level text features; determining text information corresponding to the image to be retrieved based on the first confidence level; when the object to be retrieved is a text to be retrieved, extracting word features of the text to be retrieved; extracting local image features corresponding to the word features from each image in a preset image library; generating global image features using local image features; calculating a second confidence level of the text to be retrieved and the global image features; and determining the image corresponding to the text to be retrieved based on the second confidence level. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 A schematic diagram of the structure of a cross-modal retrieval model provided by an embodiment of the present invention;

[0071] Figure 2 A flowchart of the steps of a method for image and text retrieval based on location information and confidence perception provided by an embodiment of the present invention;

[0072] Figure 3 A structural block diagram of a graphic and text retrieval device based on location information and confidence perception provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The embodiments of the present invention provide a method and device for image and text retrieval based on location information and confidence perception, which are used to solve the technical problem of poor performance of existing cross-modal retrieval methods.

[0074] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0075] See also Figure 1 , Figure 1 A structural diagram of a cross-modal retrieval model provided for an embodiment of the present invention; the cross-modal retrieval model may include a feature extraction part and a semantic alignment part, wherein the feature extraction part includes an image visual feature extraction module, an image position information aggregation module and a text semantic feature extraction module; the image visual feature extraction module is used to obtain regional features of the image; the image position information aggregation module is used to aggregate the image region position information into the image region features; the text semantic feature extraction module is used to obtain features of words contained in the text; the image region features and the text word features constitute high-level semantic features of different modalities.

[0076] The semantic alignment part includes an attention module and a confidence matching module; the attention module is used to map the high-level semantic features of data from different modalities into the same common latent space, and assign different attention scores to image region features or text word features through the attention mechanism. The attention score is used as a weighting coefficient to perform weighted averaging on the corresponding features to obtain semantically relevant text vectors matching the image region or semantically relevant image region vectors matching the text words.

[0077] The cross-modal retrieval model can be trained using an image library and a corresponding corpus as training sets. During the training process, a triplet loss can be used as the objective function. This objective function is used in the field of cross-modal retrieval to narrow the distance between similar samples and push the distance between heterogeneous samples away. It has been proven to maximize the relevance score of related image-text pairs and minimize the relevance score of irrelevant image-text pairs. Specifically, based on the ranking of the true matching positive samples in the returned results, different penalty weights are assigned to each triplet to train the cross-modal retrieval model and finally obtain the optimal solution of the model. The loss function is defined as:

[0078]

[0079] Where [x] + =max(x,0), β is the margin, and in order to ensure the efficiency of the training process, the training set is divided into multiple mini-batches for training, S(I,T * ) and S(I * ,T) represent the hardest negative pair in each mini-batch, that is, the negative pair closest to the positive pair.

[0080] After training, the object to be retrieved can be input into the cross-modal retrieval model to obtain the corresponding retrieval results.

[0081] based on Figure 1 The present invention provides a cross-modal retrieval model based on location information and confidence perception.

[0082] See also Figure 2 , Figure 2 A flowchart of the steps of a method for image and text retrieval based on location information and confidence perception provided by an embodiment of the present invention.

[0083] The present invention provides a flowchart of a method for image and text retrieval based on location information and confidence perception, which may specifically include the following steps:

[0084] Step 201: Obtain an object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved;

[0085] In an embodiment of the present invention, the object to be retrieved in cross-modal retrieval can be an image to be retrieved or a text to be retrieved. When the object to be retrieved is an image to be retrieved, the retrieval target is text. When the object to be retrieved is text to be retrieved, the retrieval target is an image.

[0086] Step 202: when the object to be retrieved is an image to be retrieved, extracting image region features of the image to be retrieved;

[0087] In an embodiment of the present invention, when the object to be retrieved is an image to be retrieved, the cross-modal retrieval model may first extract image region features of the image to be retrieved, and retrieve corresponding text through the image region features.

[0088] In one example, the step of extracting image region features of the image to be retrieved may include the following sub-steps:

[0089] S2021, dividing the image to be retrieved into multiple regions and extracting visual features of each region;

[0090] S2022, obtaining a location index of each area, and using the location index to generate location information of each area;

[0091] S2023: Combine the visual features and position information corresponding to each region to obtain image region features of each region.

[0092] In the specific implementation, the FasterR-CNN model can be used to detect objects in the image to be retrieved. In order to obtain better feature representation, the detected objects can be input into the pre-trained ResNet-152 to extract visual features and obtain the D-dimensional global feature vector v g In local feature extraction, for each input image I, a top-up attention method is used to extract the feature expression of M regions, and then a fully connected layer is added to convert the output dimension into a D-dimensional vector as the local feature expression V = {v1,v2,···,v m}, v i ∈R D , where v i Represents the visual features of a local area, where D = 2048 dimensions.

[0093] After collecting the visual features of the image to be retrieved, the position information of the image to be retrieved can be obtained, and the position information and the visual features can be combined to obtain the image region features of each region of the image to be retrieved.

[0094] In one example, given an image I = {v1, v2, ···, v m}, v i ∈R D , in order to display the area vi The relative position in the entire image I can be first divided into K×K blocks B, and each block is used as the basic position unit. The position of each block is initially indexed by k∈[1,K 2 ] to represent it. First, define a vector a for the region vi i ∈R z Used to distinguish the importance of different locations, and then according to the area v i Overlap with the fixed block to locate the region v i Position in the entire image I: Let p i ∈R z Represents region v i The position index vector is defined as the position index vector corresponding to the region v i The indices of the Z blocks that overlap.

[0095] where p ij ∈[1,K 2 ] represents the block index of the jth block that overlaps with region vi, overlap(v i ,b q ) represents the intersection pixels of region vi and the qth block.

[0096]

[0097] Then the area p i The final position representation and visual feature v i Stitching together to allow the regional features to carry position information:

[0098]

[0099] The final region is represented by Projection to D-dimensional features , as the image region feature of the region.

[0100] Step 203: extracting text word features from each text information in the preset corpus;

[0101] In an embodiment of the present invention, to achieve cross-modal retrieval, it is necessary to establish an association relationship between different modalities. The present invention is based on features to achieve the connection between the two modalities of image and text.

[0102] In one example, step 203 may include the following sub-steps:

[0103] S2031, splitting each text information of the preset corpus into a plurality of words;

[0104] S2032, obtaining the forward word vector and the backward word vector of each word;

[0105] S2033, obtaining a first hidden state of the forward word vector and a second hidden state of the backward word vector;

[0106] S2034, calculating the average of the first hidden state and the second hidden state to obtain the text word feature of the word.

[0107] In a specific implementation, text features can be obtained through a recurrent neural network, where the recurrent neural network can be any one of an LSTM network and a GRU network.

[0108] Taking the bidirectional GRU network (Bi-GRU) as an example, the Bi-GRU architecture is improved on the basis of GRU and consists of a forward GRU and a backward GRU. For a text sentence T, assuming that the number of words it contains is N, a GRU-based text encoder is used to map each word into a D-dimensional word vector: e j =W e w j , j∈{1,···,N}.

[0109] By summarizing the information in two directions in the sentence, a bidirectional GRU is used to map the word vector and the context semantics to obtain the final text word features. The bidirectional GRU consists of a forward GRU and a backward GRU. The forward GRU follows the direction from w1 to w N Read the sentences in the following order: The reverse GRU is in the opposite order: in, and Represent the hidden states generated from the forward GRU and the reverse GRU respectively. The text feature of each word is represented as the average of the hidden states of the forward GRU and the reverse GRU: The global feature vector t of the full text T g Expressed as: The weight w j It is t j With t ave The normalized similarity between

[0110] Step 204, generating sentence-level text features using text word features;

[0111] After obtaining the text word features of each text information in the corpus, sentence-level text features corresponding to each image region feature can be generated.

[0112] In one example, step 204 may include the following sub-steps:

[0113] S2041, obtaining the attention score of the text word feature;

[0114] S2042: Use the attention score as a weighting coefficient to perform weighted averaging on the text word features to obtain sentence-level text features.

[0115] In the specific implementation, the high-level semantic features of data from different modalities can be mapped into the same common latent space, and the attention score can be assigned to the text word features through the attention mechanism. The attention score is used as the weighting coefficient to perform weighted averaging on the corresponding features to obtain sentence-level text features that match the image area.

[0116] Step 205, calculating a first confidence level of the image region feature and the sentence-level text feature;

[0117] In this embodiment of the present invention, step 205 may include the following sub-steps:

[0118] S2051, calculating the similarity matrix between each image region feature and each text word feature;

[0119] S2052, normalizing the similarity matrix to obtain a normalized similarity matrix;

[0120] S2053, generating a similarity score between each image region feature and each text word feature according to the normalized similarity matrix;

[0121] S2054, using the similarity score as a weight, performing weighted averaging on the text word features to obtain semantically relevant text that matches the image region features;

[0122] S2055, calculating the local semantic similarity vector between the image region feature and the semantically related text;

[0123] S2056, calculating the global image feature vector of the image to be retrieved and the global text feature vector of the text information;

[0124] S2057, calculating a global semantic similarity vector between the global image feature vector and the global text feature vector;

[0125] S2058: Calculate the first confidence of the image region feature and the sentence-level text feature using the local semantic similarity vector and the global semantic similarity vector.

[0126] Confidence refers to the degree of consistency between the visual semantics of each region and the global image-text perspective. This can filter out region-word matches that are inconsistent with the global semantics. Match confidence is first inferred based on the degree to which the local similarity between an image region and its matching semantically related text is included in the global image-text semantic similarity, that is, the relative degree to which the region is truly described by the text.

[0127] In the specific implementation, for an image-text pair, all regions in the image and all words in the text are combined into region-word pairs, and their similarity matrix is ​​calculated using the cosine similarity function, that is:

[0128]

[0129] in, Represents the image region feature that carries location information, e j Represents text word features.

[0130] Furthermore, the similarity matrix is ​​normalized, that is:

[0131]

[0132] Among them, [sim(i,j)] + =max{[sim(i,j)],0}

[0133] Furthermore, the normalized similarity score is used as the attention score of the text word feature. The more semantic information a word contains, the higher the attention score will be.

[0134] Furthermore, the obtained attention score is used as a new weighting coefficient for each local feature, namely the correlation score. With the correlation score as the weight, all local features are weighted averaged to obtain the correlation coefficient with the image region feature. Matching semantically related text

[0135]

[0136] Here, i∈[1,m], and the hyperparameter α is the inverse temperature parameter in the softmax function, which controls the distribution of attention. To smooth the probability distribution of the softmax output and avoid falling into local optimal solutions during training, the inverse temperature parameter α is set to 9.0.

[0137] Next, the local semantic similarity vector is calculated using the image region features and semantically related text. In order to depict the detailed alignment relationship between vision and language, cross-modal alignment of vision and semantics, a normalized distance vector is used to represent the semantic similarity between heterogeneous modalities. Specifically, the image region and the semantically related text it matches The local semantic similarity vector between Calculated as:

[0138]

[0139] in is the learnable parameter matrix.

[0140] Then measure the global image feature vector v of the entire image I g and the global text feature vector t of the full text T g The global semantic similarity vector of:

[0141]

[0142] in is the learnable parameter matrix.

[0143] Then use the global semantic similarity vector s g and To measure the first confidence of the corresponding area:

[0144]

[0145] Where W n ∈R 1×p is a learnable parameter vector, and ⊙ refers to the corresponding element-wise multiplication operation. The first confidence is inferred from the extent to which the local similarity between the image region and its matched semantically related text is included in the global semantic similarity of the image-text. It indicates the relative degree to which the region is truly described from a global image-text perspective.

[0146] Step 206: determining text information corresponding to the image to be retrieved according to the first confidence level;

[0147] In a specific implementation, untrustworthy local alignment information that is inconsistent with the global semantics can be filtered out in the overall relevance aggregation according to the first confidence level. The semantic relevance between image I and text T is:

[0148]

[0149] According to the semantic relevance, the text information corresponding to the image to be retrieved can be obtained.

[0150] Step 207, when the object to be searched is a text to be searched, extracting word features of the text to be searched;

[0151] Step 208: extracting local image features corresponding to word features from each image in the preset image library;

[0152] Step 209, using local image features to generate global image features;

[0153] Step 210, calculating a second confidence level of the text to be retrieved and the global image feature;

[0154] Step 211: Determine the image corresponding to the text to be retrieved according to the second confidence level.

[0155] When the object to be retrieved is a text to be retrieved, the method adopted is similar to that of the image to be retrieved. For details, please refer to steps 202 to 206, which will not be repeated here.

[0156] The present invention improves the performance of cross-modal retrieval by obtaining an object to be retrieved; when the object to be retrieved is an image to be retrieved or a text to be retrieved; when the object to be retrieved is an image to be retrieved, extracting image region features of the image to be retrieved; extracting text word features from each text information in a preset corpus; generating sentence-level text features using the text word features; calculating a first confidence level between the image region features and the sentence-level text features; determining text information corresponding to the image to be retrieved based on the first confidence level; when the object to be retrieved is a text to be retrieved, extracting word features of the text to be retrieved; extracting local image features corresponding to the word features from each image in a preset image library; generating global image features using the local image features; calculating a second confidence level between the text to be retrieved and the global image features; and determining the image corresponding to the text to be retrieved based on the second confidence level.

[0157] See also Figure 3 , Figure 3 A structural block diagram of a graphic and text retrieval device based on location information and confidence perception provided by an embodiment of the present invention.

[0158] An embodiment of the present invention provides a graphic and text retrieval device based on location information and confidence perception, comprising:

[0159] The object to be retrieved acquisition module 301 is used to acquire the object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved;

[0160] The image region feature extraction module 302 is used to extract the image region features of the image to be retrieved when the object to be retrieved is the image to be retrieved;

[0161] A text word feature extraction module 303 is used to extract text word features from each text information in a preset corpus;

[0162] A sentence-level text feature generation module 304 is used to generate sentence-level text features using text word features;

[0163] A first confidence calculation module 305, configured to calculate first confidences of image region features and sentence-level text features;

[0164] A text information determination module 306, configured to determine text information corresponding to the image to be retrieved according to the first confidence level;

[0165] The word feature extraction module 307 is used to extract the word features of the text to be retrieved when the object to be retrieved is the text to be retrieved;

[0166] A local image feature extraction module 308 is used to extract local image features corresponding to word features from each image in a preset image library;

[0167] A global image feature generation module 309 is configured to generate global image features using local image features;

[0168] A second confidence calculation module 310, configured to calculate a second confidence between the text to be retrieved and the global image feature;

[0169] The image retrieval module 311 is configured to determine an image corresponding to the text to be retrieved according to the second confidence level.

[0170] In this embodiment of the present invention, the image region feature extraction module 302 includes:

[0171] The visual feature extraction submodule is used to divide the image to be retrieved into multiple regions and extract the visual features of each region;

[0172] The location information generation submodule is used to obtain the location index of each area and generate the location information of each area using the location index;

[0173] The stitching submodule is used to stitch the visual features and position information corresponding to each region to obtain the image region features of each region.

[0174] In this embodiment of the present invention, the text word feature extraction module 303 includes:

[0175] The splitting submodule is used to split each text information of the preset corpus into several words;

[0176] The word vector acquisition submodule is used to obtain the forward word vector and backward word vector of each word;

[0177] A hidden state acquisition submodule is used to obtain the first hidden state of the forward word vector and the second hidden state of the backward word vector;

[0178] The text word feature calculation submodule is used to calculate the average value of the first hidden state and the second hidden state to obtain the text word feature of the word.

[0179] In this embodiment of the present invention, the sentence-level text feature generation module 304 includes:

[0180] The attention score acquisition submodule is used to obtain the attention scores of text word features;

[0181] The sentence-level text feature generation submodule is used to use the attention score as the weighting coefficient to perform weighted averaging on the text word features to obtain sentence-level text features.

[0182] In this embodiment of the present invention, the first confidence calculation module 305 includes:

[0183] A similarity matrix calculation submodule is used to calculate the similarity matrix between each image region feature and each text word feature;

[0184] The normalization submodule is used to normalize the similarity matrix to obtain a normalized similarity matrix;

[0185] A similarity score generation submodule is used to generate a similarity score between each image region feature and each text word feature according to the normalized similarity matrix;

[0186] The semantically relevant text acquisition submodule is used to perform weighted averaging of text word features using similarity scores as weights to obtain semantically relevant text that matches the image region features;

[0187] Local semantic similarity vector calculation submodule, used to calculate the local semantic similarity vector between image region features and semantically related text;

[0188] A global text feature vector calculation submodule is used to calculate the global image feature vector of the image to be retrieved and the global text feature vector of the text information;

[0189] A global semantic similarity vector calculation submodule is used to calculate the global semantic similarity vector between the global image feature vector and the global text feature vector;

[0190] The first confidence calculation submodule is used to calculate the first confidence of the image region feature and the sentence-level text feature using the local semantic similarity vector and the global semantic similarity vector.

[0191] An embodiment of the present invention further provides a text-based entity recognition device, the device including a processor and a memory:

[0192] The memory is used to store program codes and transmit the program codes to the processor;

[0193] The processor is used to execute the image-text retrieval method based on location information and confidence perception according to the instructions in the program code.

[0194] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the image-text retrieval method based on location information and confidence perception of an embodiment of the present invention.

[0195] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0196] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0197] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0202] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0203] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image and text retrieval based on location information and confidence perception, characterized in that: include: Obtaining an object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved; When the object to be retrieved is an image to be retrieved, extracting image region features of the image to be retrieved; Extracting text word features from each text information in a preset corpus; Generating sentence-level text features using the text word features; Calculating a first confidence level of the image region feature and the sentence-level text feature; Determining text information corresponding to the image to be retrieved according to the first confidence level; When the object to be retrieved is a text to be retrieved, extracting word features of the text to be retrieved; Extracting local image features corresponding to the word features from each image in a preset image library; Generating global image features using the local image features; Calculating a second confidence level between the text to be retrieved and the global image feature; Determining an image corresponding to the text to be retrieved according to the second confidence level; The step of extracting the image region features of the image to be retrieved includes: Dividing the image to be retrieved into multiple regions and extracting visual features of each region; Obtaining a location index for each area, and using the location index to generate location information for each area; Combine the visual features and position information corresponding to each region to obtain the image region features of each region; The step of calculating the first confidence of the image region feature and the sentence-level text feature includes: Calculating a similarity matrix between each of the image region features and each of the text word features; Normalizing the similarity matrix to obtain a normalized similarity matrix; Generating a similarity score between each of the image region features and each of the text word features according to the normalized similarity matrix; Taking the similarity scores as weights, performing weighted averaging on the text word features to obtain semantically relevant text that matches the image region features; Calculating a local semantic similarity vector between the image region feature and the semantically related text; Calculating a global image feature vector of the image to be retrieved and a global text feature vector of the text information; Calculating a global semantic similarity vector between the global image feature vector and the global text feature vector; The local semantic similarity vector and the global semantic similarity vector are used to calculate a first confidence of the image region feature and the sentence-level text feature.

2. The method according to claim 1, characterized in that The step of extracting text word features from each text information in the preset corpus includes: Split each text information of the preset corpus into several words; Get the forward word vector and backward word vector for each word; Obtaining a first hidden state of the forward word vector and a second hidden state of the backward word vector; An average of the first hidden state and the second hidden state is calculated to obtain a text word feature of the word.

3. The method according to claim 1, characterized in that The step of generating sentence-level text features using the text word features comprises: Obtaining an attention score for the text word feature; The attention score is used as a weighting coefficient to perform weighted averaging on the text word features to obtain sentence-level text features.

4. A graphic and text retrieval device based on location information and confidence perception, characterized in that: include: The object to be retrieved acquisition module is used to acquire the object to be retrieved; the object to be retrieved is an image to be retrieved or a text to be retrieved; An image region feature extraction module, configured to extract image region features of an image to be retrieved when the object to be retrieved is an image to be retrieved; A text word feature extraction module is used to extract text word features from each text information in a preset corpus; A sentence-level text feature generation module, configured to generate sentence-level text features using the text word features; a first confidence calculation module, configured to calculate a first confidence of the image region feature and the sentence-level text feature; a text information determination module, configured to determine text information corresponding to the image to be retrieved according to the first confidence level; A word feature extraction module is used to extract word features of the text to be retrieved when the object to be retrieved is the text to be retrieved; A local image feature extraction module is used to extract local image features corresponding to the word features from each image in a preset image library; A global image feature generation module, configured to generate global image features using the local image features; a second confidence calculation module, configured to calculate a second confidence between the text to be retrieved and the global image feature; An image retrieval module, configured to determine an image corresponding to the text to be retrieved according to the second confidence level; Wherein, the image region feature extraction module includes: A visual feature extraction submodule, configured to divide the image to be retrieved into a plurality of regions and extract visual features of each of the regions; A location information generation submodule, configured to obtain a location index for each area and generate location information for each area using the location index; The stitching submodule is used to stitch the visual features and position information corresponding to each region to obtain the image region features of each region; The first confidence calculation module includes: A similarity matrix calculation submodule is used to calculate the similarity matrix between each image region feature and each text word feature; The normalization submodule is used to normalize the similarity matrix to obtain a normalized similarity matrix; A similarity score generation submodule is used to generate a similarity score between each image region feature and each text word feature according to the normalized similarity matrix; The semantically relevant text acquisition submodule is used to perform weighted averaging of text word features using similarity scores as weights to obtain semantically relevant text that matches the image region features; Local semantic similarity vector calculation submodule, used to calculate the local semantic similarity vector between image region features and semantically related text; A global text feature vector calculation submodule is used to calculate the global image feature vector of the image to be retrieved and the global text feature vector of the text information; A global semantic similarity vector calculation submodule is used to calculate the global semantic similarity vector between the global image feature vector and the global text feature vector; The first confidence calculation submodule is used to calculate the first confidence of the image region feature and the sentence-level text feature using the local semantic similarity vector and the global semantic similarity vector.

5. The device according to claim 4, characterized in that The text word feature extraction module includes: The splitting submodule is used to split each text information of the preset corpus into several words; The word vector acquisition submodule is used to obtain the forward word vector and backward word vector of each word; A hidden state acquisition submodule, configured to acquire a first hidden state of the forward word vector and a second hidden state of the backward word vector; The text word feature calculation submodule is used to calculate the average value of the first hidden state and the second hidden state to obtain the text word feature of the word.

6. A text-based entity recognition device, characterized in that The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the image-text retrieval method based on location information and confidence perception according to any one of claims 1-3 according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the image-text retrieval method based on location information and confidence perception as described in any one of claims 1-3.

Images