Image-text cross-modal retrieval method, device, electronic device, and storage medium

Abstract

The present invention provides a method, device, electronic device, and storage medium for cross-modal image and text retrieval. The method includes: screening samples in a preset image library and a preset text library to obtain image samples and text samples; the image samples and text samples include at least one entity target of the same type; extracting entities from the text sample to obtain entity targets of the text sample; segmenting the image sample to obtain M×N blocks of the image sample; calculating the weight of the entity targets within each block, and determining high-frequency entity targets of the image sample based on the weights; extracting text prompt information from each block and appending the text prompt information to the text corresponding to the entity target; extracting feature vectors from each block and the text corresponding to the entity target after the text prompt information is appended, and merging the feature vectors based on the feature vectors and the high-frequency entity targets. The present invention can determine the relative position of entities, further improving the accuracy of cross-modal retrieval.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval technology, and in particular to a method, device, electronic device and storage medium for cross-modal image and text retrieval. Background Art

[0002] The rapid development of mobile internet has led to an explosive growth in multimedia data such as text, images, audio, and video. Conventional search can no longer meet people's ever-changing search needs, giving rise to cross-modal search applications.

[0003] Currently, there are two main methods for cross-modal retrieval: (1) Cross-modal retrieval based on deep learning. Through deep learning algorithms, data from different data modalities can be mapped to a common low-dimensional vector space, thereby encoding the semantic information of the data into a vector representation. In this vector space, similar data points are close to each other, and dissimilar data points are far away, so that the semantic similarity calculation and matching between different modal data can be achieved. Deep learning technology can automatically learn feature representations to achieve multimodal data fusion. In cross-modal retrieval, deep learning technology can achieve the fusion of multimodal data by constructing a common representation layer. At the same time, deep learning technology can also optimize cross-modal retrieval performance by training the common representation layer. (2) Generative model based on natural encoder. The idea of ​​the generative model is to obtain the internal probability distribution of a type of data to generate similar data. It is a probabilistic model. The idea of ​​the natural encoder is extended by mapping features to the same space through the encoder to reduce the correlation learning error between different modalities, and then mapping the features back to the original space through the decoder to reduce the feature learning error within each modality. Most related technologies only focus on modeling rough data, resulting in the inability to accurately optimize the features of different modal data. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device, electronic device, and storage medium for cross-modal image-text retrieval to solve the problem of being unable to detect the relative positions of entities in an image during image-text retrieval.

[0005] In a first aspect, an embodiment of the present invention provides a method for cross-modal image and text retrieval, comprising:

[0006] Screening samples in a preset picture library and a preset text library to obtain picture samples and text samples; the picture samples and the text samples include at least the same entity target;

[0007] Performing entity extraction on the text sample to obtain entity targets of the text sample; segmenting the image sample to obtain M×N blocks of the image sample; calculating the weight of the entity target in each block respectively, and determining the high-frequency entity targets of the image sample according to the weight;

[0008] Extract text prompt information from each block and add the text prompt information to the text corresponding to the entity target; wherein the text prompt information includes the identifier of the block to which it belongs and the object tag generated by the block; respectively extract the feature vectors of the text corresponding to each block and the entity target after the added information, and merge them based on the feature vectors and the high-frequency entity target.

[0009] In a possible implementation, there is at least one entity target, and each entity target includes at least one entity target; and calculating the weights of the entity targets in each block and obtaining the high-frequency entity targets of the image sample according to the weights includes:

[0010] For each entity target, perform the following steps:

[0011] Detecting the area of ​​a first rectangular region containing the physical object in the image sample;

[0012] For each block, obtaining the area of ​​a second rectangular region containing the physical object in the block, dividing the area of ​​the second rectangular region by the area of ​​the first rectangular region to obtain a weight of the physical object in the block;

[0013] For each entity target, the weight of each entity target belonging to the entity target in each block is accumulated to obtain the weight of the entity target in the image sample;

[0014] The weight of each entity target in the image sample is sorted according to a sorting algorithm, and the entity target with the largest corresponding weight is used as the high-frequency entity target of the image sample.

[0015] In a possible implementation, extracting feature vectors of text corresponding to each block and the entity target after adding information includes:

[0016] Use the image encoder to extract features from each block and obtain the first feature vector of each block;

[0017] A text encoder is used to perform feature extraction on the text corresponding to the entity target after the information is added to obtain a second feature vector.

[0018] In a possible implementation, the merging based on the feature vector and the high-frequency entity target includes:

[0019] Encoding the first eigenvector and the second eigenvector, and mapping the encoded first eigenvector and the second eigenvector to a preset shared subspace;

[0020] Multiplying the first feature vector and the second feature vector after encoding in the shared subspace according to the constructed tensor to obtain text corresponding to each block and the entity target after cross-modal fusion;

[0021] Merge the blocks after the cross-modal fusion, the text corresponding to the entity target, and the high-frequency entity target.

[0022] In a possible implementation, extracting entities from the text sample to obtain entity targets of the text sample includes:

[0023] Extracting entities contained in the text in the preset text library to obtain a vocabulary of the preset text library;

[0024] Filtering entities in the vocabulary to obtain multiple high-frequency entities;

[0025] The multiple high-frequency entities are compared with the entities extracted from the text sample, and the entity target of the text sample is obtained according to the comparison result.

[0026] In a possible implementation, after segmenting the image sample to obtain M×N blocks of the image sample, the method further includes:

[0027] Input each block into the pre-trained neural network to obtain the feature map corresponding to each block;

[0028] Setting a region of interest for each point in the feature map, and obtaining at least one candidate region of interest based on the set region of interest;

[0029] Performing binary classification and bounding box regression on the at least one candidate region of interest according to a preset region proposal network to screen the at least one candidate region of interest;

[0030] According to the at least one candidate region of interest that has been screened, each block is matched with a pixel of the feature map, and the feature map is matched with a corresponding fixed feature;

[0031] According to the corresponding blocks and pixels of the feature map, as well as the feature map and the corresponding fixed features, the region of interest is classified, bounding box regression is performed, and mask generation is performed to obtain the entity target of each block.

[0032] In a possible implementation, the process of constructing the preset text library includes:

[0033] Acquire text from a text library, and remove noise from the text according to a regular expression;

[0034] Segment the noise-removed text into separate text units using a word segmentation tool;

[0035] Performing stop word recognition on the text unit using a preset stop word list, and deleting the recognized stop words in the text unit;

[0036] The text units after the stop words are deleted are processed according to preset text rules to obtain standardized text units, and a preset text library is obtained based on the standardized text units.

[0037] The process of constructing the preset image library includes: performing preprocessing operations on the images in the preset image library, wherein the preprocessing operations at least include cropping, scaling, denoising and enhancing.

[0038] In a second aspect, an embodiment of the present invention provides a cross-modal image-text retrieval device, comprising:

[0039] A data preprocessing module is used to screen samples in a preset image library and a preset text library to obtain image samples and text samples; the image samples and the text samples include at least the same entity target;

[0040] The first processing module is configured to perform entity extraction on the text sample to obtain entity targets of the text sample; segment the image sample to obtain M×N blocks of the image sample; calculate the weight of the entity targets in each block respectively, and determine the high-frequency entity targets of the image sample based on the weights;

[0041] The second processing module is used to extract the text prompt information of each block and add the text prompt information to the text corresponding to the entity target; wherein the text prompt information includes the identifier of the block to which it belongs and the object tag generated by the block; the feature vectors of the text corresponding to each block and the entity target after the added information are respectively extracted, and merged based on the feature vectors and the high-frequency entity target.

[0042] In a third aspect, an embodiment of the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method described in the first aspect or any possible implementation of the first aspect.

[0044] Embodiments of the present invention provide a method, device, electronic device, and storage medium for cross-modal image-text retrieval. The method comprises: screening samples in a preset image library and a preset text library to obtain image samples and text samples; the image samples and text samples contain at least one entity target of the same type; performing entity extraction on the text sample to obtain entity targets for the text sample; segmenting the image sample to obtain M×N blocks of the image sample; calculating the weight of the entity targets within each block, and determining the high-frequency entity targets of the image sample based on the weights; extracting text prompt information from each block and appending the text corresponding to the entity targets; wherein the text prompt information includes an identifier of the block to which it belongs and an object tag generated by the block; extracting feature vectors of the text corresponding to each block and the entity targets after appending the information, and merging the text based on the feature vectors and the high-frequency entity targets. The embodiment of the present invention segments the image, generates block information, and generates object tags to capture detailed information in the image, thereby improving the accuracy of cross-modal retrieval. Simultaneously, the text is processed, templates are added, and the relative positions of entities are determined, further improving the accuracy and efficiency of retrieval. By statistically analyzing the weights of entities after image segmentation, the method of judging entities based on the weights after image segmentation not only improves the accuracy and efficiency of retrieval, but also can effectively identify and distinguish the importance of entities in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a flowchart of an implementation method of a cross-modal image-text retrieval method provided by an embodiment of the present invention;

[0047] Figure 2 This is an architectural diagram of a method for extracting entity objects from image samples provided by an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of a segmented image provided by an embodiment of the present invention;

[0049] Figure 4 This is a framework diagram of cross-modal fusion provided by an embodiment of the present invention;

[0050] Figure 5 This is a schematic structural diagram of a method for cross-modal image and text retrieval provided by an embodiment of the present invention;

[0051] Figure 61 is a schematic structural diagram of a cross-modal image-text retrieval device provided by an embodiment of the present invention;

[0052] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0055] Figure 1 This is a flowchart of the implementation of the image-text cross-modal retrieval method provided by the embodiment of the present invention, referring to Figure 1 , as detailed below:

[0056] In step 101, sample screening is performed in a preset picture library and a preset text library to obtain picture samples and text samples; the picture samples and the text samples include at least the same entity target.

[0057] In this embodiment, when samples are randomly screened, the instance construction method adopted is a method of constructing examples by randomly extracting B samples from A different categories, and obtaining the number of batch samples E=A×B. For example, a certain number n1 of picture samples are screened from the picture library. Using the instance construction method, b samples are randomly extracted from a different categories, and the number of picture samples obtained is n1=a×b. A certain number n2 of text samples are screened from the text library. Using the instance construction method, d samples are randomly extracted from c different categories, and the number of text samples obtained is n2=c×d. For example, the number n1 of picture texts screened from the picture library is equal to the number n2 of text samples screened from the text library. This embodiment can perform multiple constructions when the number of samples is small, thereby improving the training rate and accuracy.

[0058] In this embodiment, the screened image samples include many types of entity targets. The corresponding text samples also include at least one entity target. At least one entity target in the text corresponds to an entity target in the image, facilitating subsequent matching. If, during subsequent training, it is found that the text samples extracted from the text library are completely inconsistent with the entity types identified in the image samples, the process returns to this step to continue screening text samples until at least a consistent text sample is obtained and subsequent training is completed.

[0059] In step 102, entity extraction is performed on the text sample to obtain entity targets of the text sample; and the image sample is segmented to obtain M×N blocks of the image sample.

[0060] In this embodiment, the Language Technology Platform (LTP) tool is used to perform named entity recognition technology to identify entities in the text, such as names of people, places, and organizations. The text library after text preprocessing is a new vocabulary, which serves as a benchmark for subsequent processes.

[0061] In this embodiment, the image samples are segmented into M×N blocks, where M and N are both integers greater than 0, and M=N or M≠N.

[0062] In some embodiments, performing entity extraction on a text sample to obtain an entity target of the text sample includes:

[0063] Entities contained in the text in the preset text library are extracted to obtain the vocabulary of the preset text library.

[0064] Filter entities in the vocabulary to obtain multiple high-frequency entities.

[0065] Compare multiple high-frequency entities with the entities extracted from the text sample, and obtain the entity target of the text sample based on the comparison results.

[0066] In this embodiment, the LTP tool is used to extract entities contained in the text from the text library. Named entity recognition technology is used to identify entities in the text, such as names of people, places, and organizations. The identified text entities are formed into a vocabulary. The entities in the vocabulary are screened. The high-frequency entities of the entities in the vocabulary are judged by combining the text retrieval frequency obtained from big data collection and other factors. There can be many high-frequency entities. Extracting different numbers of samples from different categories for construction can improve the speed and accuracy of training with a small number of samples. By constructing samples of different categories, the diversity of training data can be increased, the risk of overfitting can be reduced, and the generalization and robustness of the model can be improved.

[0067] In some embodiments, after the image sample is segmented to obtain M×N blocks of the image sample, the method further includes:

[0068] Input each block into the pre-trained neural network to obtain the feature map corresponding to each block.

[0069] A region of interest is set for each point in the feature map, and at least one candidate region of interest is obtained according to the set region of interest.

[0070] According to the preset region proposal network, at least one candidate region of interest is subjected to binary classification and bounding box regression to screen at least one candidate region of interest.

[0071] According to at least one candidate region of interest that has been screened, each block is matched with a pixel of a feature map, and the feature map is matched with a corresponding fixed feature.

[0072] According to the pixels of the corresponding blocks and feature maps, as well as the feature maps and corresponding fixed features, the regions of interest are classified, bounding boxes are regressed, and masks are generated to obtain the entity targets of each block.

[0073] In this embodiment, if Figure 2 As shown, Figure 2 This is an architectural diagram of a method for extracting entity targets from image samples provided by an embodiment of the present invention. Each image is subjected to target detection using the above method, and the entity data contained in each image is obtained to divide the image into N×N blocks. The target object and background of each block of the image are separated. Each segmented block needs to be input into a pre-trained neural network to obtain the corresponding feature map. Subsequently, a predetermined region of interest (ROI) is set for each point in the feature map to obtain multiple candidate regions of interest. Finally, the candidate regions of interest are sent to the Region Proposal Network (RPN) for binary classification and bounding box regression to filter and select some regions of interest. Then, based on the filtered regions of interest, the pixels of the original image and the feature map are first matched, and then the feature map and the fixed features are matched. Finally, these regions of interest are classified, bounding box regressed, and masks (MASK) are generated. Ultimately, the recognition of the entity targets of the image samples is achieved.

[0074] In step 103 , the weights of the physical objects in each block are calculated respectively, and the high-frequency physical objects of the image samples are determined according to the weights.

[0075] In this embodiment, an image sample is divided into multiple blocks. Each block contains at least one physical object. The weight of the physical object in each block is calculated, and the weights of physical objects of the same type are accumulated. The at least one physical object with the highest accumulated value is determined as the high-frequency physical object for the image sample.

[0076] In some embodiments, there is at least one entity target, and each entity target includes at least one entity target. Step 103 includes:

[0077] Step 1: For each entity target, perform the following steps A and B:

[0078] Step A: Detect the area of ​​a first rectangular region containing the physical object in the image sample.

[0079] Step B: for each block, obtain the area of ​​a second rectangular region containing the physical target in the block, divide the area of ​​the second rectangular region by the area of ​​the first rectangular region, and obtain the weight of the physical target in the block.

[0080] Step 2: For each entity target, the weight of each entity target belonging to this type of entity target in each block is accumulated to obtain the weight of this type of entity target in the image sample.

[0081] Step 3: Sort the weight of each entity target in the image sample according to the sorting algorithm, and take the entity target with the largest corresponding weight as the high-frequency entity target of the image sample.

[0082] In this embodiment, a method of entity extraction and weighted statistics is used to establish a connection between images and text. The area of ​​each entity rectangle obtained from the initial object detection is used as the denominator. Object detection is then performed on each block obtained by image segmentation, and the area of ​​the same entity rectangle is used as the numerator. The resulting ratio is then used as the weight of each entity in the image. The highest weight for each entity is 1, meaning that a single entity appears completely within the segmented block.

[0083] For example, Figure 3 As stated, Figure 3 This is a schematic diagram of a segmented image provided by an embodiment of the present invention. Figure 3The image sample of the instance is divided into 9 blocks. The image sample includes multiple entity targets such as "tree" and "tower". Among them, there are many "trees" in the image sample, that is, there are many entity targets belonging to the category of "tree", and each tree is an entity target of the category "tree". There is only one "tower" in the image sample, so there is only one entity target belonging to the category of "tower". If the same entity target is recognized in the text sample, perform the following steps. Take the tree as an example. The target recognizes a tree, calculates the minimum enclosing rectangle area of ​​the tree in the entire image and uses it as the denominator, recognizes the minimum enclosing rectangle area of ​​the tree in block 4 and uses it as the numerator, and obtains the weight of the tree in block 4. Traverse each block to calculate the weight of the tree. And calculate the weights of several other trees, and accumulate the weights corresponding to each tree under the "tree" category. Then use the same method to calculate the weights of entity targets of other categories. And sort them to get the target entity with the largest weight as the highest identifier of the image sample. For example, after calculation, in Figure 3 In the example, "tree" is a high-frequency entity target in the image. It should be understood that the above examples are for illustrative purposes only and do not constitute a specific limitation. By statistically analyzing the weights of entities after image segmentation and sorting them according to a sorting algorithm, and using the weights as special identifiers, cross-modal retrieval can be achieved more efficiently. The method of determining entities based on weights after image segmentation not only improves the accuracy and efficiency of retrieval, but also effectively identifies and distinguishes the importance of entities in the image.

[0084] In step 104, the text prompt information of each block is extracted and added to the text corresponding to the entity target; wherein the text prompt information includes the identifier of the block to which it belongs and the object tag generated by the block.

[0085] In this embodiment, by segmenting the image, generating block information, and generating object tags, we can better capture detailed information in the image and improve the accuracy of cross-modal retrieval. Furthermore, by processing the text, adding templates, and determining the relative positions of entities, we can further improve the accuracy and efficiency of retrieval. This method of using positional information to indicate the relative positions of text entities enables more accurate and efficient cross-modal retrieval in the field of image retrieval.

[0086] Specifically, we obtain the entity targets in each block and use them as object tags. Based on the obtained object tags and the identifier of the block to which they belong, we can generate a simple text prompt:

[0087] “The block[P]has a[O].”

[0088] Among them, P∈{1,…,M×N} represents the identifier of the block to which it belongs, which is used to indicate the target location; O represents the object label generated for block P.

[0089] For a specific P, there may be different O options in the scheme because the block may contain multiple objects. For this case, an O is randomly selected each time. Through this method, each sentence incorporates fine-grained object location and language into a model, providing a new way to align objects and related text.

[0090] Specifically, this embodiment can obtain the entities contained in each block or the entities contained in several blocks. According to the number of blocks, the position of each block relative to other blocks can be determined. Figure 3 As shown in the figure, when the image sample is divided into 3×3 blocks, block 4 is located above block 7, and block 4 is located to the upper left of block 8. This image contains only one entity, the lighthouse, located in block 4, to the left of the image sample. Each entity in a block is positioned relative to other entities based on the relative position of the blocks. This relative position information is stored in the text prompt. This allows for the determination of the relative positions of entities, supplementing the training text.

[0091] In step 105, feature vectors of the text corresponding to each block and the entity target after adding information are extracted respectively, and merged based on the feature vectors and high-frequency entity targets.

[0092] In some embodiments, extracting feature vectors of text corresponding to each block and the entity target after adding information includes:

[0093] Use the image encoder to extract features from each block and obtain the first feature vector of each block;

[0094] A text encoder is used to extract features from the text corresponding to the entity target after adding information to obtain a second feature vector.

[0095] In this embodiment, a convolutional neural network is used in the image encoder to extract image features, compress them into a fixed-dimensional vector, and convert the image into a vector representation so that this vector can be matched with text queries. A Transformer deep learning model is used in the text encoder to extract text features, compress them into a fixed-dimensional vector, and convert the text into a vector representation so that this vector can be matched with image queries. This provides a prerequisite method for cross-modal fusion of text and images.

[0096] In some embodiments, merging based on feature vectors and high-frequency entity targets includes:

[0097] Encoding the first eigenvector and the second eigenvector, and mapping the encoded first eigenvector and the second eigenvector to a preset shared subspace;

[0098] The first eigenvector and the second eigenvector after encoding in the shared subspace are multiplied according to the constructed tensor to obtain the text corresponding to each block and entity target after cross-modal fusion;

[0099] The text corresponding to each block and entity target after cross-modal fusion and high-frequency entity targets are merged.

[0100] In this embodiment, if Figure 4 As shown, Figure 4 This is a framework diagram of cross-modal fusion provided by an embodiment of the present invention. After the original data passes through the image segmentation and text adding modules, the block information of the image and the text information of the added template are obtained. These two modal information can be mapped into a shared subspace after encoding processing. The "multiplication" joint method is adopted to fuse the text and image modalities together, and a tensor is constructed to accommodate them. The tensor is composed of the output product of all unimodal feature vectors, thereby realizing the modal fusion of text and images. The advantage of this framework is that it is a simple and effective fusion method. At the same time, the shared subspace usually has semantic invariance, which helps to transfer knowledge from one modality to another in the machine learning model.

[0101] In summary, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for cross-modal image-text retrieval. The method comprises: screening samples in a preset image library and a preset text library to obtain image samples and text samples; the image samples and text samples include at least one entity target of the same type; performing entity extraction on the text sample to obtain the entity target of the text sample; segmenting the image sample to obtain M×N blocks of the image sample; calculating the weight of the entity target within each block, and determining the high-frequency entity target of the image sample based on the weight; extracting text prompt information from each block and appending the text prompt information to the text corresponding to the entity target; wherein the text prompt information includes an identifier of the block to which it belongs and an object tag generated by the block; extracting feature vectors of the text corresponding to each block and the entity target after the text prompt information is appended, and merging the feature vectors based on the feature vectors and the high-frequency entity targets. Embodiments of the present invention segment the image, generate block information, and generate object tags to capture detailed information in the image, thereby improving the accuracy of cross-modal retrieval. Simultaneously, text is processed, templates are added, and the relative positions of entities are determined, further improving the accuracy and efficiency of retrieval. By statistically analyzing the weights of entities after image segmentation, the method of judging entities based on the weights after image segmentation not only improves the accuracy and efficiency of retrieval, but also can effectively identify and distinguish the importance of entities in the image.

[0102] In some embodiments, the process of building a preset text library includes:

[0103] Retrieve the text in the text library and remove the noise of the text according to the regular expression;

[0104] Segment the text after removing the noise into separate text units according to the word segmentation tool;

[0105] Use the preset stop word list to identify stop words in the text units, and delete the identified stop words in the text units;

[0106] Process the text units after deleting the stop words according to the preset text rules to obtain standardized text units, and obtain the preset text library according to the standardized text units.

[0107] Exemplarily, first use the regular expression to remove the noise in the text data, such as HTML tags, special characters, punctuation marks, etc. Use the jieba word segmentation tool to segment the text into separate words or vocabulary units; then use the known stop word list for stop word filtering, and delete the stop words from the text to reduce the data volume and noise. Stop words refer to words that do not provide much information about the meaning of the text, such as "de" (of), "shi" (is), etc. Then regularize the text and convert the text to a standard format, such as converting all text to lowercase letters, removing numbers and special symbols, etc. The text library after text preprocessing is a new word list, which serves as a benchmark in the subsequent process. Text preprocessing refers to a series of processing on the original text data before applying the text data to the model, so as to better extract features and perform model training.

[0108] The construction process of the preset picture library includes: performing preprocessing operations on the pictures in the preset picture library, where the preprocessing operations at least include cropping, scaling, denoising, and enhancement.

[0109] The above cross-modal retrieval method of pictures and texts will be described below through an implementation example. Refer to Figure 5 , Figure 5 is a schematic structural diagram of the cross-modal retrieval method of pictures and texts provided by an embodiment of the present invention. The structure implemented by this method includes three modules, namely a data preprocessing module, a picture and text processing module, and a cross-modal data fusion module. The data preprocessing module includes a preset text library, a preset picture library, a preset picture library sample random screening unit of the preset text library, and a unit for extracting the word list from the text library. The picture and text processing module is used to segment the picture to obtain the feature vector corresponding to the picture in the picture encoder, and is also used to extract entities from the text sample and combine them with the entities of the picture sample to generate corresponding feature vectors. The cross-modal data fusion module is used to perform cross-modal fusion on the feature vectors corresponding to the picture sample and the feature vectors corresponding to the text sample, and fuse the weights calculated for the corresponding entities of the text sample in the picture sample. The cross-modal retrieval of pictures and texts is realized, and the accuracy and efficiency of the retrieval are improved.

[0110] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0111] The following are device embodiments of the present invention. For details not fully described therein, reference may be made to the corresponding method embodiments described above.

[0112] Figure 6 A schematic diagram of the structure of a cross-modal image-text retrieval device provided by an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

[0113] like Figure 6 As shown, the image-text cross-modal retrieval device 60 includes: a data pre-processing module 61, a first processing module 62 and a second processing module 63.

[0114] The data preprocessing module 61 is used to screen samples in a preset image library and a preset text library to obtain image samples and text samples; the image samples and the text samples contain at least the same entity target;

[0115] The first processing module 62 is configured to extract entities from the text sample to obtain entity targets of the text sample; segment the image sample to obtain M×N blocks of the image sample; calculate the weight of the entity targets in each block, and determine the high-frequency entity targets of the image sample based on the weights;

[0116] The second processing module 63 is used to extract the text prompt information of each block and add the text prompt information to the text corresponding to the entity target; wherein the text prompt information includes the identifier of the block to which it belongs and the object tag generated by the block; the feature vectors of the text corresponding to each block and the entity target after the added information are extracted respectively, and merged based on the feature vectors and high-frequency entity targets.

[0117] Optionally, the first processing module 62 is configured to perform the following steps for each entity target:

[0118] Detect the area of ​​the first rectangular region containing the entity target in the image sample;

[0119] For each block, obtain the area of ​​a second rectangular region containing the physical target in the block, divide the area of ​​the second rectangular region by the area of ​​the first rectangular region, and obtain the weight of the physical target in the block;

[0120] For each entity target, the weight of each entity target belonging to this type of entity target in each block is accumulated to obtain the weight of this type of entity target in the image sample;

[0121] The weight of each entity target in the image sample is sorted according to the sorting algorithm, and the entity target with the largest corresponding weight is regarded as the high-frequency entity target of the image sample.

[0122] Optionally, the second processing module 63 is configured to use an image encoder to perform feature extraction on each block to obtain a first feature vector of each block;

[0123] A text encoder is used to extract features from the text corresponding to the entity target after adding information to obtain a second feature vector.

[0124] Optionally, the second processing module 63 is configured to perform encoding processing on the first eigenvector and the second eigenvector, and map the encoded first eigenvector and the second eigenvector to a preset shared subspace;

[0125] The first eigenvector and the second eigenvector after encoding in the shared subspace are multiplied according to the constructed tensor to obtain the text corresponding to each block and entity target after cross-modal fusion;

[0126] The text corresponding to each block and entity target after cross-modal fusion and high-frequency entity targets are merged.

[0127] Optionally, the first processing module 62 is configured to extract entities contained in texts in a preset text library to obtain a vocabulary of the preset text library;

[0128] Filter entities in the vocabulary to obtain multiple high-frequency entities;

[0129] Compare multiple high-frequency entities with the entities extracted from the text sample, and obtain the entity target of the text sample based on the comparison results.

[0130] Optionally, the first processing module 62 is further configured to input each block into a pre-trained neural network to obtain a feature map corresponding to each block;

[0131] Setting a region of interest for each point in the feature map, and obtaining at least one candidate region of interest based on the set region of interest;

[0132] Perform binary classification and bounding box regression on at least one candidate region of interest according to a preset region proposal network, and screen at least one candidate region of interest;

[0133] According to at least one candidate region of interest that has been screened, each block is matched with a pixel of a feature map, and the feature map is matched with a corresponding fixed feature;

[0134] According to the pixels of the corresponding blocks and feature maps, as well as the feature maps and corresponding fixed features, the regions of interest are classified, bounding boxes are regressed, and masks are generated to obtain the entity targets of each block.

[0135] Optionally, the data preprocessing module 61 is further configured to obtain text from a text library and remove noise from the text according to a regular expression;

[0136] Segment the de-noised text into separate text units using a word segmentation tool;

[0137] Using a preset stop word list to identify stop words in the text unit, and deleting the identified stop words in the text unit;

[0138] The text units after the stop words are deleted are processed according to preset text rules to obtain standardized text units, and a preset text library is obtained based on the standardized text units.

[0139] Optionally, the data preprocessing module 61 further presets that the process of constructing the picture library includes: performing preprocessing operations on the pictures in the preset picture library, wherein the preprocessing operations at least include cropping, scaling, denoising and enhancing.

[0140] Figure 7 FIG is a schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps of the above-mentioned various embodiments of the cross-modal image-text retrieval method are implemented.

[0141] The computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the electronic device 7.

[0142] The electronic device 7 can be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The electronic device can include, but is not limited to, a processor 70 and a memory 71. It can be understood by those skilled in the art that Figure 3 It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0143] The processor 70 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0144] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Furthermore, the memory 71 may include both an internal storage unit of the electronic device 7 and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the electronic device. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0146] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0148] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0151] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A cross-modal image and text retrieval method, characterized in that: include: Screen samples in a preset image library and a preset text library to obtain image samples and text samples; The image sample and the text sample include at least one same entity target; Performing entity extraction on the text sample to obtain an entity target of the text sample; segmenting the image sample to obtain M×N blocks of the image sample; Calculating the weight of the entity target in each block respectively, and determining the high-frequency entity target of the image sample according to the weight; Extracting text prompt information from each block and appending the text prompt information to the text corresponding to the entity target; wherein the text prompt information includes an identifier of the block to which it belongs and an object tag generated by the block; extracting feature vectors of the text corresponding to each block and the entity target after appending the text prompt information, and merging the feature vectors based on the feature vectors and the high-frequency entity target; The extracting feature vectors of the text corresponding to each block and the entity target after adding information, and merging based on the feature vectors and the high-frequency entity target, includes: Use the image encoder to extract features from each block and obtain the first feature vector of each block; Using a text encoder to perform feature extraction on the text corresponding to the entity target after the information is added to obtain a second feature vector; Encoding the first eigenvector and the second eigenvector, and mapping the encoded first eigenvector and the second eigenvector to a preset shared subspace; Multiplying the first feature vector and the second feature vector after encoding in the shared subspace according to the constructed tensor to obtain text corresponding to each block and the entity target after cross-modal fusion; Merge the blocks after the cross-modal fusion, the text corresponding to the entity target, and the high-frequency entity target.

2. The method according to claim 1, characterized in that There is at least one type of physical target, and each type of physical target includes at least one physical target; and calculating the weights of the physical targets in each block and obtaining the high-frequency physical targets of the image sample according to the weights includes: For each entity target, perform the following steps: Detecting the area of ​​a first rectangular region containing the physical object in the image sample; For each block, obtaining the area of ​​a second rectangular region containing the physical object in the block, dividing the area of ​​the second rectangular region by the area of ​​the first rectangular region to obtain a weight of the physical object in the block; For each entity target, the weight of each entity target belonging to the entity target in each block is accumulated to obtain the weight of the entity target in the image sample; The weight of each entity target in the image sample is sorted according to a sorting algorithm, and the entity target with the largest corresponding weight is used as the high-frequency entity target of the image sample.

3. The method according to claim 1, characterized in that The performing entity extraction on the text sample to obtain the entity target of the text sample includes: Extracting entities contained in the text in the preset text library to obtain a vocabulary of the preset text library; Filtering entities in the vocabulary to obtain multiple high-frequency entities; The multiple high-frequency entities are compared with the entities extracted from the text sample, and the entity target of the text sample is obtained according to the comparison result.

4. The method according to claim 1, wherein After the image samples are segmented to obtain M×N image samples, the method further includes: Input each block into the pre-trained neural network to obtain the feature map corresponding to each block; Setting a region of interest for each point in the feature map, and obtaining at least one candidate region of interest based on the set region of interest; Performing binary classification and bounding box regression on the at least one candidate region of interest according to a preset region proposal network to screen the at least one candidate region of interest; According to the at least one candidate region of interest that has been screened, each block is matched with a pixel of the feature map, and the feature map is matched with a corresponding fixed feature; According to the corresponding blocks and pixels of the feature map, as well as the feature map and the corresponding fixed features, the region of interest is classified, bounding box regression is performed, and mask generation is performed to obtain the entity target of each block.

5. The method according to claim 1, wherein The process of building the preset text library includes: Acquire text from a text library, and remove noise from the text according to a regular expression; Segment the noise-removed text into separate text units using a word segmentation tool; Performing stop word recognition on the text unit using a preset stop word list, and deleting the recognized stop words in the text unit; Processing the text units after deleting stop words according to preset text rules to obtain standardized text units, and obtaining a preset text library based on the standardized text units; The process of constructing the preset image library includes: performing preprocessing operations on the images in the preset image library, wherein the preprocessing operations at least include cropping, scaling, denoising and enhancing.

6. A cross-modal image and text retrieval device, characterized in that: include: The data preprocessing module is used to screen samples in a preset image library and a preset text library to obtain image samples and text samples; The image sample and the text sample include at least one same entity target; The first processing module is configured to extract entities from the text sample to obtain entity targets of the text sample; and segment the image sample to obtain M×N blocks of the image sample; Calculating the weight of the entity target in each block respectively, and determining the high-frequency entity target of the image sample according to the weight; A second processing module is configured to extract textual hint information from each block and append the textual hint information to the text corresponding to the entity target; wherein the textual hint information includes an identifier of the block to which it belongs and an object tag generated by the block; extract feature vectors of the text corresponding to each block and the entity target after appending the textual hint information, and merge the feature vectors based on the feature vectors and the high-frequency entity target; The second processing module is specifically configured to: Use the image encoder to extract features from each block and obtain the first feature vector of each block; Using a text encoder to perform feature extraction on the text corresponding to the entity target after the information is added to obtain a second feature vector; Encoding the first eigenvector and the second eigenvector, and mapping the encoded first eigenvector and the second eigenvector to a preset shared subspace; Multiplying the first feature vector and the second feature vector after encoding in the shared subspace according to the constructed tensor to obtain text corresponding to each block and the entity target after cross-modal fusion; Merge the blocks after the cross-modal fusion, the text corresponding to the entity target, and the high-frequency entity target.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the image-text cross-modal retrieval method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image-text cross-modal retrieval method as described in any one of claims 1 to 5 are implemented.

Images