Image retrieval method and device, equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence, in particular to the field of image retrieval, and provides an image retrieval method and device, equipment and a storage medium. The method comprises the steps of determining a to-be-drawn area associated with a target retrieval text in a candidate retrieval image based on the target retrieval text containing cue words, re-drawing image content described by the target retrieval text in the to-be-drawn area, and generating a target retrieval image conforming to the text content, by means of the method, automatic image-text association based on the language model is achieved, the restriction of the image library data size on the retrieval range in a traditional image retrieval method is broken through, the image retrieval range is further expanded, and the image retrieval precision of the model is effectively improved.

Description

Background Art

[0002] Nowadays, image information occupies an increasingly large proportion on the Internet. In order to meet people's growing demand for image search, text-based image search technology is often used to perform image retrieval tasks.

[0003] As a machine learning model, the Contrastive Language-Image Pretraining (CLIP) model has achieved good performance in natural language understanding, image classification, image retrieval and other tasks. Among them, when performing image retrieval tasks, the CLIP model extracts features from the retrieval text input by the target object and each original retrieval image in the gallery, obtains the text feature matrix and each image feature matrix, and then retrieves the target retrieval image that meets the retrieval text requirements from multiple original retrieval images based on the cosine similarity between the text feature matrix and each image feature matrix.

[0004] However, due to the limited number of images in the gallery, when the gallery does not include images of the type described by the search text, when performing image search tasks on images already included in the gallery, the cosine similarity between each original search image and the search text does not exceed the set threshold, and the search text is difficult to retrieve the target image that accurately matches the search text, affecting the image retrieval accuracy of the model. For example, the search text is "a pair of pink scissors with a bow on the handle", because the gallery does not include this type of scissors image, resulting in the CLIP model not retrieving the scissors image that is consistent with the search text description.

[0005] Therefore, in order to overcome the current problem of low image retrieval accuracy, this application proposes a new image retrieval method. Summary of the invention

[0006] The embodiments of the present application provide an image retrieval method, apparatus, device and storage medium to solve the problem of low image retrieval accuracy.

[0007] In a first aspect, an embodiment of the present application provides an image retrieval method, comprising:

[0008] Based on the original search text sent by the target object, at least one candidate search image is obtained;

[0009] When it is determined that the degree of match between the at least one candidate search image and the original search text is lower than a first set threshold, adding prompt words representing the intention of the target object to the original search text, and generating a target search text based on the obtained prompt words;

[0010] For each candidate search image whose search text matching degree is lower than the first set threshold, the following operations are performed respectively:

[0011] Performing attention interaction between a candidate retrieval image and the target retrieval text to obtain a to-be-drawn area in the candidate retrieval image that is associated with the target retrieval text;

[0012] The image content described by the target retrieval text is redrawn in the area to be drawn to obtain a corresponding target retrieval image.

[0013] In a second aspect, the present application also provides an image retrieval device, including:

[0014] An image retrieval module, used to obtain at least one candidate retrieval image based on the original retrieval text sent by the target object;

[0015] a processing module, configured to add prompt words representing the intention of the target object to the original search text when it is determined that the matching degree between the at least one candidate search image and the original search text is lower than a first set threshold, and generate a target search text based on the obtained prompt words;

[0016] The image drawing module is used to perform the following operations for each candidate search image whose search text matching degree is lower than the first set threshold:

[0017] Performing attention interaction between a candidate retrieval image and the target retrieval text to obtain a to-be-drawn area in the candidate retrieval image that is associated with the target retrieval text;

[0018] The image content described by the target retrieval text is redrawn in the area to be drawn to obtain a corresponding target retrieval image.

[0019] Optionally, before obtaining a plurality of attention weight features based on each text feature and each image feature, the image retrieval device further includes a preprocessing module, wherein the preprocessing module is used to:

[0020] Based on the pre-constructed first fully connected layer and the second fully connected layer, the dimensions of the text features and the image features are aligned to obtain the text features and the image features with the same dimensions.

[0021] Optionally, the image drawing module performs the following operations with respect to the image features and corresponding attention weight features of each pixel:

[0022] The image vector of each dimension in an image feature is multiplied by the attention weight vector of the same dimension in the attention weight feature corresponding to a pixel point to obtain a multimodal attention feature containing a multidimensional vector.

[0023] Optionally, the processing module is used to:

[0024] Performing part-of-speech detection on each original word in the original search text to obtain the respective part-of-speech types;

[0025] After the first original word whose part of speech type is a noun, adding a prompt word representing the intention of the target object;

[0026] When it is determined that a second original word adjacent to the first original word is a modified word, the prompt word is added after the second original word.

[0027] Optionally, the image drawing module is used to:

[0028] According to the target search text, the pixel value of each pixel point in the area to be drawn is adjusted to draw the image content described by the target search text;

[0029] Re-rendering the candidate retrieval image containing the image content to generate a corresponding target retrieval image.

[0030] In a third aspect, an embodiment of the present application further provides a computer device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned image retrieval methods.

[0031] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of any one of the above-mentioned image retrieval methods.

[0032] In a fifth aspect, an embodiment of the present application further provides a computer program product, including computer instructions, which are executed by a processor to perform the steps of any one of the above-mentioned image retrieval methods.

[0033] The beneficial effects of this application are as follows:

[0034] The embodiment of the present application provides an image retrieval method, apparatus, device and storage medium, the method comprising: when it is determined that there is at least one candidate retrieval image with a low degree of matching with the text, adding prompt words to the original retrieval text to generate a corresponding target retrieval text, so as to help the model better understand natural language, and help improve the detection accuracy and retrieval accuracy of the model; further, based on the attention interaction between the text features of the target retrieval text and the image features of the above-mentioned candidate retrieval images, image detection is performed on the above-mentioned candidate retrieval images, and the model is guided to focus on the image features with a high degree of correlation with the text features of the target retrieval text, and the area to be drawn associated with the target retrieval text in the image is determined, which is also helpful to generate a target retrieval image that matches the image and text; finally, according to the text content described by the target retrieval text, the relevant image content is redrawn in the area to be drawn of the similar image, and the target retrieval image with accurate matching of the image and text is generated, breaking through the restriction of the amount of library data on the retrieval scope in the traditional image retrieval method, realizing automatic association of image and text based on the language model, and redrawing the similar images based on the text content input by the target object on the basis of the existing library, generating an image that is more in line with the semantic content, and realizing the need to improve the image retrieval accuracy of the model without increasing the number of retrievals.

[0035] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1A A logical diagram for performing image classification tasks using the CLIP model;

[0038] Figure 1B A logical diagram for performing image retrieval tasks using the CLIP model;

[0039] Figure 2A This is an optional schematic diagram of an application scenario in an embodiment of the present application;

[0040] Figure 2B A schematic diagram of the structure of the image retrieval model provided in the embodiment of the present application;

[0041] Figure 3A A schematic diagram of a process for performing retrieval using an image retrieval model provided in an embodiment of the present application;

[0042] Figure 3B A logical schematic diagram of retrieval using an image retrieval model provided in an embodiment of the present application;

[0043] Figure 3C A schematic diagram of the process of performing the first search of the gallery provided in the embodiment of the present application;

[0044] Figure 3D A logical schematic diagram of the first search of the gallery provided in the embodiment of the present application;

[0045] Figure 3E A schematic diagram of the angle between the characteristic vectors a and b provided in an embodiment of the present application;

[0046] Figure 3F A schematic diagram of a process for detecting an area to be drawn provided in an embodiment of the present application;

[0047] Figure 3G A logical schematic diagram of detecting an area to be drawn provided in an embodiment of the present application;

[0048] Figure 3H A logical schematic diagram of dimension alignment provided for an embodiment of the present application;

[0049] Fig. 3I A logical schematic diagram of obtaining an attention weight feature provided in an embodiment of the present application;

[0050] Figure 3J A logical schematic diagram of obtaining a one-dimensional attention weight vector provided in an embodiment of the present application;

[0051] Figure 3K A logical schematic diagram of obtaining multimodal attention features provided in an embodiment of the present application;

[0052] Figure 3L A logical schematic diagram for generating an area to be drawn provided in an embodiment of the present application;

[0053] Figure 3M Another logical schematic diagram for generating an area to be drawn provided in an embodiment of the present application;

[0054] Figure 3N A logical schematic diagram of performing binarization processing on an image containing an area to be drawn provided in an embodiment of the present application;

[0055] Fig.3O A logical schematic diagram of generating a boat image provided by an embodiment of the present application;

[0056] Figure 3P A logical diagram of generating pink scissors provided in an embodiment of the present application;

[0057] Figure 4A A schematic diagram of a process for searching for chat pictures in a social software provided in an embodiment of the present application;

[0058] Figure 4B A logical schematic diagram of searching for chat pictures in social software provided by an embodiment of the present application;

[0059] Figure 5A A schematic diagram of a process for automatically searching for emoticon images in social software provided in an embodiment of the present application;

[0060] Figure 5B A logical schematic diagram of automatically searching for emoticon package images in social software provided in an embodiment of the present application;

[0061] Fig. 6A A schematic diagram of a process for manually searching for emoticon images in social software provided in an embodiment of the present application;

[0062] Figure 6B A logical schematic diagram of manually searching for emoticon images in social software provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the structure of an image retrieval device provided in an embodiment of the present application;

[0064] Figure 8 A schematic diagram of a hardware structure of a computer device to which an embodiment of the present application is applied;

[0065] Fig. 9 A schematic diagram of the hardware composition structure of another computer device to which the embodiment of the present application is applied. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0067] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0068] 1. Artificial Intelligence (AI):

[0069] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0070] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies; artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0071] With the research and advancement of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, robots, smart medical care, etc. I believe that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0072] 2. Machine Learning:

[0073] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0074] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence, including deep learning, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0075] 3. Comparative learning:

[0076] Generally speaking, machine learning tasks can be divided into supervised learning and unsupervised learning. Contrastive learning, as an important part of self-supervised learning, is widely used in computer vision, natural language processing and other fields. Its goal is to shorten the distance between features of the same category as much as possible and to increase the distance between features of different categories as much as possible.

[0077] 4. Computer vision is a comprehensive subject that integrates computer science, signal processing, physics, applied mathematics, statistics, neurophysiology and other disciplines. It is also an important and challenging research direction in the scientific field.

[0078] This discipline uses various imaging systems instead of visual organs as input means, and computers replace the brain to complete processing and interpretation, so that computers can have the ability to observe and understand the world through vision like humans. Among them, the sub-fields of computer vision include face detection, face comparison, facial features detection, blink detection, liveness detection, fatigue detection, etc.

[0079] 5. Natural Language Processing (NLP):

[0080] Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0081] 6. Word embedding is a vector of real numbers mapped from words or phrases in the vocabulary. Conceptually, it involves the mathematical embedding of each word from a one-dimensional space to a continuous vector space with a lower dimension. Methods for generating such mappings include but are not limited to: dictionary order, one-shot encoding, matrix-based distributed representation, and neural networks.

[0082] The following is a brief introduction to the design concept of the embodiment of the present application:

[0083] Nowadays, the proportion of image information in the Internet is gradually increasing. In order to meet people's growing demand for image search, text-based image search technology is often used to perform image retrieval tasks. As a machine learning model, the CLIP model performs well in natural language understanding, image classification, image retrieval and other tasks.

[0084] Among them, Figure 1AAs shown in the figure, when performing image classification tasks, the CLIP model constructs respective label texts (such as "A photo of a Dog") for each label in the target task dataset, and then uses the text encoder (TextEncoder) to extract the text feature matrix of each label text, and uses the image encoder (Image Encoder) to extract the image feature matrix of the image to be classified, and calculates the inner product between the image feature matrix and each text feature matrix, and takes the label corresponding to the larger inner product value as the classification result of the image to be classified.

[0085] like Figure 1B As shown in the figure, when performing image retrieval tasks, the CLIP model uses a text encoder to extract the text feature matrix of the retrieved text, and uses an image encoder to extract the image feature matrix of each original retrieval image in the gallery, calculates the cosine similarity between the text feature matrix and each image feature matrix, and uses the original retrieval image corresponding to the larger similarity value as the target retrieval image, which is fed back to the target object that triggered the task.

[0086] However, due to the limited number of images in the gallery, when the image type described by the search text is not included in the gallery, when performing image search tasks on the images already included in the gallery, the cosine similarity between each original search image and the search text does not exceed the set threshold, making it difficult to retrieve the target image that accurately matches the search text, which in turn affects the image retrieval accuracy of the model. For example, the search text is "a pair of pink scissors with a bow on the handle", and since the gallery does not include this type of scissors image in advance, the CLIP model fails to successfully retrieve the scissors image that matches its description.

[0087] In order to overcome the current problem of low image retrieval accuracy, the embodiments of the present application provide an image retrieval method, device, equipment and storage medium. The method includes: based on the original retrieval text sent by the target object, obtaining at least one candidate retrieval image, when determining that the matching degree between at least one candidate retrieval image and the original retrieval text is lower than a first set threshold, adding prompt words representing the intention of the target object to the original retrieval text, and generating a target retrieval text based on each obtained prompt word; then for each candidate retrieval image whose matching degree is lower than the first set threshold, performing the following operations respectively: performing attention interaction between a candidate retrieval image and the target retrieval text, obtaining an area to be drawn in the image associated with the target retrieval text, redrawing the image content described by the target retrieval text in the area to be drawn, and obtaining a corresponding target retrieval image.

[0088] Based on the target retrieval text containing the prompt word, the area to be drawn in the candidate retrieval image that is associated with the target retrieval text is determined, and then the image content described by the target retrieval text is redrawn in the area to be drawn to generate a target retrieval image that conforms to the text content. This method realizes automatic association between images and text based on a language model, breaks through the constraints of the amount of library data on the retrieval scope in traditional image retrieval methods, further expands the image retrieval scope, and effectively improves the image retrieval accuracy of the model.

[0089] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0090] The image retrieval method provided in the embodiment of the present application can be applied in various scenarios such as social software, browsers, AI drawing, etc., to execute image retrieval tasks triggered by the target object and feed back the retrieval results to the object.

[0091] Figure 2A One application scenario is shown, in which two terminal devices 210 and a server 230 are included. The terminal device 210 establishes a communication connection with the server 230 via a wired network or a wireless network.

[0092] Among them, the terminal device 210 includes but is not limited to: mobile phones, computers (such as tablets, laptops, desktop computers, etc.), smart home appliances, smart voice interaction devices (such as smart watches, smart speakers, etc.), vehicle-mounted terminals, aircraft, etc.

[0093] The server 230 of the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. This application does not make any restrictions here.

[0094] The terminal device 210 obtains the original search text input by the target object through the search bar of the image search interface 220, which contains text information describing the image content. The terminal device 210 then sends the original search text to the server 230 through the established communication channel, and the server 230 calls the image search model to perform the image search task and feeds back the target search image that matches the text content to the target object.

[0095] like Figure 2BAs shown, the image retrieval model includes: a feature extraction module, an image retrieval module, a judgment module, and an image drawing module.

[0096] The feature extraction module is composed of a first text encoder and a first image encoder, wherein the first text encoder is used to extract text features of the original query text, and the first image encoder is used to extract image features of each original search image in the gallery;

[0097] As the name implies, the image retrieval module is a functional module that retrieves images that are close to the original search text in the gallery based on the obtained text features and image features. The judgment module makes an accurate match judgment on each image retrieved for the first time to filter out images that do not meet the accurate match requirements.

[0098] The image drawing module consists of a second text encoder, a second image encoder, a prompt word editor, an image detector, and an image drawer. The prompt word extractor is used to automatically add prompt words to the original search text to generate the corresponding target search text. The second text encoder is used to extract text features of the target query text. The second image encoder is used to extract image features of candidate search images that do not meet the precise matching requirements. The image detector detects the area to be drawn in the candidate search image that is associated with the target search text based on the obtained text features and image features. The image drawer draws the corresponding image content in the area to be drawn according to the text information described in the target search text, so as to obtain a target search image that better meets the semantic requirements.

[0099] As a machine learning model, the image retrieval model can be deployed in the server 230 to realize the function of calling the local model, or it can be deployed in other independent servers in the network to realize the function of calling the model remotely, or it can be deployed in a server cluster or distributed system in the network to realize the function of calling the distributed model. This application does not limit this.

[0100] Taking the image retrieval model deployed on server 230 as an example, the process of calling the model for image retrieval is as follows:

[0101] The server 230 issues a call instruction to call the image retrieval model deployed locally, and then inputs the received original search text into the image retrieval model, which retrieves candidate search images associated with the text content in the gallery. When it is determined that at least one candidate search image does not meet the requirements of accurate matching, the image retrieval model adds prompt words representing the intention of the target object to the original search text, and generates a target search text based on the obtained prompt words.

[0102] For each candidate retrieval image with a matching degree lower than the first set threshold, the following operations are respectively performed: Interact the attention between a candidate retrieval image and the target retrieval text to obtain the area to be drawn in the image that is associated with the target retrieval text, and redraw the image content described by the target retrieval text in the area to be drawn to obtain the corresponding target retrieval image.

[0103] The server 230 sends the target retrieval images that are precisely matched in the image library and the target retrieval images drawn by the model later to the terminal device 210, and the terminal device 210 displays the corresponding retrieval results to the target object through the image retrieval interface 220.

[0104] Next, as Figure 3A-3B shown, the process of the image retrieval model constructed in this application when performing the image retrieval task is as follows:

[0105] S301: Based on the original retrieval text sent by the target object, obtain at least one candidate retrieval image.

[0106] Input the original retrieval text and each original retrieval image included in the image library into the feature extraction module. As Figure 3C to 3D shown, by performing the following operations, at least one candidate retrieval image is retrieved:

[0107] S3011: Extract the features of each original word in the original retrieval text to obtain the text features of each original word respectively, and extract the features of each original retrieval image respectively to obtain their respective image feature matrices.

[0108] To ensure that the model can better understand natural language, in the first step of natural language processing, the first text encoder performs vectorization representation on each original word in the original retrieval text, maps it from a one-dimensional space to a continuous vector space with a lower dimension, and obtains the word vector of each original word respectively. Then, the encoder encodes each word vector to obtain the text features of each original word respectively, and forms the text feature matrix of the original retrieval text by combining multiple text features.

[0109] The methods for generating such a mapping include but are not limited to: lexicographic order, one-shot encoding, matrix-based distributed representation, and neural networks. This application does not make a limitation here. For example, perform one-shot encoding on the character "sea" to obtain the word vector of the character "sea" as [0 0 0 0 0 1 0 1 0 0].

[0110] Moreover, to ensure that the model can better understand the image content, the first image encoder is also called to extract the image features of each pixel point in each original retrieval image respectively, and form the image feature matrix of the image by combining multiple image features belonging to the same image.

[0111] S3012: Calculate the cosine similarity between the text feature matrix and each image feature matrix respectively, and use the images whose cosine similarity exceeds a third set threshold as candidate retrieval images.

[0112] Cosine similarity uses the cosine value of the angle between two eigenvectors in the vector space as a measure of the difference between the two eigenvectors. The value range of cosine similarity is [-1, 1]. Figure 3E As shown, the closer the cosine value is to 1, the closer the angle between the eigenvectors a and b is to 0 degrees, which further indicates that the two eigenvectors have a high similarity.

[0113] S302: When it is determined that the matching degree between at least one candidate search image and the original search text is lower than a first set threshold, a prompt word representing the intention of the target object is added to the original search text, and a target search text is generated based on the obtained prompt words.

[0114] The feature extraction module sends at least one candidate search image obtained by the search to the judgment module, and the judgment module performs an accurate matching judgment on each image to filter out images that do not meet the accurate matching requirements.

[0115] When making a judgment, the judgment module can use cosine similarity as the matching degree between the image and the text, and judge the candidate retrieval image that exceeds the third set threshold and is lower than the first set threshold as an image that does not meet the precise matching requirements; and judge the candidate retrieval image that exceeds the first set threshold as the target retrieval image for image and text matching.

[0116] In order to further improve the matching detection accuracy, the original search text can be matched with each candidate search image based on the machine learning image-text matching model, and the candidate search images below the first set threshold are judged as images that do not meet the precise matching requirements; the candidate search images above the first set threshold are judged as target search images for image-text matching.

[0117] In the field of artificial intelligence, prompt is a very important concept. It is a marker word that expresses the intention of the target object, helping machine learning models to better understand natural language and then detect the areas to be drawn in the image that are associated with the text content, which helps to improve the image retrieval accuracy of the model.

[0118] In the past, there was no clear standard for adding prompt words, and they were usually added manually based on the developer's experience. However, this was time-consuming and labor-intensive, and it was difficult to ensure the accuracy of the addition, which in turn affected image detection and drawing target retrieval images, and increased the retrieval risk of the model.

[0119] To solve this problem, the embodiments of the present application adopt a prompt word editor based on a neural network to automatically add prompt words to the original retrieval text to generate the corresponding target retrieval text.

[0120] For example, if the original retrieval text is "I want a gift box", this sentence indicates that the target object hopes that the target retrieval image is an image of a gift box. Prompt words should be added next to "a" and "gift box" to help the machine learning model better understand natural language.

[0121] Since the intentions of the target object are usually nouns and modifiers near the nouns, and the modifiers include but are not limited to: quantifiers, modal particles, and words indicating colors. Therefore, the prompt word editor is used to detect the part-of-speech of each original word in the original retrieval text respectively to obtain their respective part-of-speech types. After the first original word with the part-of-speech type of noun, a prompt word representing the intention of the target object is added. When the second original word adjacent to the first original word is a modifier, a prompt word is added after the second original word.

[0122] For example, if the original retrieval text is "a little dog sitting on the grass", the part-of-speech detection result is "sitting (verb) grass (noun) on the little dog (noun)". Prompt words mask are added after the two nouns, and the generated target retrieval text is "a little dog sitting on the grass [mask] [mask]".

[0123] S303: For each candidate retrieval image whose matching degree with the original query text retrieval text is lower than the first set threshold, the following operations are respectively performed: An attention interaction is performed between a candidate retrieval image and the target query text retrieval text to obtain the area to be drawn in the candidate retrieval image that is associated with the target query text retrieval text, and the image content described by the target query text retrieval text is redrawn in the area to be drawn to obtain the corresponding target retrieval image.

[0124] The attention mechanism originated from the research on human vision. In cognitive science, due to the bottleneck of information processing, humans will selectively focus on a part of all information while ignoring other visible information. For example, just seeing the character "鹄" may not be able to understand its meaning, but after seeing its context "鸿鹄之志", it can be understood that "鹄" in this sentence means "swan". Therefore, the context information of words or phrases helps to enhance their semantic representation. At the same time, the roles played by different words or phrases in the context in enhancing semantic representation are often different. Continuing with the example of "鸿鹄之志", the character "鸿" plays the greatest role in understanding the character "鹄", while the character "之" plays a relatively small role.

[0125] The embodiment of the present application is based on the attention mechanism, which interacts the attention of a candidate retrieval image with the target query text retrieval text, distinguishes the degree of attention of each text feature to the image feature, and guides the image detector to focus on the image features that have a greater impact on image detection, so as to obtain the area to be drawn in the image that is associated with the target query text retrieval text.

[0126] like Figures 3F to 3G As shown in FIG. 1 , the process of detecting the area to be drawn associated with the target retrieval text in a candidate retrieval image is as follows:

[0127] S3031: Perform feature extraction on each target word in the target search text to obtain the text features of each target word, and perform feature extraction on each pixel point in a candidate search image to obtain the image features of each pixel point.

[0128] In order to ensure that the model can better understand natural language, in the first step of natural language processing, the second text encoder vectorizes each target word in the target retrieval text, maps it from a one-dimensional space to a continuous vector space with a lower dimension, and obtains the word vector of each target word. After that, the encoder encodes each word vector to obtain the text features of each target word. Among them, the method of generating this mapping includes but is not limited to: dictionary order, one-shot encoding, matrix-based distributed representation and neural network, which is not limited in this application.

[0129] Moreover, in order to ensure that the model can better understand the image content, the second image encoder will be called to extract the image features of each pixel in each candidate retrieval image.

[0130] S3032: Based on each text feature and each image feature, a plurality of attention weight features are obtained, each attention weight feature representing: a correlation degree between an image feature and each of the text features.

[0131] like Figure 3H As shown, both text features and image features are multi-dimensional feature vectors. In order to facilitate operations between features, before obtaining multiple attention weight features, based on the pre-constructed first fully connected layer and the second fully connected layer, the dimensions of each text feature and each image feature are aligned to obtain each text feature and each image feature with the same dimension.

[0132] After performing the dimension alignment operation, in order to better mine the correlation between image and text features, such as Fig. 3I As shown, the following operations are performed for each image feature to obtain a corresponding attention weight feature, each attention weight feature representing: the degree of association between an image feature and each of the text features:

[0133] Perform weighted summation of an image feature and each text feature to obtain their respective attention weight vectors, and concatenate the attention weight vectors of multiple dimensions to obtain a corresponding attention weight feature.

[0134] According to the previous introduction, an attention weight feature is composed of a multi-dimensional attention weight vector. Figure 3J As shown, for each text feature, the following operations are performed to obtain a one-dimensional attention weight vector:

[0135] The text vectors of each dimension in a text feature are multiplied with the image vector of the same dimension in an image feature to obtain their respective weighted vectors, and then the weighted vectors of multiple dimensions are added together to obtain a one-dimensional attention weight vector.

[0136] S3033: Based on the obtained attention weight features and image features, a multimodal attention feature of each pixel is obtained, and each multimodal attention feature represents: the degree of association between a pixel and each target word contained in the target retrieval text.

[0137] Among them, Figure 3K As shown, for each pixel’s image features and corresponding attention weight features, the following operations are performed:

[0138] The image vector of each dimension in an image feature is multiplied by the attention weight vector of the same dimension in the attention weight feature corresponding to a pixel point to obtain a multimodal attention feature containing a multidimensional vector.

[0139] S3034: Based on the pixel points whose feature values ​​of the multimodal attention feature exceed the second set threshold, a to-be-drawn area associated with the target retrieval text is delineated in this candidate retrieval image.

[0140] Obtain the pixel points whose characteristic values ​​of the multimodal attention feature exceed the second set threshold, and use any of the following strategies to obtain the area to be drawn:

[0141] Strategy 1: By comparing the coordinate positions of each pixel point, the pixel points on the boundary line are taken as boundary points, and by connecting each boundary point, the following Figure 3L The area to be drawn is shown;

[0142] Strategy 2: By comparing the coordinate positions of each pixel point, the pixel points on the boundary line are obtained. Two adjacent pixel points on the boundary line form a group of pixel points. Each group of pixel points is read in turn to generate multiple boundary points. By connecting each boundary point, the following is generated: Figure 3MThe area to be drawn is shown; wherein, each time a group is read, the coordinate position of a boundary point is generated based on the coordinate positions of two pixel points in the group.

[0143] By comparison Figure 3L and Figure 3M It can be clearly seen that the area to be drawn based on method 1 is drawn along the contour edge of the original image content, and the area to be drawn based on method 2 is based on the contour edge of the original image content, and a rectangular area covering the content is generated. The former extracts the object range more accurately, and the latter has a higher fault tolerance rate. Each has its own advantages, and the specific area generation strategy can be selected according to the actual situation.

[0144] In order to directly highlight the area to be drawn in the image, the above image can also be binarized, and different pixel values ​​are set for the pixels in the area to be drawn and the pixels outside the area. Figure 3N The pixel value of the island area is set to 1 (i.e. white), and the pixel value of the background area is set to 0 (i.e. black), resulting in an image with a black background and white blocks.

[0145] After determining the area to be drawn, the pixel value of each pixel in the area to be drawn is adjusted according to the target retrieval text, and the image content described by the target retrieval text is drawn; the candidate retrieval image containing the image content is re-rendered to generate the corresponding target retrieval image.

[0146] For example, according to the target search text "a boat [mask] sailing on the sea [mask]", draw a boat in the area to be drawn, and get Fig.3O The target retrieval image is shown.

[0147] For another example, according to the target search text "a pair of pink [mask] scissors [mask] with a bow [mask] on the handle [mask]", a pair of pink scissors with a bow [mask] on the handle is drawn in the area to be drawn, and the result is Figure 3P The target retrieval image is shown.

[0148] If there are multiple images that do not meet the precise matching requirements, these images are redrawn by executing steps 301 to 303 to generate corresponding target retrieval images. Since the original candidate retrieval images depict different contents, the redrawn target retrieval images are also different images containing the target retrieval text description content, and are not identical images with exactly the same content.

[0149] The image retrieval method provided in the embodiment of the present application can be applied in various scenarios such as social software, browsers, AI drawing, etc., to perform image retrieval tasks triggered by the target object and feed back the retrieval results to the object. Next, several specific embodiments will be used as examples to introduce the application process of the method in the corresponding scenarios.

[0150] Specific embodiment 1: Searching for related chat pictures in social software.

[0151] like Figures 4A to 4B As shown, using the method provided in the embodiment of the present application, the process of retrieving chat pictures in social software is as follows:

[0152] S401: The social software displays an image search interface including a search bar and a search control in response to a search operation triggered by a target object;

[0153] S402: the social software responds to the input operation of the target object on the search bar and obtains the original search text "a ship sailing on the sea" input by the target object;

[0154] S403: The social software sends the original search text to the backend server in response to the target object's click operation on the search control;

[0155] S404: The server calls the image retrieval model, controls the model to perform image retrieval in the original image library according to the original retrieval text, obtains multiple candidate retrieval images, and further determines that two of the images are target retrieval images that are accurate matches, and the other image is an image that does not meet the accurate match requirement;

[0156] S405: the image retrieval model adds prompt words representing the intention of the target object to the original retrieval text, and generates a target retrieval text “a ship [mask] sailing on the sea surface [mask]” based on the obtained prompt words;

[0157] S406: For the candidate retrieval image that does not meet the precise matching requirement, an attention mechanism is used to determine the area to be drawn in the image that is associated with the target retrieval text, and the image content described by the target retrieval text is redrawn in the area to be drawn to obtain the corresponding target retrieval image;

[0158] S407: The server sends the multiple target retrieval images output by the model to the social software, so that the social software displays the multiple target retrieval images in the image retrieval interface;

[0159] S408: In response to the target object's click operation on the target search image, the social software sends the selected target search image as a chat picture to other objects in the chat group.

[0160] Specific Embodiment 2: Automatically search for relevant meme pictures in a social software.

[0161] As Figures 5A-5B shown, the process of retrieving meme pictures in a social software by using the method provided in the embodiment of the present application is as follows:

[0162] S501: In response to a click operation of a target object on any chat group, the social software jumps from the social main interface to the chat interface of the group, and the chat interface includes at least one chat input box;

[0163] S502: In response to an input operation of the target object on the chat input box, the social software automatically reads the original retrieval text input by the target object according to a set scanning period. For each original retrieval text read, it is sent to the server in the background for image correlation retrieval;

[0164] S503: The server calls an image retrieval model, controls the model to perform image retrieval in the meme picture library according to the obtained original retrieval text "haha", obtains multiple candidate retrieval images, and after further judgment, determines that four of the images are target retrieval images with precise matching, and one image does not meet the requirements of precise matching;

[0165] S504: The image retrieval model adds a prompt word representing the intention of the target object to the original retrieval text, and generates a target retrieval text "haha [mask]" based on the obtained prompt words;

[0166] S505: For the candidate retrieval image that does not meet the requirements of precise matching, the attention mechanism is used to determine the area to be drawn in the image that is associated with the target retrieval text, and the originally crying expression is modified to a laughing expression in the area to be drawn to obtain the corresponding target retrieval image;

[0167] S506: The server sends the multiple target retrieval images output by the model to the social software, so that the social software displays the obtained multiple meme pictures of laughing above the chat input box.

[0168] Specific Embodiment 2: Manually search for relevant meme pictures in a social software.

[0169] As shown in FIGS. 6A - 6B, the process of retrieving meme pictures in a social software by using the method provided in the embodiment of the present application is as follows:

[0170] S601: In response to a click operation of a target object on any chat group, the social software jumps from the social main interface to the chat interface of the group, and the chat interface includes at least one expression control;

[0171] S602: In response to the target object's click operation on the emoticon control, the social software displays an emoticon search sub-interface in the chat interface, where the sub-interface includes at least a search bar and a search control;

[0172] S603: the social software responds to the target object's input operation on the search bar and obtains the original search text "angry" input by the target object;

[0173] S604: The social software sends the original search text to the backend server in response to the target object's click operation on the search control;

[0174] S605: The server calls the image retrieval model, controls the model to perform image retrieval in the original image library according to the original retrieval text, obtains multiple candidate retrieval images, and further determines that all images meet the requirements of accurate matching;

[0175] S606: The server sends the multiple target retrieval images output by the model to the social software, so that the social software displays the multiple angry emoticon images in the emoticon search sub-interface.

[0176] In addition, it should be noted that in the specific implementation of this application, it involves object data related to the collection of original search texts, the collection of original search images, the redrawing of original images, etc. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0177] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides an image retrieval device. Figure 7 As shown, the image retrieval device 700 may include:

[0178] The image retrieval module 701 is used to obtain at least one candidate retrieval image based on the original retrieval text sent by the target object;

[0179] The processing module 702 is used to add prompt words representing the intention of the target object to the original search text when it is determined that the matching degree between at least one candidate search image and the original search text is lower than a first set threshold, and generate a target search text based on the obtained prompt words;

[0180] The image drawing module 703 is used to perform the following operations for each candidate search image whose search text matching degree is lower than the first set threshold:

[0181] Perform attention interaction between a candidate retrieval image and a target retrieval text to obtain a to-be-drawn area in the candidate retrieval image that is associated with the target retrieval text;

[0182] The image content described by the target retrieval text is redrawn in the area to be drawn to obtain a corresponding target retrieval image.

[0183] Optionally, the image drawing module 703 is used to:

[0184] Perform feature extraction on each target word in the target retrieval text to obtain the text features of each target word, and perform feature extraction on each pixel in a candidate retrieval image to obtain the image features of each pixel;

[0185] Based on each text feature and each image feature, multiple attention weight features are obtained, each attention weight feature represents: the degree of association between an image feature and each text feature;

[0186] Based on the obtained attention weight features and image features, a multimodal attention feature of each pixel is obtained. Each multimodal attention feature represents: the degree of association between a pixel and each target word contained in the target retrieval text;

[0187] Based on the pixel points whose feature values ​​of the multimodal attention feature exceed the second set threshold, a to-be-drawn area associated with the target retrieval text is delineated in a candidate retrieval image.

[0188] Optionally, the image drawing module 703 performs the following operations for each image feature to obtain a corresponding attention weight feature:

[0189] Perform weighted summation of an image feature and each text feature to obtain their respective attention weight vectors;

[0190] Concatenate the attention weight vectors of multiple dimensions to obtain a corresponding attention weight feature.

[0191] Optionally, the image drawing module 703 performs the following operations for each text feature:

[0192] Multiply the text vectors of each dimension in a text feature with the image vectors of the same dimension in an image feature to obtain their respective weighted vectors;

[0193] Add the weighted vectors of multiple dimensions to obtain a one-dimensional attention weight vector.

[0194] Optionally, before obtaining a plurality of attention weight features based on each text feature and each image feature, the image retrieval device 700 further includes a preprocessing module 704, which is used to:

[0195] Based on the pre-built first fully connected layer and the second fully connected layer, the dimensions of each text feature and each image feature are aligned to obtain each text feature and each image feature with the same dimension.

[0196] Optionally, the image drawing module 703 performs the following operations with respect to the image features of each pixel and the corresponding attention weight features:

[0197] The image vector of each dimension in an image feature is multiplied by the attention weight vector of the same dimension in the attention weight feature corresponding to a pixel point to obtain a multimodal attention feature containing a multidimensional vector.

[0198] Optionally, the processing module 702 is used to:

[0199] Perform part-of-speech detection on each original word in the original search text to obtain the respective part-of-speech types;

[0200] After the first original word whose part of speech type is a noun, add a prompt word representing the intention of the target object;

[0201] When it is determined that the second original word adjacent to the first original word is a modified word, a prompt word is added after the second original word.

[0202] Optionally, the image drawing module 703 is used to:

[0203] According to the target search text, the pixel value of each pixel point in the area to be drawn is adjusted to draw the image content described by the target search text;

[0204] Re-render the candidate retrieval images containing the image content to generate the corresponding target retrieval images.

[0205] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0206] After introducing the image retrieval method and apparatus according to an exemplary embodiment of the present application, next, a computer device according to another exemplary embodiment of the present application is introduced.

[0207] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0208] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a computer device. In one embodiment, the computer device may be a server, such as Figure 2A In this embodiment, the structure of the computer device is as follows: Figure 8 As shown, it may include at least a memory 801 , a communication module 803 , and at least one processor 802 .

[0209] The memory 801 is used to store computer programs executed by the processor 802. The memory 801 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0210] The memory 801 may be a volatile memory, such as a random-access memory (RAM); the memory 801 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 801 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 801 may be a combination of the above memories.

[0211] The processor 802 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 802 is used to implement the above-mentioned image retrieval method when calling the computer program stored in the memory 801 .

[0212] The communication module 803 is used to communicate with terminal devices and other servers.

[0213] The specific connection medium between the memory 801, the communication module 803 and the processor 802 is not limited in the embodiment of the present application. Figure 8 In the embodiment, the memory 801 and the processor 802 are connected via a bus 804. The bus 804 is Figure 8 The connections between the other components are described with bold lines, which are only for illustrative purposes and are not intended to be limiting. The bus 804 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 8 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0214] The memory 801 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the image retrieval method of the embodiment of the present application. The processor 802 is used to execute the above-mentioned image retrieval method, such as Figure 3A shown.

[0215] In another embodiment, the computer device may also be other computer devices, such as Figure 2A The terminal device 210 shown in FIG. 1 is a terminal device 210 shown in FIG. 1 . In this embodiment, the structure of the computer device can be as follows: Fig. 9 As shown, it includes: a communication component 910, a memory 920, a display unit 930, a camera 940, a sensor 950, an audio circuit 960, a Bluetooth module 970, a processor 980 and other components.

[0216] The communication component 910 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the object to send and receive information through the WiFi module.

[0217] The memory 920 can be used to store software programs and data. The processor 980 executes various functions and data processing of the terminal device 210 by running the software programs or data stored in the memory 920. The memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The memory 920 stores an operating system that enables the terminal device 210 to run. In the present application, the memory 920 can store an operating system and various application programs, and can also store a computer program for executing the image retrieval method of the embodiment of the present application.

[0218] The display unit 930 may also be used to display information input by the object or information provided to the object and a graphical user interface (GUI) of various menus of the terminal device 210. Specifically, the display unit 930 may include a display screen 932 disposed on the front of the terminal device 210. The display screen 932 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 930 may be used to display a defect detection interface, a model training interface, etc. in the embodiment of the present application.

[0219] The display unit 930 can also be used to receive input digital or character information and generate signal input related to the object setting and function control of the terminal device 210. Specifically, the display unit 930 may include a touch screen 931 arranged on the front of the terminal device 210, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.

[0220] The touch screen 931 can be covered on the display screen 932, or the touch screen 931 and the display screen 932 can be integrated to realize the input and output functions of the terminal device 210, and the integrated touch screen can be referred to as a touch display screen. In this application, the display unit 930 can display the application and the corresponding operation steps.

[0221] The camera 940 can be used to capture static images, and the subject can publish the images taken by the camera 940 through the application. The camera 940 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 980 to convert it into a digital image signal.

[0222] The terminal device may further include at least one sensor 950, such as an acceleration sensor 951, a distance sensor 952, a fingerprint sensor 953, and a temperature sensor 954. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0223] The audio circuit 960, the speaker 961, and the microphone 962 can provide an audio interface between the object and the terminal device 210. The audio circuit 960 can transmit the electrical signal converted from the received audio data to the speaker 961, which is converted into a sound signal for output. The terminal device 210 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 962 converts the collected sound signal into an electrical signal, which is received by the audio circuit 960 and converted into audio data, and then the audio data is output to the communication component 910 to be sent to, for example, another terminal device 210, or the audio data is output to the memory 920 for further processing.

[0224] The Bluetooth module 970 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 970 to exchange data.

[0225] The processor 980 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 920 and calling data stored in the memory 920. In some embodiments, the processor 980 may include one or more processing units; the processor 980 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 980. In the present application, the processor 980 can run the operating system, application programs, user interface display and touch response, as well as the image retrieval method of the embodiment of the present application. In addition, the processor 980 is coupled to the display unit 930.

[0226] In some possible implementations, various aspects of the image retrieval method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the image retrieval method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 3A Follow the steps shown in .

[0227] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0228] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0229] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0230] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0231] The computer program for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program may be executed entirely on the user's computer device, partially on the user's computer device, as a separate software package, partially on the user's computer device and partially on a remote computer device, or entirely on the remote computer device. In the case of a remote computer device, the remote computer device may be connected to the user's computer device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer device (e.g., via the Internet using an Internet service provider).

[0232] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0233] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0234] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0235] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the command executed by the processor of the computer or other programmable data processing device produces a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0236] These computer program commands may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the commands stored in the computer-readable memory produce a manufactured product including a command device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0237] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the commands executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0238] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0239] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image retrieval method, characterized in that: include: Based on the original search text sent by the target object, at least one candidate search image is obtained; When it is determined that the degree of match between the at least one candidate search image and the original search text is lower than a first set threshold, adding prompt words representing the intention of the target object to the original search text, and generating a target search text based on the obtained prompt words; For each candidate search image whose search text matching degree is lower than the first set threshold, the following operations are performed respectively: Performing attention interaction between a candidate retrieval image and the target retrieval text to obtain a to-be-drawn area in the candidate retrieval image that is associated with the target retrieval text; The image content described by the target retrieval text is redrawn in the area to be drawn to obtain a corresponding target retrieval image.

2. The method according to claim 1, characterized in that The step of performing attention interaction between a candidate search image and the target search text to obtain a to-be-drawn area in the candidate search image that is associated with the target search text includes: Performing feature extraction on each target word in the target search text to obtain the text feature of each target word, and performing feature extraction on each pixel point in the candidate search image to obtain the image feature of each pixel point; Based on each text feature and each image feature, a plurality of attention weight features are obtained, each attention weight feature representing: a correlation degree between an image feature and each text feature; Based on the obtained attention weight features and the image features, a multimodal attention feature of each pixel is obtained, each multimodal attention feature represents: a correlation degree between a pixel and each target word contained in the target retrieval text; Based on the pixel points whose feature values ​​of the multimodal attention feature exceed a second set threshold, an area to be drawn associated with the target retrieval text is delineated in the candidate retrieval image.

3. The method according to claim 2, characterized in that The method obtains a plurality of attention weight features based on each text feature and each image feature; The following operations are performed for each of the image features to obtain a corresponding attention weight feature: Perform weighted summation of an image feature and each of the text features to obtain respective attention weight vectors; Concatenate the attention weight vectors of multiple dimensions to obtain a corresponding attention weight feature.

4. The method according to claim 3, characterized in that The image feature is weightedly summed with each of the text features to obtain respective attention weight vectors; For each of the text features, the following operations are performed: Multiplying the text vectors of each dimension in the text feature with the image vectors of the same dimension in the image feature to obtain respective weighted vectors; Add the weighted vectors of multiple dimensions to obtain a one-dimensional attention weight vector.

5. The method according to claim 2, characterized in that Before obtaining multiple attention weight features based on each text feature and each image feature, it also includes: Based on the pre-constructed first fully connected layer and the second fully connected layer, the dimensions of the text features and the image features are aligned to obtain the text features and the image features with the same dimensions.

6. The method according to claim 2, characterized in that Based on the obtained attention weight features and the image features, obtaining the multimodal attention features of each pixel; For each pixel's image feature and corresponding attention weight feature, the following operations are performed: The image vector of each dimension in an image feature is multiplied by the attention weight vector of the same dimension in the attention weight feature corresponding to a pixel point to obtain a multimodal attention feature containing a multidimensional vector.

7. The method according to claim 1, characterized in that The adding of prompt words representing the intention of the target object in the original search text includes: Performing part-of-speech detection on each original word in the original search text to obtain the respective part-of-speech types; After the first original word whose part of speech type is a noun, adding a prompt word representing the intention of the target object; When it is determined that a second original word adjacent to the first original word is a modified word, the prompt word is added after the second original word.

8. The method according to any one of claims 1 to 7, characterized in that: Redrawing the image content described by the target retrieval text in the area to be drawn to obtain a corresponding target retrieval image, including: According to the target search text, the pixel value of each pixel point in the area to be drawn is adjusted to draw the image content described by the target search text; Re-rendering the candidate retrieval image containing the image content to generate a corresponding target retrieval image.

9. An image retrieval device, characterized in that: include: An image retrieval module, used to obtain at least one candidate retrieval image based on the original retrieval text sent by the target object; a processing module, configured to add prompt words representing the intention of the target object to the original search text when it is determined that the matching degree between the at least one candidate search image and the original search text is lower than a first set threshold, and generate a target search text based on the obtained prompt words; The image drawing module is used to perform the following operations for each candidate search image whose search text matching degree is lower than the first set threshold: Performing attention interaction between a candidate retrieval image and the target retrieval text to obtain a to-be-drawn area in the candidate retrieval image that is associated with the target retrieval text; The image content described by the target retrieval text is redrawn in the area to be drawn to obtain a corresponding target retrieval image.

10. The device according to claim 9, characterized in that The image drawing module is used for: Performing feature extraction on each target word in the target search text to obtain the text feature of each target word, and performing feature extraction on each pixel point in the candidate search image to obtain the image feature of each pixel point; Based on each text feature and each image feature, a plurality of attention weight features are obtained, each attention weight feature representing: a correlation degree between an image feature and each text feature; Based on the obtained attention weight features and the image features, a multimodal attention feature of each pixel is obtained, each multimodal attention feature represents: a correlation degree between a pixel and each target word contained in the target retrieval text; Based on the pixel points whose feature values ​​of the multimodal attention feature exceed a second set threshold, an area to be drawn associated with the target retrieval text is delineated in the candidate retrieval image.

11. The device according to claim 10, characterized in that The image drawing module performs the following operations on each of the image features to obtain a corresponding attention weight feature: Perform weighted summation of an image feature and each of the text features to obtain respective attention weight vectors; Concatenate the attention weight vectors of multiple dimensions to obtain a corresponding attention weight feature.

12. The device according to claim 11, characterized in that The image drawing module performs the following operations for each of the text features: Multiplying the text vectors of each dimension in the text feature with the image vectors of the same dimension in the image feature to obtain respective weighted vectors; Add the weighted vectors of multiple dimensions to obtain a one-dimensional attention weight vector.

13. A computer device, characterized in that: It comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

14. A computer-readable storage medium, characterized in that: It includes program codes. When the program codes are run on a computer device, the program codes are used to make the computer device execute the steps of the method according to any one of claims 1 to 8.

15. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images