Image-text retrieval method and apparatus, terminal device, and storage medium

Abstract

The present application is applied to the technical field of data processing, and provides an image-text retrieval method and apparatus, a terminal device, and a storage medium. The image-text retrieval method comprises: acquiring images to be retrieved and a first retrieval text; performing object detection on said images to obtain a plurality of target images; calculating the similarity between each target image and the first retrieval text, and using the target images having similarities greater than or equal to a first preset threshold as candidate images; and respectively inputting each candidate image among all the candidate images and the first retrieval text into a first preset model, to obtain a retrieval result of each candidate image. In the image-text retrieval method, target images having a low correlation with the first retrieval text can be filtered out, and by inputting the candidate images and the first retrieval text into the first preset model for multi-modal matching, user intents can be more comprehensively understood, thereby improving the accuracy and efficiency of retrieval.

Description

Image and text retrieval methods, devices, terminal equipment, and storage media Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a method, apparatus, terminal device and storage medium for image and text retrieval.

[0002] This application claims priority to Chinese Patent Application No. 202410985616.2, filed on July 22, 2024, entitled "Image and Text Retrieval Method, Apparatus, Terminal Equipment and Storage Medium", the entire contents of which are incorporated herein by reference. Background Technology

[0003] With the development of internet technology, various applications and web pages generate a large number of images and texts every day, and these images and texts can be related. In practical applications, cross-modal retrieval algorithms can be used to retrieve the image most relevant to a given text description from a set of images.

[0004] However, when existing image and text retrieval schemes use a large amount of training data to train CLIP (Contrastive Language-Image Pretraining) models, the large-scale web page image and text data comes from a wide range of sources. It not only contains noise, inaccurate labels and low-quality samples, but also easily encounters a lot of data that does not match the specific application scenario during the CLIP model training process. This affects the training effect of the CLIP model and makes the CLIP model output retrieval results with low accuracy. Technical issues

[0005] This application provides a text and image retrieval method, apparatus, terminal device, and storage medium, which can solve the problem of low accuracy of retrieval results obtained by existing text and image retrieval schemes using the CLIP model.

[0006] In a first aspect, embodiments of this application provide a text and image retrieval method, including:

[0007] Retrieve the image to be searched and the first search text;

[0008] Target detection is performed on the image to be retrieved, resulting in multiple target images;

[0009] Calculate the similarity between each target image and the first search text, and select target images with a similarity greater than or equal to a first preset threshold as candidate images;

[0010] Each candidate image in the candidate image pool and the first search text are input into the first preset model to obtain the search results for each candidate image.

[0011] In one possible implementation of the first aspect, object detection is performed on the image to be retrieved to obtain multiple target images, including:

[0012] The images to be retrieved are preprocessed to obtain images that meet the input requirements of the object detection model;

[0013] The images to be retrieved that meet the input requirements of the object detection model are input into the object detection model to obtain the bounding box and confidence score of each object.

[0014] Bounding boxes that do not meet the first condition are filtered out, and the multiple bounding boxes and corresponding image regions obtained after filtering are used as target images; the first condition is that the overlap of the bounding boxes is less than the second preset threshold and the confidence of the target object is greater than the third preset threshold.

[0015] In one possible implementation of the first aspect, calculating the similarity between each target image and the first retrieved text includes:

[0016] The first search text is encoded to obtain a text vector, which is used to represent the semantic features of the first search text.

[0017] Each target image is image encoded to obtain an image vector, which is used to represent the visual features and potential semantic features of the target image;

[0018] Calculate the similarity between the text vector and the image vector corresponding to each target image to obtain the similarity between the first search text and each target image.

[0019] In one possible implementation of the first aspect, the image and text retrieval method further includes:

[0020] Convert the first search text into a first query text in question form;

[0021] Accordingly, each candidate image in the candidate image pool and the first search text are input into the first preset model to obtain the search results for each candidate image, including:

[0022] Each candidate image and the first query text are input into the first preset model to obtain the first retrieval result for each candidate image. The first retrieval result is used to characterize the matching result between the candidate image and the first query text.

[0023] In one possible implementation of the first aspect, converting the first search text into a first query text in the form of a question includes:

[0024] The first search text is input into the large language model to obtain the first query text in the form of a question.

[0025] In one possible implementation of the first aspect, after inputting each candidate image and the first query text into the first preset model to obtain the first retrieval result for each candidate image, the method further includes:

[0026] Retrieve the second search text;

[0027] Candidate images that do not meet the second condition are filtered out from the first search results to obtain the filtered images; the second condition is obtained based on the matching results between the candidate images and the first query text.

[0028] Each of the filtered images is input into the first preset model along with the second search text to obtain the search results for each filtered image.

[0029] In one possible implementation of the first aspect, the image and text retrieval method further includes:

[0030] Convert the second search text into a second query text in the form of a question;

[0031] Accordingly, each of the filtered images is input into the first preset model along with the second search text to obtain the search results for each filtered image, including:

[0032] Each of the filtered images is input into the first preset model along with the second query text to obtain a second retrieval result for each filtered image. The second retrieval result is used to characterize the matching result between the filtered image and the second query text.

[0033] Secondly, embodiments of this application provide an image and text retrieval device, including:

[0034] The acquisition module is used to acquire the image to be searched and the first search text;

[0035] The object detection module is used to perform object detection on the image to be retrieved, and obtain multiple object images;

[0036] The similarity calculation module is used to calculate the similarity between each target image and the first search text, and to select target images with a similarity greater than or equal to a first preset threshold as candidate images.

[0037] The processing module is used to input each candidate image and the first search text into the first preset model to obtain the search results for each candidate image.

[0038] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image and text retrieval method as described in any of the first aspects above.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image and text retrieval method as described in any of the first aspects above.

[0040] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the image and text retrieval method described in any of the first aspects above.

[0041] The beneficial effects of the embodiments of this application compared with the prior art are:

[0042] The image-text retrieval method provided in this application firstly filters out background information irrelevant to the retrieval intent by performing target detection on the image to be retrieved, thereby extracting multiple target images that may be related to the retrieval intent from the image to be retrieved, thus effectively narrowing the retrieval scope and reducing interference from irrelevant information. Then, by calculating the similarity between each target image and the text to be retrieved and setting a first preset threshold, multiple candidate images that may match the first retrieval text are selected to exclude target images irrelevant to the first retrieval text and retain target images with higher relevance to the text to be retrieved as candidate images. Finally, by inputting the candidate images and the first retrieval text into a first preset model for multimodal matching, the semantic features of the first retrieval text can be fully considered, and combined with the visual features and potential semantic features in the candidate images, thereby more comprehensively understanding the user's retrieval intent and improving the accuracy and efficiency of image-text retrieval.

[0043] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 is a schematic flowchart of an image and text retrieval method provided in an embodiment of this application;

[0046] Figure 2 is a flowchart illustrating a sub-step of step S102 provided in an embodiment of this application;

[0047] Figure 3 is a flowchart illustrating a sub-step of step S103 provided in an embodiment of this application;

[0048] Figure 4 is a schematic diagram of the overall steps of a text and image retrieval method provided in an embodiment of this application;

[0049] Figure 5 is a flowchart illustrating another image and text retrieval method provided in an embodiment of this application;

[0050] Figure 6 is a schematic diagram of the structure of a text and image retrieval device provided in an embodiment of this application;

[0051] Figure 7 is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Embodiments of the present invention

[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0053] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0054] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0055] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0056] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0057] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0058] Existing image and text retrieval solutions utilize large amounts of training data to train CLIP (Contrastive Language-Image Pretraining) models. However, due to the wide variety of sources of large-scale web page image and text data, there are not only noise-containing, inaccurate labels, and low-quality samples, but also a lot of data that is not matched with specific application scenarios during the CLIP model training process. This affects the training effect of the CLIP model, resulting in the CLIP model outputting search results with lower accuracy.

[0059] To address the aforementioned issues, the image-text retrieval method provided in this application firstly filters out irrelevant background information by performing target detection on the image to be retrieved. This allows for the extraction of multiple target images potentially related to the retrieval intent, effectively narrowing the retrieval scope and reducing interference from irrelevant information. Next, by calculating the similarity between each target image and the text to be retrieved and setting a first preset threshold, multiple candidate images potentially matching the first retrieval text are selected. This excludes target images irrelevant to the first retrieval text and retains target images with higher relevance to the text as candidate images. Finally, by inputting the candidate images and the first retrieval text into a first preset model for multimodal matching, the semantic features of the first retrieval text are fully considered, combined with the visual features and potential semantic features of the candidate images, thereby more comprehensively understanding the user's retrieval intent and improving the accuracy and efficiency of image-text retrieval.

[0060] Referring to Figure 1, which is a schematic flowchart of the image and text retrieval method provided in the embodiment of this application, the image and text retrieval method may include the following steps S101 to S104 as an example and not a limitation.

[0061] Step S101: Obtain the image to be searched and the first search text.

[0062] The image to be retrieved can be an image from a specific target scene. Images from a target scene may contain various information, such as different objects, people, and backgrounds. For example, when the target scene is a vehicle driving scene, images from such a scene might include various types of vehicles, road conditions, traffic signs, and pedestrian behavior. The first search text is a text description related to a portion of the content in the image to be retrieved. In the example of a vehicle driving scene, the first search text might describe a specific type of vehicle (e.g., "red sedan"), a specific road condition (e.g., "congested highway"), a traffic sign (e.g., a "speed limit 60" sign), or a specific pedestrian behavior (e.g., "pedestrians are crossing the road").

[0063] Specifically, the image to be searched can be obtained by the user uploading an image to the application or webpage; or by capturing images in real time from sensors such as cameras. The first search text can be obtained by the user manually entering text in a text box on the application or webpage; or by capturing the user's voice and converting it into text. Besides the examples above, other methods can also be used to obtain the image to be searched and the first search text; the specific methods for obtaining these elements are not limited here.

[0064] Step S102: Perform target detection on the image to be retrieved to obtain multiple target images.

[0065] The purpose of object detection is to identify target objects (such as people, vehicles, animals, etc.) in an image to be retrieved and to determine the location of these target objects within the image. Specifically, object detection algorithms can be used to perform object detection on the image to be retrieved. Object detection algorithms include traditional machine learning algorithms and deep learning algorithms. Traditional machine learning algorithms include, for example, the HOG algorithm with an SVM classifier, and the DPM algorithm. Deep learning algorithms include, for example, the R-CNN algorithm, the Fast R-CNN algorithm, and the YOLO algorithm. In this embodiment, a suitable algorithm can be selected to implement the object detection task according to specific needs and scenarios.

[0066] In some embodiments, FIG2 shows a flowchart of the sub-steps of step S102 provided in the embodiments of this application. As shown in FIG2, step S102 may include steps S1021 to S1023.

[0067] Step S1021: Preprocess the image to be retrieved to obtain an image that meets the input requirements of the object detection model.

[0068] To ensure the accuracy of object detection, the images to be retrieved first need to be preprocessed to obtain images that meet the input requirements of the object detection model. Preprocessing steps may include grayscale conversion and resizing. When the image to be retrieved is a color image, grayscale conversion transforms it into a grayscale image containing only luminance information and not color information, thereby reducing the data complexity of the image and improving data processing speed.

[0069] After converting the image to grayscale, the object detection model has specific requirements for the size of the input image. Therefore, before inputting the grayscale image into the object detection model, the size of the grayscale image needs to be adjusted to match the input size requirements of the object detection model. This means resizing the grayscale image, for example, by scaling or cropping it, to obtain an image that meets the input size requirements of the object detection model.

[0070] Step S1022: Input the image to be retrieved that meets the input requirements of the object detection model into the object detection model to obtain the bounding box and confidence score of each object.

[0071] After obtaining the image to be retrieved that meets the input requirements (grayscale and size requirements) of the object detection model, the image is input into the object detection model, such as the YOLO model. The specific steps of the YOLO model in performing object detection include feature extraction, grid generation and region classification, bounding box generation, confidence scoring, and filtering out bounding boxes that do not meet the first condition.

[0072] Specifically, first, the YOLO model uses a deep convolutional neural network (such as Darknet) to extract features from the input image (the image to be retrieved that meets the input requirements of an object detection model). The deep convolutional neural network extracts visual features from the input image layer by layer, from basic edge and texture information to more advanced shape and object patterns. Then, after feature extraction, the YOLO model divides the input image into multiple grids, each grid responsible for predicting a target object within a certain region. Each grid predicts a fixed number of bounding boxes and generates a class probability distribution and a confidence score for each bounding box. The confidence score reflects the probability that the YOLO model believes the bounding box contains a target object. Next, when the YOLO model determines that a particular grid may contain a target object, it generates a bounding box based on the predicted bounding box parameters (such as center coordinates, width, and height) to mark the target object's location in the input image. The size and position of this bounding box are predicted by the YOLO model based on the extracted features and the parameters learned during the YOLO model's training. Finally, for each detected target object (i.e., each bounding box), the YOLO model assigns a confidence score. This confidence score reflects the probability that the YOLO model predicts that a target object actually exists within the bounding box. It is calculated by the YOLO model based on the degree of overlap between the bounding box and the real target object (such as IOU, Intersection over Union) and the classification probability of the bounding box by the YOLO model.

[0073] Step S1023: Filter out bounding boxes that do not meet the first condition, and take the multiple bounding boxes and corresponding image regions obtained after filtering as target images; the first condition is that the overlap of the bounding boxes is less than the second preset threshold and the confidence of the target object is greater than the third preset threshold.

[0074] Because the YOLO model may generate multiple overlapping bounding boxes around the same target object during object detection, it is necessary to filter out bounding boxes whose overlap is greater than or equal to a second preset threshold and whose target object confidence is less than or equal to a third preset threshold. Multiple bounding boxes with overlap less than the second preset threshold and target object confidence greater than the third preset threshold are retained. The retained bounding boxes and their corresponding image regions are used as target images. If multiple bounding boxes are retained, there are multiple target images.

[0075] In the YOLO model, the Non-Maximum Suppression (NMS) algorithm is used to filter out bounding boxes that do not meet the first condition. The specific process of the Non-Maximum Suppression (NMS) algorithm is as follows:

[0076] 1. Removing bounding boxes: First, from all the bounding boxes obtained by the object detection algorithm, remove the bounding boxes with a confidence level less than or equal to the third preset threshold (e.g., 0.2), and only keep the bounding boxes with higher confidence levels.

[0077] 2. Sorting: Sort the remaining bounding boxes according to their confidence scores and select the bounding box with the highest confidence score as the reference box.

[0078] 3. Calculate the overlap: Calculate the overlap between each bounding box in the remaining bounding boxes and the reference box. The overlap is usually measured using the Intersection over Union (IoU), which is the ratio of the intersection area to the union area of ​​the two bounding boxes.

[0079] 4. Suppress overlapping boxes: For each non-reference box, if its IoU with the reference box is greater than or equal to the second preset threshold (e.g., 0.5), it is considered that the bounding box and the reference box detect the same target object, and it is removed from the candidate box list.

[0080] 5. Iterative processing: Remove the current reference box from the candidate box list, and then select the next bounding box with the highest confidence from the remaining candidate boxes as the new reference box. Repeat the above steps (calculate IoU, suppress overlapping boxes) until all bounding boxes except the reference box have been traversed in the remaining bounding boxes.

[0081] 6. Output results: After the above filtering process, the remaining bounding boxes are the final detection results. The target image regions corresponding to these bounding boxes are the locations of the target objects identified by the target detection model (such as YOLO). At this time, multiple target images are obtained.

[0082] Step S103: Calculate the similarity between each target image and the first search text, and select target images with a similarity greater than or equal to a first preset threshold as candidate images.

[0083] The similarity between a target image and the first search text reflects the degree of semantic association between the two. A high similarity indicates a strong match between the visual content of the target image and the semantic content described in the first search text. For example, if the first search text describes "a cute cat," a highly similar target image is likely to depict a cat. Conversely, a low similarity indicates a weak match between the visual content of the target image and the semantic content described in the first search text. For instance, if the first search text describes "a cute cat," a low-similar target image is likely to depict content semantically unrelated to "a cute cat," such as a plant or other non-animal element.

[0084] There are various methods for calculating the similarity between a target image and the first search text, such as the CLIP model, mean hash algorithm, and pHash algorithm. In this embodiment, the most suitable method can be flexibly selected according to specific application requirements and scenarios to achieve the task of calculating the similarity between the target image and the first search text. For example, to obtain a more accurate and comprehensive semantic matching degree, the CLIP model can be selected to calculate the similarity between each target image and the first search text, so as to ensure the accuracy and reliability of the search results.

[0085] In some embodiments, FIG3 shows a flowchart of the sub-steps of step S102 provided in the embodiments of this application. As shown in FIG3, step S103 may include steps S1031 to S1033.

[0086] Step S1031: Encode the first search text to obtain a text vector, which is used to represent the semantic features of the first search text.

[0087] Specifically, the first retrieved text is input into the text encoder of the CLIP model. First, the text encoder performs word segmentation on the first retrieved text, breaking it down into sub-word units. Then, each sub-word unit passes through an embedding layer, which converts it into a fixed-dimensional embedding vector. This embedding vector can be seen as the position of the sub-word unit in the semantic space. Next, the embedding vectors are fed into a series of neural network layers, which may include convolutional layers, recurrent neural network layers (such as LSTM and GRU), or self-attention mechanisms (such as multi-head self-attention in Transformer). These neural network layers further capture the sequence and contextual information in the first retrieved text, extracting deep semantic features. Finally, the text encoder outputs a text vector containing the semantic features of the first retrieved text. This text vector is a high-dimensional continuous vector that effectively represents key information in the text, such as topic, sentiment, and intent.

[0088] Step S1032: Encode each target image to obtain an image vector, which is used to represent the visual features and potential semantic features of the target image.

[0089] Specifically, the target image is input into the CLIP model's image encoder. First, the target image undergoes a series of preprocessing steps, such as scaling, cropping, and normalization, to ensure the icon image's size and format meet the requirements of the image encoder. Next, since the CLIP model uses numerous image-text pairs as supervision during training, it establishes semantic associations between images and text by learning the similarity between image and text vectors. Therefore, in the image encoding stage, the image encoder utilizes structures such as convolutional neural networks (CNNs) to extract visual features from the target image. These visual features may include low-level features such as edges, corners, and textures, as well as higher-level abstract features such as shapes and objects. These visual features are encoded into a series of feature maps to capture local and global information of the target image. Finally, the image encoder further processes these feature maps to generate an image vector containing both the visual features and latent semantic features of the target image. This image vector is a high-dimensional continuous vector that effectively represents key information in the image, including color, texture, shape, objects, and latent semantic features learned by the model and associated with the text.

[0090] Step S1033: Calculate the similarity between the text vector and the image vector corresponding to each target image to obtain the similarity between the first search text and each target image.

[0091] Specifically, after obtaining the text vector of the first retrieved text using the CLIP model's text encoder and the image vector of the target image using the CLIP model's image encoder, the CLIP model uses cosine similarity to calculate the similarity between the text vector and the image vector. The formula for calculating cosine similarity is: For text vector A and image vector B, cosine similarity = (A·B) / (||A|| * ||B||), where A·B represents the dot product of text vector A and image vector B, ||A|| represents the modulus (length) of text vector A, and ||B|| represents the modulus (length) of image vector B. The value of cosine similarity ranges from -1 to 1; a larger value indicates greater similarity between text vector A and image vector B. In the CLIP model, when the calculated cosine similarity is close to 1, it indicates a high semantic match between the text vector and image vector; when the similarity is close to -1, it indicates a complete semantic mismatch; and when the similarity is close to 0, it indicates no significant semantic correlation between the text vector and image vector.

[0092] After obtaining the similarity between the text vector and the image vector corresponding to each target image, a first preset threshold value is set, for example, the first preset threshold is set to 0.2, and target images with a similarity less than 0.2 are filtered out, and target images with a similarity greater than or equal to 0.2 are obtained and used as candidate images.

[0093] It should be noted that the first preset threshold, the second preset threshold, and the third preset threshold can be set according to actual conditions and requirements. This application embodiment does not impose any limitations on the specific values ​​of the first preset threshold, the second preset threshold, and the third preset threshold.

[0094] Step S104: Input each candidate image and the first search text into the first preset model to obtain the search result for each candidate image.

[0095] The first preset model is a pre-trained machine learning model specifically designed for image and text retrieval matching tasks. In this embodiment, the first preset model is, for example, a Multi-Modal Large Language Model (MM-LLM). The MM-LLM model can simultaneously process data of different modalities (such as text, images, etc.) and learn the relationships between these different modalities.

[0096] Specifically, candidate images and the first search text are input into the MM-LLM model. The MM-LLM model's text encoder converts the first search text into text vectors, which capture the semantic features of the first search text. The MM-LLM model's image encoder converts candidate images into image vectors, which capture the visual and semantic features of the candidate images. Then, the MM-LLM model calculates the similarity between the text vectors and the image vectors of the candidate images, obtaining the similarity between the candidate images and the first search text. All candidate images and the first search text are sequentially input into the MM-LLM model, and the MM-LLM model outputs the similarity between each candidate image and the first search text. The similarity result is used as the retrieval result for the candidate images and the first search text.

[0097] In some embodiments, the image and text retrieval method further includes:

[0098] Convert the first search text into a first query text in the form of a question.

[0099] Converting the initial search text into a question-based query text offers several advantages in multimodal image-text retrieval, including enhanced clarity, reduced false positive rates, improved user experience, support for multimodal retrieval, and increased search efficiency. This conversion method helps to better meet users' search needs and improve the overall quality of search services.

[0100] In a specific example, transforming the initial search text "a person squatting next to a trash can" into the question-based query text "Does the image contain a person squatting next to a trash can?" reveals that the question-based query text more directly expresses the user's search intent; they want to find images that contain a specific scene (squatting next to a trash can) and a subject (a person). The initial query text reduces the false positive rate of potentially irrelevant results during the search process. Only images that meet the condition "a person squatting next to a trash can" will be retrieved, thus improving the accuracy of the search results.

[0101] In some embodiments, converting the first search text into a first query text in the form of a question includes:

[0102] The first search text is input into the large language model to obtain the first query text in the form of a question.

[0103] Among them, the Large Language Model (LLM) possesses a large number of parameters and an extensive knowledge base, enabling it to understand and generate complex natural language text. By training on large-scale corpora, the LLM learns to understand the syntax, semantics, and contextual relationships of a language, and is able to generate sentences that conform to language rules.

[0104] When the initial search text is input into the LLM model, the model analyzes its meaning and attempts to transform it into a more explicit and specific question. This transformation process may involve multiple steps, such as syntactic analysis, entity recognition, and semantic reasoning. LLM models leverage their powerful natural language processing capabilities to integrate these steps, generating an initial query text that both matches the user's search intent and is suitable for processor processing.

[0105] After converting the first search text into the first query text, step S104 may include step S1041.

[0106] Step S1041: Input each candidate image and the first query text into the first preset model to obtain the first retrieval result for each candidate image. The first retrieval result is used to characterize the matching result between the candidate image and the first query text.

[0107] Because the first query text better understands the user's search intent than the first search result text, after inputting candidate images and the first query text into the MM-LLM model, the MM-LLM model analyzes the association between the user's search intent in the first query text and the candidate images, and determines the matching result between the first query text and the candidate images based on this association. If the MM-LLM model deems the relevance between the candidate images and the first query text high enough to satisfy the user's search intent, then a match will be obtained; conversely, if the MM-LLM model deems the relevance between the candidate images and the first query text insufficient and does not match the user's search intent, then a mismatch will be obtained.

[0108] In a specific example, Figure 4 illustrates the overall flowchart of the image-text retrieval method provided in this application embodiment. As shown in Figure 4, the image to be retrieved is a vehicle road scene image, and the first search text is "a person squatting next to a trash can". After obtaining the image to be retrieved and the first search text, firstly, the YOLO algorithm is used to perform target retrieval on the image to be retrieved, resulting in multiple target images. Next, the CLIP model is used to calculate the similarity between each target image and the first search text, filtering out target images with a similarity value less than 0.2, and obtaining target images with a similarity value greater than or equal to 0.2. These target images with a similarity value greater than or equal to 0.2 are then used as candidate images. Afterwards, the candidate images and the first search text can be input into the MM-LLM model to obtain the retrieval results of the candidate images and the first search text. This retrieval result is the similarity between the candidate images and the first search text. To more accurately determine the matching degree between candidate images and the first search text, this embodiment of the application uses an LLM model to convert the first search text into the first query text, that is, converting "a person squatting next to a trash can" into "is there a person squatting next to a trash can in the image?" Then, the first query text "is there a person squatting next to a trash can in the image?" and the candidate images are input into the MM-LLM model to obtain the matching result of the first query text and the candidate images output by the MM-LLM. If the MM-LLM model outputs "no", it means that the first query text and the candidate images do not match; if the MM-LLM model outputs "yes", it means that the first query text and the candidate images match.

[0109] In some embodiments, FIG5 shows a flowchart of the sub-steps of step S102 provided in the embodiments of this application. As shown in FIG5, the image and text retrieval method further includes steps S105 to S107.

[0110] Step S105: Obtain the second search text.

[0111] During image and text search, users often need to change their search intent or add new criteria according to their needs. For example, when a user wants to change their intent from "a person squatting next to a trash can" to "a person squatting next to a trash can wearing red clothes," they will manually enter this new search criterion, or second search text, into a text box on the application or webpage. The processor will capture this second search text in real time.

[0112] Step S106: Filter out candidate images that do not meet the second condition in the first search result to obtain the filtered images; the second condition is obtained based on the matching result between the candidate images and the first query text.

[0113] Specifically, after obtaining the second search text, the candidate images of "no one is squatting next to the trash can in the image" are first filtered out based on the first search result, and the candidate images of "someone is squatting next to the trash can in the image" are retained as the filtered images, that is, the candidate images that successfully match the first query text are retained.

[0114] Step S107: Input each of the filtered images and the second search text into the first preset model to obtain the search results for each filtered image.

[0115] Specifically, the candidate image that successfully matches the first query text and the second search text "a person squatting next to a trash can wearing red clothes" are input into the MM-LLM model. The MM-LLM model outputs the retrieval results of the similarity between the candidate image that successfully matches the first query text and the second search text.

[0116] In some embodiments, the image and text retrieval method further includes:

[0117] Convert the second search text into a second query text in the form of a question.

[0118] Specifically, inputting "a person squatting next to a trash can and wearing red clothes" into the LLM model, the LLM model outputs "is there a person wearing red clothes squatting next to a trash can in the image", that is, the LLM model outputs the second query text.

[0119] After converting the second search text into the second query text, step S104 may include step S10411.

[0120] Step S10411: Input each of the filtered images and the second query text into the first preset model to obtain the second retrieval result for each filtered image. The second retrieval result is used to characterize the matching result between the filtered image and the second query text.

[0121] Since the second query text has a better understanding of the user's search intent than the second search text, after the filtered images and the second query text are input into the MM-LLM model, the MM-LLM model analyzes the association between the user's search intent in the second query text and the filtered images, and determines the matching result between the second query text and the filtered images based on the association between the user's search intent in the second query text and the filtered images.

[0122] In this embodiment of the application, the second query text "Is there a person wearing red clothes squatting next to a trash can in the image?" and the filtered images are input into the MM-LLM model to obtain the matching result of the second query text and the filtered images output by the MM-LLM. If the MM-LLM model outputs "no", it means that the second query text and the filtered images do not match; if the MM-LLM model outputs "yes", it means that the second query text and the filtered images match.

[0123] In this embodiment, the MM-LLM model enables interactive retrieval, allowing users to dynamically modify search criteria during the retrieval process, such as changing or adding new criteria. Subsequently, the retrieval is further refined based on the first search result without restarting the entire process. By filtering out candidate images that successfully match the first query text, the scope of the second search is narrowed, thereby improving retrieval efficiency. By progressively refining the search criteria, false positives and false negatives can be reduced. Users can adjust the criteria based on the first search result for more precise matching, improving the accuracy of the search results. Extensive experimental data from the applicant shows that the image-text retrieval method provided in this embodiment can improve the recall accuracy of image-text retrieval, with an accuracy greater than 90% for the first 20 recalled targets. It can also add support for search commands to achieve interactive retrieval. Furthermore, it can utilize a small LLM (such as 1B parameter size), thereby keeping the inference memory usage below 12GB and reducing hardware requirements.

[0124] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0125] Corresponding to the image and text retrieval method in the above embodiments, Figure 6 is a structural block diagram of the image and text retrieval device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0126] Referring to Figure 6, the image and text retrieval device includes:

[0127] The acquisition module 61 is used to acquire the image to be searched and the first search text.

[0128] The object detection module 62 is used to perform object detection on the image to be retrieved, and obtain multiple object images.

[0129] The similarity calculation module 63 is used to calculate the similarity between each target image and the first search text, and to select target images with a similarity greater than or equal to a first preset threshold as candidate images.

[0130] The processing module 64 is used to input each candidate image and the first search text into the first preset model to obtain the search result for each candidate image.

[0131] In some embodiments, the target detection module 62 is further configured to:

[0132] The images to be retrieved are preprocessed to obtain images that meet the input requirements of the object detection model;

[0133] The images to be retrieved that meet the input requirements of the object detection model are input into the object detection model to obtain the bounding box and confidence score of each object.

[0134] Bounding boxes that do not meet the first condition are filtered out, and the multiple bounding boxes and corresponding image regions obtained after filtering are used as target images; the first condition is that the overlap of the bounding boxes is less than the second preset threshold and the confidence of the target object is greater than the third preset threshold.

[0135] In some embodiments, the similarity calculation module 63 is further configured to:

[0136] The first search text is encoded to obtain a text vector, which is used to represent the semantic features of the first search text.

[0137] Each target image is image encoded to obtain an image vector, which is used to represent the visual features and potential semantic features of the target image;

[0138] Calculate the similarity between the text vector and the image vector corresponding to each target image to obtain the similarity between the first search text and each target image.

[0139] In some embodiments, the image and text retrieval device further includes:

[0140] The first conversion module 65 is used to convert the first search text into a first query text in the form of a question.

[0141] Accordingly, the processing module 64 is also used for:

[0142] Each candidate image and the first query text are input into the first preset model to obtain the first retrieval result for each candidate image. The first retrieval result is used to characterize the matching result between the candidate image and the first query text.

[0143] In some embodiments, the conversion module 65 is further configured to:

[0144] The first search text is input into the large language model to obtain the first query text in the form of a question.

[0145] In some embodiments, the image and text retrieval device further includes:

[0146] The second search text acquisition module 66 is used to acquire the second search text.

[0147] The filtering module 67 is used to filter out candidate images that do not meet the second condition in the first search result, and obtain the filtered images; the second condition is obtained based on the matching result between the candidate images and the first query text.

[0148] The input / output module 68 is used to input each of the filtered images and the second search text into the first preset model to obtain the search results for each filtered image.

[0149] In some embodiments, the image and text retrieval device further includes:

[0150] The second conversion module 69 is used to convert the second search text into a second query text in the form of a question.

[0151] Input / output module 68 is also used for:

[0152] Each of the filtered images is input into the first preset model along with the second query text to obtain a second retrieval result for each filtered image. The second retrieval result is used to characterize the matching result between the filtered image and the second query text.

[0153] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0154] In addition, the image and text retrieval device shown in Figure 6 can be a software unit, a hardware unit, or a combination of software and hardware built into an existing terminal device, or it can be integrated into the terminal device as an independent component, or it can exist as an independent terminal device.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0156] Figure 7 is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. As shown in Figure 7, the terminal device 7 of this embodiment includes: at least one processor 70 (only one is shown in Figure 7), a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70. When the processor 70 executes the computer program 72, it implements the steps in any of the above-described embodiments of the image and text retrieval method.

[0157] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that Figure 7 is merely an example of terminal device 7 and does not constitute a limitation on terminal device 7. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0158] The processor 70 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] In some embodiments, the memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. In other embodiments, the memory 71 may be an external storage device of the terminal device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 7. Furthermore, the memory 71 may include both internal and external storage units of the terminal device 7. The memory 71 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0160] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0161] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0162] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0163] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0164] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of retrieving text and graphics, characterized by, The method comprises the following steps: obtaining a picture to be searched and a first search text; performing target detection on the picture to be searched to obtain a plurality of target pictures; calculating the similarity between each target picture and the first search text, and taking the target picture with a similarity greater than or equal to a first preset threshold as a candidate picture; inputting each candidate picture and the first search text into a first preset model to obtain a search result of each candidate picture.

2. The document retrieval method according to claim 1, characterized in that, The target detection on the picture to be searched to obtain a plurality of target pictures comprises the following steps: preprocessing the picture to be searched to obtain a picture to be searched meeting the input requirements of a target detection model; inputting the picture to be searched meeting the input requirements of the target detection model into the target detection model to obtain a bounding box and a confidence degree of each target object; screening out the bounding boxes that do not meet a first condition, and taking the plurality of bounding boxes obtained after screening out and the corresponding image regions as target pictures; the first condition is that the overlap degree of the bounding box is less than a second preset threshold and the confidence degree of the target object is greater than a third preset threshold.

3. The document retrieval method according to claim 1, characterized by, The calculation of the similarity between each target picture and the first search text comprises the following steps: text encoding is performed on the first search text to obtain a text vector, which is used to represent the semantic features of the first search text; image encoding is performed on each target picture to obtain an image vector, which is used to represent the visual features and potential semantic features of the target picture; the similarity between the text vector and the image vector corresponding to each target picture is calculated to obtain the similarity between the first search text and each target picture.

4. The document retrieval method according to claim 1, characterized by, The image-text search method further comprises the following steps: converting the first search text into a first query text in the form of a question; correspondingly, the inputting of each candidate picture and the first search text into the first preset model to obtain a search result of each candidate picture comprises the following steps: inputting each candidate picture and the first query text into the first preset model to obtain a first search result of each candidate picture, wherein the first search result is used to represent the matching result of the candidate picture and the first query text.

5. The document retrieval method according to claim 4, wherein The conversion of the first search text into a first query text in the form of a question comprises the following steps: inputting the first search text into a large language model to obtain a first query text in the form of a question.

6. The document retrieval method according to claim 4, wherein After the inputting of each candidate picture and the first query text into the first preset model to obtain a first search result of each candidate picture, the method further comprises the following steps: obtaining a second search text; screening out the candidate pictures corresponding to the first search result that does not meet a second condition to obtain screened pictures; the second condition is obtained according to the matching result of the candidate picture and the first query text; inputting each screened picture and the second search text into the first preset model to obtain a search result of each screened picture.

7. The document retrieval method according to claim 6, wherein The picture-text retrieval method further comprises: converting the second retrieval text into a second query text in the form of a question; Correspondingly, the second retrieval text is input into the first preset model together with each of the filtered pictures to obtain a retrieval result of each of the filtered pictures, comprising: the second retrieval text is input into the first preset model together with each of the filtered pictures to obtain a second retrieval result of each of the filtered pictures, wherein the second retrieval result is used to represent the matching result of the filtered picture and the second query text.

8. A document retrieval apparatus characterized by comprising: comprising: an acquisition module configured to acquire a to-be-retrieved picture and a first retrieval text; a target detection module configured to perform target detection on the to-be-retrieved picture to obtain a plurality of target pictures; a similarity calculation module configured to calculate the similarity between each of the target pictures and the first retrieval text, and take a target picture with a similarity greater than or equal to a first preset threshold as a candidate picture; a processing module configured to input each of the candidate pictures into a first preset model together with the first retrieval text to obtain a retrieval result of each of the candidate pictures.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to implement the method of any one of claims 1 to 7.

Images