Image retrieval method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image retrieval method and device, equipment and a storage medium, and relates to the technical field of deep learning. According to the specific implementation scheme, a query text is obtained; inputting the query text and the first cue word into a pre-trained first large language model to obtain a dependency degree of a query intention of the query text depending on each preset query dimension; each preset query dimension comprises any multiple of the following items: a semantic dimension representing text semantics, a visual dimension representing image visual features, and a structured dimension representing structured information; determining a target query dimension from the preset query dimensions; the dependency degree corresponding to the target query dimension is higher than that corresponding to other preset query dimensions; and searching an image matched with the query text from the to-be-queried image by utilizing a query mode corresponding to the target query dimension. Therefore, through the scheme, the retrieval precision of cross-modal retrieval can be improved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to an image retrieval method, apparatus, device, and storage medium. Background Technology

[0002] Modern smart home and security devices often record a large amount of user behavior information, including information in multiple modalities such as images, videos, and text. Currently, multimodal model image encoders are typically used to encode the query image into an image vector and store it in a database. When a user needs to accurately search for relevant images using natural language input, a multimodal model text encoder is used to encode the query text into a text vector, and the image vector that is closest to the text vector is found from the database to achieve semantic search.

[0003] However, cross-modal information alignment often suffers from a natural semantic gap, resulting in lower accuracy of retrieval results.

[0004] Therefore, improving the retrieval accuracy of cross-modal retrieval has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide an image retrieval method, apparatus, device, and storage medium to improve the retrieval accuracy of cross-modal retrieval. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide an image retrieval method, the method comprising:

[0007] Get the query text;

[0008] The query text and the first prompt word are input into a pre-trained first language model to obtain the query intent of the query text depending on the degree of dependence of each preset query dimension; wherein, each preset query dimension includes any number of the following: semantic dimension representing text semantics, visual dimension representing image visual features, and structured dimension representing structured information; the structured information includes at least one of the following: the category of the main object in the image, the behavior of the main object in the image, and the scene in the image;

[0009] The target query dimension is determined from each preset query dimension; wherein the dependency degree corresponding to the target query dimension is higher than that corresponding to other preset query dimensions.

[0010] Using the query methods corresponding to the target query dimensions, images matching the query text are retrieved from the images to be queried; wherein, the query method corresponding to the semantic dimension is: determining the query images whose corresponding text vectors match the query text; the query method corresponding to the visual dimension is: determining the query images whose corresponding multimodal vectors match the query text; and the query method corresponding to the structured dimension is: determining the query images whose corresponding structured information matches the query text.

[0011] Optionally, using the query method corresponding to the target query dimension, images matching the query text are retrieved from the images to be queried, including:

[0012] If the target query dimension includes a semantic dimension, then a text vector matching the query text is retrieved from a preset database; if the target query dimension includes a visual dimension, then a multimodal vector matching the query text is retrieved from a preset database; if the target query dimension includes a structured dimension, then structured information matching the query text is retrieved from a preset database.

[0013] The preset database stores the text vector of the descriptive text of the image to be queried, the multimodal vector of the image to be queried, and the structured information of the image to be queried.

[0014] Based on the images to which the queried text vector belongs, the images to which the multimodal vector belongs, and the images to which the structured information belongs, the images that match the queried text are determined.

[0015] Optionally, based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs, the image that matches the query text is determined, including:

[0016] If a text vector is found, the descriptive text and structured information of the image to which the text vector belongs are combined to obtain the object to be sorted;

[0017] If structured information is found, the found structured information and the description text of the image to which the structured information belongs are combined to obtain the object to be sorted;

[0018] If a multimodal vector is found, the image to which the multimodal vector belongs is obtained, thus obtaining the object to be sorted;

[0019] The obtained objects to be sorted and the query text are input into the reordering model to obtain the similarity between the input objects to be sorted and the query text;

[0020] The images belonging to the preset number of objects to be sorted with the highest similarity are determined as the images that match the query text.

[0021] Optionally, before determining the image matching the query text based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs, the method further includes:

[0022] If the target query dimension includes a structured dimension, and no structured information matching the query text is found in the preset database, then a text vector matching the query text is retrieved from the preset database.

[0023] Optionally, obtaining the query text includes:

[0024] Retrieve the original query statement;

[0025] The original query statement and the second prompt word are input into the pre-trained second language model to obtain the query text; wherein, the second prompt word is used to instruct the second language model to extract the core words for querying from the original query statement and concatenate them into the query text according to the natural word order.

[0026] Optionally, the text vector, structured information, and multimodal vector of the image to be queried are constructed through the following steps:

[0027] Each image to be queried is input into a pre-trained graph-to-text model to obtain the structured information and descriptive text of the image to be queried, and the obtained descriptive text is encoded into a text vector;

[0028] Each query image is input into a pre-trained multimodal model to obtain the multimodal vector of that query image.

[0029] Secondly, embodiments of this application provide an image retrieval device, the device comprising:

[0030] The retrieval module is used to retrieve the query text;

[0031] The model processing module is used to input the query text and the first prompt word into a pre-trained first large language model to obtain the degree of dependence of the query intent of the query text on each preset query dimension; wherein, each preset query dimension includes any multiple of the following: semantic dimension representing text semantics, visual dimension representing image visual features, and structured dimension representing structured information; the structured information includes at least one of the following: the category of the main object in the image, the behavior of the main object in the image, and the scene in the image;

[0032] The determination module is used to determine the target query dimension from each preset query dimension; wherein the dependency degree corresponding to the target query dimension is higher than the dependency degree corresponding to other preset query dimensions;

[0033] The retrieval module is used to retrieve images matching the query text from the images to be queried using the query methods corresponding to the target query dimensions; wherein, the query method corresponding to the semantic dimension is: determining the query images whose corresponding text vectors match the query text; the query method corresponding to the visual dimension is: determining the query images whose corresponding multimodal vectors match the query text; and the query method corresponding to the structured dimension is: determining the query images whose corresponding structured information matches the query text.

[0034] Optionally, the retrieval module includes:

[0035] The query submodule is used to: if the target query dimension includes a semantic dimension, query a text vector that matches the query text from a preset database; if the target query dimension includes a visual dimension, query a multimodal vector that matches the query text from a preset database; if the target query dimension includes a structured dimension, query structured information that matches the query text from a preset database.

[0036] The preset database stores the text vector of the descriptive text of the image to be queried, the multimodal vector of the image to be queried, and the structured information of the image to be queried.

[0037] The determination submodule is used to determine the image that matches the query text based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs.

[0038] Optionally, the determining submodule includes:

[0039] The first combination unit is used to combine the descriptive text and structured information of the image to which the text vector belongs, if a text vector is found, to obtain the object to be sorted.

[0040] The second combination unit is used to combine the queried structured information and the description text of the image to which the structured information belongs if structured information is found, to obtain the object to be sorted.

[0041] The acquisition unit is used to acquire the image to which the multimodal vector belongs if a multimodal vector is found, and obtain the object to be sorted.

[0042] The sorting unit is used to input the obtained objects to be sorted and the query text into the rearrangement model to obtain the similarity between the input objects to be sorted and the query text.

[0043] The output unit is used to determine the images of the preset number of objects to be sorted with the highest similarity as the images that match the query text.

[0044] Optionally, the retrieval module further includes:

[0045] The text matching submodule is used to perform the following action before the determining submodule performs the determination of the image that matches the query text based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs: if the target query dimension includes a structured dimension and no structured information matching the query text is found in the preset database, then the text vector that matches the query text is retrieved from the preset database.

[0046] Optionally, the acquisition module includes:

[0047] The `get` submodule is used to retrieve the original query statement;

[0048] The conversion submodule is used to input the original query statement and the second prompt word into the pre-trained second language model to obtain the query text; wherein, the second prompt word is used to instruct the second language model to extract the core words for querying from the original query statement and concatenate them into the query text according to the natural word order.

[0049] Optionally, the text vector, structured information, and multimodal vector of the image to be queried are constructed through the following modules:

[0050] The first output module is used to input each query image into the pre-trained graph-to-text model to obtain the structured information and descriptive text of the query image, and encode the obtained descriptive text into a text vector.

[0051] The second output module is used to input each query image into a pre-trained multimodal model to obtain the multimodal vector of the query image.

[0052] Thirdly, embodiments of this application provide an electronic device, including:

[0053] Memory, used to store computer programs;

[0054] A processor, when executing a program stored in memory, implements any of the image retrieval methods described above.

[0055] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image retrieval method described in any of the preceding claims.

[0056] Fifthly, embodiments of this application provide a computer program product comprising executable instructions that, when executed on a computer, cause the computer to perform the image retrieval method described in any of the preceding claims.

[0057] Beneficial effects of the embodiments in this application:

[0058] The solution provided in this application utilizes a first language model to analyze the query intent of the query text, obtaining the degree of dependence of the query intent on each preset query dimension. Then, based on the degree of dependence, a target query dimension is determined from the preset query dimensions. Using the query method corresponding to the target query dimension, images matching the query text are retrieved from the images to be queried. Therefore, this solution can select and combine at least one preset query dimension as the target query dimension according to the query intent of the query text. Furthermore, if the degree of dependence of the selected target query dimension is higher than that of other preset query dimensions, then using the query method corresponding to the target query dimension can perform retrieval according to the true query intent of the query text, thereby improving the retrieval accuracy of cross-modal retrieval.

[0059] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0061] Figure 1 A flowchart illustrating an image retrieval method provided in this application embodiment;

[0062] Figure 2 A flowchart illustrating an index construction method provided in this application embodiment;

[0063] Figure 3 A flowchart of a dynamic adaptive method provided in an embodiment of this application;

[0064] Figure 4 A flowchart illustrating a cross-modal sorting method provided in this application embodiment;

[0065] Figure 5 This is a schematic diagram of the structure of an image retrieval device provided in an embodiment of this application;

[0066] Figure 6 A block diagram of an electronic device for implementing the image retrieval method provided in the embodiments of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0068] The image retrieval method provided in this application can be applied to various electronic devices, such as servers and other devices with data processing capabilities. In one practical application scenario, the electronic device is a server. Users can enter queries in the search box of a client device corresponding to a smart home device (e.g., a smart doorbell). When the client detects that the user has triggered a search command, it sends the user's query to the server. The server uses the received query to perform a search and returns the search results to the client, allowing the user to view the results through the client. The client can be installed on the user's smart device (e.g., a mobile phone, computer, tablet, etc.). Furthermore, it is understood that the image retrieval method provided in this application can be implemented through software, hardware, or a combination of both.

[0069] like Figure 1 As shown, the image retrieval method provided in this application embodiment includes steps S101-S104:

[0070] S101, retrieve the query text;

[0071] For example, in practical applications, users can enter query content in the search box of the client corresponding to the smart home system. The electronic device executing the image retrieval method provided in this embodiment obtains the query content entered by the user from the client as the query text. For example, the query text could be "Help me check if my dad is smoking in the living room," or it could be "Find an image of someone at the door," and so on.

[0072] In one implementation, step S101 may include steps A1-A2:

[0073] A1 retrieves the original query statement;

[0074] A2 inputs the original query statement and the second prompt word into the pre-trained second language model to obtain the query text; wherein, the second prompt word is used to instruct the second language model to extract the core words for querying from the original query statement and concatenate them into the query text according to the natural word order.

[0075] In this implementation, the original query statement can be the query content entered by the user in the search box of the client corresponding to the smart home. The electronic device executing the image retrieval method provided in this embodiment obtains the query content entered by the user from the client as the original query statement. For example, the original query statement could be "Check if my dad is smoking in the living room".

[0076] Understandably, when it is necessary to retrieve images using query text, it is usually necessary to first generate descriptive text for the images in order to match the descriptive text of each image with the query text, thereby retrieving images that are semantically similar to the query text.

[0077] After obtaining the original query statement, the natural language expression of the original query statement contains information irrelevant to the query, such as auxiliary words, prepositions, and modal particles. These words cause a semantic shift between the original query statement and the pre-generated image description text, resulting in decreased retrieval accuracy. Therefore, to improve retrieval accuracy, this implementation transforms the original query statement. Specifically, it uses a second language model to process the original query statement, extracting the core retrieval information from the natural language expression, and then generating query text based on the extracted core information.

[0078] Understandably, when the second language model is needed to process the original query statement to generate query text, the original query statement and the second prompt word need to be input into the second language model so that the second language model can understand the task requirements based on the second prompt word and process the original query statement according to the task requirements to accurately output content that meets the task requirements.

[0079] In this implementation, the second prompt word instructs the second language model to extract core words for the query from the original query statement and concatenate them into query text according to natural word order. For example, if the original query statement is "Help me check if Dad is smoking in the living room," then the core words for the query include: Dad, living room, smoking, and the resulting query text is "Dad is smoking in the living room." In this case, it is only necessary to retrieve images whose semantics match "Dad is smoking in the living room" from the images to be queried.

[0080] For example, the second prompt word may be a number of transformation examples pre-built by relevant technical personnel, and / or, to instruct the second language model on the specific processing logic for converting the original query statement into query text.

[0081] For example, the second major language model can be trained as follows: First, construct multiple positive sample pairs, each containing the original query statement and a sample query text that closely reflects the user's actual query intent. Introduce a certain proportion of negative sample pairs to improve generalization performance. Next, qwen3-1.7b (a lightweight language model from the Tongyi Qianwen series) can be used as the base model for the second major language model. Perform SFT (Supervised Fine-Tuning) training on all parameters, providing corresponding prompts during training to ensure the output matches the actual scenario.

[0082] It is understood that the second major language model and the first major language model mentioned below are major language models distinguished only by their names. The second major language model and the first major language model can be the same model or different models. This application does not limit the specific types of the first major language model and the second major language model. For example, the second major language model can be Wenxin Yiyan, ChatGPT (Chat Generative Pre-trained Transformer), etc.

[0083] S102, input the query text and the first prompt word into the pre-trained first language model to obtain the query intent of the query text depends on the degree of dependence of each preset query dimension; wherein, each preset query dimension includes any number of the following: semantic dimension representing text semantics, visual dimension representing image visual features, and structured dimension representing structured information; the structured information includes at least one of the following: the category of the main object in the image, the behavior of the main object in the image, and the scene in the image;

[0084] In this embodiment, the first prompt word is used to instruct the first language model to perform semantic understanding on the input query text in order to analyze the query intent of the query text and output the degree of dependence of the query intent on each preset query dimension.

[0085] For example, each preset query dimension may include a semantic dimension representing text semantics and a visual dimension representing image visual features, or it may include a semantic dimension representing text semantics, a visual dimension representing image visual features, and a structured dimension representing structured information, all of which are reasonable.

[0086] Structured information includes at least one of the following: the category of the subject object in the image (e.g., father, grandfather), the behavior of the subject object in the image (e.g., smoking, drinking), and the scene in the image (e.g., object, bedroom). Accordingly, the query intent of the query text depends on the degree of dependence of the structured dimension, including the degree of dependence of the query intent of the query text on the category of the subject object, the degree of dependence on the behavior of the subject object, and the degree of dependence on the scene.

[0087] For example, if the query text is "Dad is wearing black clothes and smoking," because the query text is limited in its definition of the query object—for example, the subject of the query object is "Dad," the behavior of the subject is "smoking," and the subject is wearing "black clothes"—the query intent of this text strongly depends on visual features (black), the category of the subject (Dad), and the behavior of the subject (smoking), and weakly depends on the context and textual semantics. Therefore, the query intent of this text has a low degree of dependence on the semantic dimension and a high degree of dependence on the visual and structural dimensions.

[0088] For example, if the query text contains little information for the query, it can rely on the semantic dimension for the query. For instance, if the query text is "There's someone at the door," since the query intent only focuses on whether there is someone at the door, and doesn't care whether the person is a father or grandfather, what color clothes they are wearing, or their behavior, then the query can be performed based on the semantic dimension. In other words, the query intent of this text strongly depends on the semantic dimension and weakly depends on the visual dimension.

[0089] For example, if the query text is "Grandpa is watching TV in the living room," because the query text is limited in its target audience—for instance, the subject of the query is "Grandpa," the action of the subject is "watching TV," and the scene is "living room"—the query intent of this text strongly depends on the category of the subject (Grandpa), the subject's action (watching TV), and the scene (living room), and weakly depends on visual features and textual semantics. Therefore, the query intent of this text has a lower dependence on the semantic and visual dimensions and a higher dependence on the structural dimension.

[0090] For example, the first language model can be trained as follows: First, construct multiple sample query texts, where the label of each sample query text is the degree to which the query intent of the sample query text depends on each preset query dimension; then, input the constructed sample query texts into the first language model for training, and calculate the model loss value based on the difference between the model output and the label. Adjust the model parameters by backpropagation that minimizes the model loss value until the model converges.

[0091] S103, determine the target query dimension from each preset query dimension; wherein, the dependency degree corresponding to the target query dimension is higher than the dependency degree corresponding to other preset query dimensions;

[0092] For example, in practical applications, numerical weights can be used to represent the degree of dependency, with higher dependency resulting in higher weights. The target query dimension can be determined by selecting the top few preset query dimensions with the highest corresponding weights, or by selecting preset query dimensions with weights greater than a preset threshold; both are reasonable approaches.

[0093] It is understandable that, since structured information includes at least one of the following: the category of the main object in the image, the behavior of the main object in the image, and the scene in the image, the structured dimension representing structured information may include at least one of the following three dimensions: the object category dimension representing the category of the main object in the image, the object behavior dimension representing the behavior of the main object in the image, and the scene dimension representing the scene in the image.

[0094] For example, if each preset query dimension includes a semantic dimension, a visual dimension, an object category dimension, an object behavior dimension, and a scene dimension, and the weight corresponding to the semantic dimension is 0.3, the weight corresponding to the visual dimension is 0.8, the weight corresponding to the object category dimension is 0.85, the weight corresponding to the object behavior dimension is 0.9, the weight corresponding to the scene dimension is 0.1, and the preset threshold is 0.5, then the visual dimension, the object category dimension, and the object behavior dimension can be determined as the target query dimensions. During the training of the first major language model, the weight range of each preset query dimension can be set to [0,1].

[0095] S104, using the query methods corresponding to the target query dimensions, retrieve images that match the query text from the images to be queried; wherein, the query method corresponding to the semantic dimension is: determine the query images that match the corresponding text vector with the query text; the query method corresponding to the visual dimension is: determine the query images that match the corresponding multimodal vector with the query text; the query method corresponding to the structured dimension is: determine the query images that match the corresponding structured information with the query text.

[0096] It is understood that before executing the image retrieval method provided in the embodiments of this application, it is necessary to construct an index for each image to be queried that is compatible with the image retrieval method provided in the embodiments of this application.

[0097] For example, when performing image retrieval via the semantic dimension, a text vector matching method is used. Therefore, when building the index, the text vector of each query image is used as its index. When performing image retrieval via the visual dimension, a multimodal vector matching method is used. Therefore, when building the index, the multimodal vector of each query image is used as its index. When performing image retrieval via the structured dimension, a structured information matching method is used. Therefore, when building the index, the structured information of each query image is used as its index. Each query image can be a keyframe image from video clips captured by a smart home system.

[0098] In one implementation, the text vector, structured information, and multimodal vector of the image to be queried are constructed through the following steps B1-B2:

[0099] B1: Input each image to be queried into the pre-trained graph-to-text model to obtain the structured information and descriptive text of the image to be queried, and encode the obtained descriptive text into a text vector;

[0100] B2 inputs each query image into a pre-trained multimodal model to obtain the multimodal vector of the query image.

[0101] In this implementation, the image to be queried is input into a pre-trained graph-to-text model, which can obtain the descriptive text of the image to be queried, as well as the category of the main object in the image to be queried, the behavior of the main object, and the scene in the image to be queried.

[0102] For example, a graph-to-text model can be trained as follows: First, acquire multiple sample images, each labeled with a descriptive text, the category of the main object, the behavior of the main object, and the scene. Then, input the multiple sample images into the initial graph-to-text model for training, calculate the model loss value based on the difference between the model output and the labels, and adjust the model parameters by backpropagation to minimize the model loss value until the model converges.

[0103] For example, the initial graph-to-text model can be qwen3-vl-7B (a medium-sized language model in the Tongyi Qianwen series), CLIP (Contrastive Language-Image Pre-training, a multimodal model based on contrastive learning), etc. The specific type of graph-to-text model is not limited in the embodiments of this application.

[0104] After generating descriptive text for the image to be queried using a graph-to-text model, the descriptive text can be input into a text encoder to obtain a text vector. For example, the text encoder can be qwen3-embedding, BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on a transformer architecture), M3E (Moka Massive Mixed Embedding), BGE (BAAI General Embedding), etc. This application does not limit the specific type of text encoder.

[0105] The multimodal model in this implementation can be CLIP or GPT-4o (Generative Pre-trained Transformer 4omni, a versatile generative pre-trained transformer), etc. The specific type of multimodal model is not limited in the embodiments of this application.

[0106] For example, a multimodal model can include two modal encoders: a text encoder and an image encoder. A multimodal model can be trained as follows: First, construct multiple sample pairs, each pair including a sample image and its descriptive text. Next, input the sample image from each pair into the image encoder and the descriptive text into the text encoder. Calculate the contrastive loss between the vectors output by the two encoders based on their similarity, and then adjust the parameters of the modal encoders based on this contrastive loss.

[0107] After training using the above method, the image to be queried is input into the image encoder in the multimodal model to obtain the multimodal vector of the image to be queried.

[0108] In one implementation, step S104, which uses the query method corresponding to the target query dimension to retrieve images matching the query text from the images to be queried, may include steps C1-C2:

[0109] C1. If the target query dimension includes a semantic dimension, then query the text vector that matches the query text from the preset database; if the target query dimension includes a visual dimension, then query the multimodal vector that matches the query text from the preset database; if the target query dimension includes a structured dimension, then query the structured information that matches the query text from the preset database.

[0110] The preset database stores the text vector of the descriptive text of the image to be queried, the multimodal vector of the image to be queried, and the structured information of the image to be queried.

[0111] In this implementation, the pre-built database stores an index for each image to be queried, namely the text vector of the description text of the image to be queried, the multimodal vector of the image to be queried, and the structured information of the image to be queried.

[0112] Understandably, if the target query dimension includes a semantic dimension, the way to query the text vector that matches the query text from the preset database can be: encode the query text into a text vector A, then compare the similarity between each text vector B (i.e., the text vector of the description text of each image to be queried) in the preset database and text vector A, and determine the text vector B with a similarity higher than a preset threshold as the text vector that matches the query text.

[0113] Since multimodal vectors express visual features more accurately, if the target query dimension includes the visual dimension, the multimodal vector matching the query text is retrieved from the preset database. The specific query method can be as follows: input the query text into the text encoder in the pre-trained multimodal model to obtain the multimodal vector A of the query text, and then compare the similarity between each multimodal vector B (i.e., the multimodal vector of each image to be queried) in the preset database and the multimodal vector A. The multimodal vector B with a similarity higher than the preset threshold is determined as the multimodal vector matching the query text.

[0114] If the target query dimension includes a structured dimension, the method for retrieving structured information matching the query text from the preset database can be as follows: extract the structured information A from the query text, and then compare whether there is structured information B containing structured information A in each structured information B (i.e., the structured information of each image to be queried) in the preset database. If it exists, then the structured information B can be determined as the structured information matching the query text. For example, the structured information can be stored in the form of key-value pairs. For instance, the structured information can include {object category: grandfather}, {object behavior: watching TV}, {scene: living room}, etc.

[0115] The preset threshold can be 80% or 90%, etc.

[0116] C2 determines the image that matches the query text based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs.

[0117] In this implementation, all images from the images belonging to the queried text vector, the images belonging to the multimodal vector, and the images belonging to the structured information can be determined as images that match the query text. Alternatively, a preset number of images can be randomly selected from the images belonging to the queried text vector, the images belonging to the multimodal vector, and the images belonging to the structured information as images that match the query text. Both of these are reasonable.

[0118] Understandably, in order to improve the accuracy of the images that match the query text, after retrieving the text vector, multimodal vector, and / or structured information, the retrieved text vector, multimodal vector, and / or structured information can be sorted across modalities to determine a preset number of images that have the highest similarity to the query text, which are then used as the images that match the query text.

[0119] The preset quantity can be 10 or 20, etc.

[0120] In one implementation, step C2, which determines the image matching the query text based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs, may include steps C21-C25:

[0121] C21, if a text vector is found, the descriptive text and structured information of the image to which the text vector belongs are combined to obtain the object to be sorted;

[0122] C22, if structured information is found, the found structured information and the description text of the image to which the structured information belongs are combined to obtain the object to be sorted;

[0123] C23, if a multimodal vector is found, then obtain the image to which the multimodal vector belongs, and get the object to be sorted;

[0124] C24: Input the obtained objects to be sorted and the query text into the reordering model to obtain the similarity between the input objects to be sorted and the query text;

[0125] C25 identifies the images belonging to the preset number of objects to be sorted with the highest similarity as the images that match the query text.

[0126] Understandably, when retrieving text vectors or structured information, the model constructs sortable objects containing both descriptive text and structured information. Specifically, for retrieved text vectors, the reordering model can combine the structured information of the image to which the text vector belongs during sorting; for retrieved structured information, the model can combine the descriptive text of the image to which the structured information belongs during sorting. In this way, the model can integrate more information for sorting, thereby obtaining more accurate sorting results.

[0127] In this implementation, the rearrangement model is a ranking model capable of cross-modal ranking, that is, it can transform images and text into a unified space for relevance scoring.

[0128] For example, the reranking model can employ jina-reranker-m0 (a multimodal, multilingual reranker). jina-reranker-m0 reuses the visual encoder and projector of the pre-trained Qwen2-VL (a multimodal visual understanding model), fine-tunes the LLM part using LoRA (low-rank adaptation), and subsequently trains an MLP (Multilayer Perceptron) to generate ranking logits that measure query-document relevance. This architecture can seamlessly combine visual and textual inputs, enabling the language model to correctly process visual information and perform complex multimodal inference combining visual and textual elements. Positive sample pairs constructed during training can be text-related text, text-related image, image-related text, or image-related image. This application does not limit the specific type of reranking model.

[0129] After obtaining the sorted objects, the sorted objects and query text are input into the reordering model. This allows the model to transform both the sorted objects and the query text into the same vector space and calculate the similarity between each sorted object and the query text. Then, the images containing the 10 most similar objects are identified as matching the query text. This preset number can be 10 or 20, etc. For example, if there are 100 objects to be sorted, the images containing the 10 most similar objects output by the reordering model can be identified as matching the query text.

[0130] In one implementation, prior to step C1, the image retrieval method may further include:

[0131] If the target query dimension includes a structured dimension, and no structured information matching the query text is found in the preset database, then the text vector matching the query text will be retrieved from the preset database.

[0132] Understandably, in practical applications, structured information is stored in a pre-defined database in the form of key-value pairs. Retrieval using this structured information employs key-value pair matching. If the values ​​in a key-value pair are synonyms and no match is found, then semantically identical synonyms will not be retrieved. For example, if the query text is "smoking," and the pre-defined database stores "smoking" but not "smoking," then a query based on the structured dimension will not find the term in the database, even though "smoking" and "smoking" are semantically identical. The inability to find the term will result in low image retrieval accuracy.

[0133] Therefore, to improve image retrieval accuracy, if the target query dimension includes a structured dimension, and no structured information matching the query text can be found in the preset database, then the text vector matching the query text is retrieved from the preset database. This allows for semantic matching to continue searching the preset database, thus avoiding situations where the query text and structured information are synonyms but cannot be retrieved.

[0134] To better understand the image retrieval method provided in the embodiments of this application, a specific example of the image retrieval method provided in the embodiments of this application is described below.

[0135] The process of image retrieval implemented in this example includes the following (1)-(5):

[0136] (1) Build an index;

[0137] like Figure 2 As shown, structured multimodal descriptive information is generated based on keyframe images, and an index is constructed. The index includes text vectors, structured entities (corresponding to the structured information mentioned above), and multimodal vectors. After the index is constructed, it is stored in a database (corresponding to the preset database mentioned above). The index construction process includes: inputting the keyframe image into a graph-to-text model to obtain the descriptive text of the keyframe image, the main objects, behaviors, and scenes in the image; inputting the keyframe image into a multimodal model to obtain multimodal vectors; then, encoding the descriptive text to obtain text vectors; and constructing the main objects, behaviors, and scenes into key-value pairs of structured entities.

[0138] (2) Query rewriting;

[0139] Because there is a semantic offset between the query content expressed in natural language and the descriptive text generated by the graph-based text model, it is necessary to extract the core information of the retrieval from the natural language expression. For example, the target query (corresponding to the original query statement above): "Help me check if Dad is smoking in the living room", and the transformed query (corresponding to the query text above): "Dad is smoking in the living room".

[0140] The specific rewrite steps include: First, using an LLM (Large Language Model) to construct target queries and transformation queries that closely resemble the user's actual thoughts, serving as positive sample pairs, and introducing a certain proportion of negative sample pairs to improve generalization performance. Next, using qwen3-1.7b as the base model for the LLM, the constructed samples are used to perform full-parameter SFT training on the model, providing corresponding prompts during training to make the output results as close to the real-world scenario as possible. The trained model can then be directly used for transformation query inference.

[0141] (3) Dynamic adaptive recall loop;

[0142] Since each word in the query statement receives different retrieval attention, a query intent analysis is performed on the user query (corresponding to the query text mentioned above) before recall. This example designs an adaptive router (corresponding to the first large language model mentioned above) based on the powerful understanding capabilities of the large language model, outputting a recall path weight vector. Each element in this weight vector represents the degree to which the query statement depends on a recall strategy. Based on the output weight vector, multiple recall strategies are dynamically selected and weighted to obtain an initial candidate set. The recall strategies include one or more of the following: text vector recall (corresponding to the query methods corresponding to the semantic dimension mentioned above), multimodal recall (corresponding to the query methods corresponding to the visual dimension mentioned above), and structured attribute recall (corresponding to the query methods corresponding to the structured dimension mentioned above).

[0143] A concrete example of an adaptive router selecting and executing a recall strategy based on a user query is as follows: Figure 3 As shown. First, the user query is input into the adaptive router. If the user query is: "Dad is wearing black clothes and smoking," the adaptive router analyzes the query intent and finds that the query strongly depends on visual (black), object (dad), behavior (smoking), and weakly depends on scene and semantics. The weight vector output by the adaptive router is (0.85, 0.9, 0.1, 0.8, 0.3). Here, 0.85 is the weight corresponding to the object, 0.9 is the weight corresponding to the behavior, 0.1 is the weight corresponding to the scene, 0.8 is the weight corresponding to visual, and 0.3 is the weight corresponding to semantics.

[0144] Therefore, since objects, behaviors, and visual attributes have relatively high weights, the system will prioritize multimodal recall and structured attribute recall. Multimodal recall involves encoding the user query into a multimodal vector for multimodal vector retrieval. Structured attribute recall involves performing structured attribute retrieval on the entities "dad" and "smoking." To avoid situations where synonyms cannot be found, text vector recall can be used as a fallback strategy.

[0145] (4) Cross-modal sorting;

[0146] The dynamic adaptive recall loop retrieves data from different modalities, employing a cross-modal re-ranking model to place text and images in a unified space for relevance scoring. During ranking, for the recall results of text vectors and structured entities, the descriptive text and structured entities are input together into the re-ranking model for deep fusion ranking. For example, ... Figure 4 As shown, after employing structured attribute recall, multimodal recall, and text vector recall, if the candidate set obtained includes multimodal vector 1, text vector 2, structured entity 3, and multimodal vector 4, the input to the re-ranking model can be constructed as follows: [keyframe image 1, descriptive text 2 + structured entity 2, descriptive text 3 + structured entity 3, keyframe image 4]. In this way, the re-ranking model can see additional information when ranking the retrieved text vectors and structured entities, making its ranking decision more accurate. The re-ranking model calculates the similarity between the user query and each input object, obtaining scores 1, 2, 3, and 4. The input object with the higher score is selected from the four scores; the image to which this input object belongs is the image that matches the user query.

[0147] (5) Self-learning mechanism;

[0148] Model optimization based on false positive feedback. When a user finds an incorrect search result, they can submit feedback, and the system will automatically record this failed query-search result pair. This recorded query-search result pair will then be used as negative samples for online fine-tuning or incremental learning of the query rewriting module, adaptive router, and re-ranking model.

[0149] As can be seen, the image retrieval method provided in this example introduces a dynamic adaptive routing strategy, which can identify user intent and select the query dimension on which the user intent depends for querying, avoiding blind recall, making retrieval more efficient and improving the quality of the candidate set. The introduction of a cross-modal re-ranking model, which integrates structured entities and descriptive text, makes the ranking more accurate, thereby effectively improving the recall and precision of the retrieval.

[0150] In the technical solution of this application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing the query text are all carried out with the user's authorization.

[0151] Corresponding to the above method embodiments, this application also provides an image retrieval device, such as... Figure 5 As shown, the device includes:

[0152] Module 510 is used to retrieve query text;

[0153] The model processing module 520 is used to input the query text and the first prompt word into a pre-trained first large language model to obtain the degree of dependence of the query intent of the query text on each preset query dimension; wherein, each preset query dimension includes any multiple of the following: semantic dimension representing text semantics, visual dimension representing image visual features, and structured dimension representing structured information; the structured information includes at least one of the following: the category of the main object in the image, the behavior of the main object in the image, and the scene in the image;

[0154] The determination module 530 is used to determine the target query dimension from each preset query dimension; wherein the dependency degree corresponding to the target query dimension is higher than the dependency degree corresponding to other preset query dimensions.

[0155] The retrieval module 540 is used to retrieve images matching the query text from the images to be queried using the query methods corresponding to the target query dimensions; wherein, the query method corresponding to the semantic dimension is: determining the query image whose corresponding text vector matches the query text; the query method corresponding to the visual dimension is: determining the query image whose corresponding multimodal vector matches the query text; and the query method corresponding to the structured dimension is: determining the query image whose corresponding structured information matches the query text.

[0156] Optionally, the retrieval module 540 includes:

[0157] The query submodule is used to: if the target query dimension includes a semantic dimension, query a text vector that matches the query text from a preset database; if the target query dimension includes a visual dimension, query a multimodal vector that matches the query text from a preset database; if the target query dimension includes a structured dimension, query structured information that matches the query text from a preset database.

[0158] The preset database stores the text vector of the descriptive text of the image to be queried, the multimodal vector of the image to be queried, and the structured information of the image to be queried.

[0159] The determination submodule is used to determine the image that matches the query text based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs.

[0160] Optionally, the determining submodule includes:

[0161] The first combination unit is used to combine the descriptive text and structured information of the image to which the text vector belongs, if a text vector is found, to obtain the object to be sorted.

[0162] The second combination unit is used to combine the queried structured information and the description text of the image to which the structured information belongs if structured information is found, to obtain the object to be sorted.

[0163] The acquisition unit is used to acquire the image to which the multimodal vector belongs if a multimodal vector is found, and obtain the object to be sorted.

[0164] The sorting unit is used to input the obtained objects to be sorted and the query text into the rearrangement model to obtain the similarity between the input objects to be sorted and the query text.

[0165] The output unit is used to determine the images of the preset number of objects to be sorted with the highest similarity as the images that match the query text.

[0166] Optionally, the retrieval module 540 further includes:

[0167] The text matching submodule is used to perform the following action before the determining submodule performs the determination of the image that matches the query text based on the image to which the queried text vector belongs, the image to which the multimodal vector belongs, and the image to which the structured information belongs: if the target query dimension includes a structured dimension and no structured information matching the query text is found in the preset database, then the text vector that matches the query text is retrieved from the preset database.

[0168] Optionally, the acquisition module 510 includes:

[0169] The `get` submodule is used to retrieve the original query statement;

[0170] The conversion submodule is used to input the original query statement and the second prompt word into the pre-trained second language model to obtain the query text; wherein, the second prompt word is used to instruct the second language model to extract the core words for querying from the original query statement and concatenate them into the query text according to the natural word order.

[0171] Optionally, the text vector, structured information, and multimodal vector of the image to be queried are constructed through the following modules:

[0172] The first output module is used to input each query image into the pre-trained graph-to-text model to obtain the structured information and descriptive text of the query image, and encode the obtained descriptive text into a text vector.

[0173] The second output module is used to input each query image into a pre-trained multimodal model to obtain the multimodal vector of the query image.

[0174] This application also provides an electronic device, such as... Figure 6 As shown, it includes:

[0175] Memory 601 is used to store computer programs;

[0176] The processor 602, when executing the program stored in the memory 601, implements the steps of any of the above image retrieval methods;

[0177] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 602, the communication interface, and the memory 601 communicating with each other via the communication bus.

[0178] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0179] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0180] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0181] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0182] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image retrieval methods.

[0183] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image retrieval methods described above.

[0184] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0185] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0186] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0187] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image retrieval method characterized by, The method comprises: acquiring a query text; inputting the query text and a first prompt word into a first large language model pre-trained to obtain a dependency degree of a query intent of the query text on each preset query dimension; wherein each preset query dimension comprises any multiple of the following: a semantic dimension representing text semantics, a visual dimension representing image visual features, and a structured dimension representing structured information; the structured information comprises at least one of the following: a category of a subject object in an image, a behavior of a subject object in an image, and a scene in an image; determining a target query dimension from each preset query dimension; wherein the target query dimension corresponds to a higher dependency degree than other preset query dimensions; using a query mode corresponding to the target query dimension to retrieve an image matching the query text from a to-be-queried image; wherein the query mode corresponding to the semantic dimension is to determine a to-be-queried image whose corresponding text vector matches the query text; the query mode corresponding to the visual dimension is to determine a to-be-queried image whose corresponding multi-modal vector matches the query text; and the query mode corresponding to the structured dimension is to determine a to-be-queried image whose corresponding structured information matches the query text.

2. The method of claim 1, wherein, using a query mode corresponding to the target query dimension to retrieve an image matching the query text from a to-be-queried image, comprising: if the target query dimension comprises a semantic dimension, querying a text vector matching the query text from a preset database; if the target query dimension comprises a visual dimension, querying a multi-modal vector matching the query text from a preset database; and if the target query dimension comprises a structured dimension, querying structured information matching the query text from a preset database; wherein the preset database stores text vectors of description texts of to-be-queried images, multi-modal vectors of to-be-queried images, and structured information of to-be-queried images; determining an image matching the query text based on an image to which the queried text vector belongs, an image to which the queried multi-modal vector belongs, and an image to which the queried structured information belongs.

3. The method of claim 2, wherein, determining an image matching the query text based on an image to which the queried text vector belongs, an image to which the queried multi-modal vector belongs, and an image to which the queried structured information belongs, comprising: if a text vector is queried, combining a description text of an image to which the text vector belongs and structured information to obtain a to-be-ordered object; if structured information is queried, combining the queried structured information and a description text of an image to which the structured information belongs to obtain a to-be-ordered object; if a multi-modal vector is queried, obtaining an image to which the multi-modal vector belongs to obtain a to-be-ordered object; inputting each obtained to-be-ordered object and the query text into a rearrangement model to obtain a similarity between the input to-be-ordered object and the query text; determining images to which a preset number of to-be-ordered objects with the highest similarity belong as images matching the query text.

4. The method of claim 2, wherein, Before determining the image matching the query text based on the image to which the text vector belongs, the image to which the multi-modal vector belongs, and the image to which the structured information belongs, the method further comprises: If the target query dimension includes a structured dimension, and no structured information matching the query text is found in the preset database, a text vector matching the query text is queried from the preset database.

5. The method of claim 1, wherein, The query text comprises: An original query sentence is obtained; The original query sentence and a second prompt word are input into a second large language model pre-trained to obtain a query text; wherein the second prompt word is used to instruct the second large language model to extract core vocabulary for querying from the original query sentence, and to splice the core vocabulary into a query text in a natural sequence.

6. The method according to any one of claims 1 to 5, characterized in that, The text vector, structured information, and multi-modal vector of the image to be queried are constructed by: Each image to be queried is input into a pre-trained image-to-text model to obtain structured information and description text of the image to be queried, and the obtained description text is encoded into a text vector; Each image to be queried is input into a pre-trained multi-modal model to obtain a multi-modal vector of the image to be queried.

7. An image retrieval apparatus characterized by comprising: The device comprises: An acquisition module for acquiring a query text; A model processing module for inputting the query text and a first prompt word into a first large language model pre-trained to obtain a dependency degree of a query intent of the query text on each preset query dimension; wherein each preset query dimension includes any multiple of the following: a semantic dimension representing text semantics, a visual dimension representing image visual features, and a structured dimension representing structured information; the structured information includes at least one of the following: a category of a subject object in an image, a behavior of a subject object in an image, and a scene in an image; A determination module for determining a target query dimension from each preset query dimension; wherein the dependency degree corresponding to the target query dimension is higher than the dependency degrees corresponding to other preset query dimensions; A retrieval module for retrieving an image matching the query text from images to be queried using a query method corresponding to the target query dimension; wherein the query method corresponding to the semantic dimension is to determine an image to be queried whose corresponding text vector matches the query text; the query method corresponding to the visual dimension is to determine an image to be queried whose corresponding multi-modal vector matches the query text; and the query method corresponding to the structured dimension is to determine an image to be queried whose corresponding structured information matches the query text.

8. An electronic device, comprising: Comprise: A memory for storing a computer program; A processor for executing the program stored in the memory to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the method of any one of claims 1-6.

10. A computer program product, characterised in that, The computer program product contains executable instructions, which, when executed on a computer, cause the computer to perform the method of any one of claims 1-6.

Images