Image retrieval method, device and storage medium

Abstract

The present application discloses an image retrieval method, device, and storage medium. The image retrieval method includes: performing multi-dimensional feature extraction on each base image to obtain a multi-dimensional feature vector database; performing multi-dimensional feature extraction on retrieval information to obtain multi-dimensional retrieval information vectors; calculating the matching degree between the retrieval information vectors of the multiple dimensions and the feature vectors in the feature vector database of the corresponding dimensions, screening out feature vectors from the feature vector database of the multiple dimensions whose matching degree meets preset conditions, and obtaining matching feature vectors corresponding to each dimension; and determining the base image associated with the matching feature vector corresponding to each dimension to obtain an image retrieval result. Multi-dimensional feature extraction can be performed on the retrieval information to improve the expressive power of the retrieval information vector, and the final image retrieval result can be obtained by integrating the multi-dimensional matching results, thereby improving the accuracy of the image retrieval result.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image retrieval method, device and storage medium. Background Art

[0002] There are two main methods for existing image retrieval. One is to perform natural language processing analysis on the input text, calculate the matching degree between the input text and the text description corresponding to the image, and determine the image that matches the input text. This method relies on the text description information of the image itself, and the text description content must be as complete as possible or conform to the user's usage scenario to achieve better results. The other is to convert both text and images into the same vector space through a multimodal embedding model, and perform image matching retrieval by calculating vector similarity. This method strongly relies on the original capabilities of the multimodal embedding model. Due to the limited amount of sample data for training the multimodal embedding model, the final accuracy of the multimodal embedding model is also relatively limited. If there is irrelevant noise in the text or image content, it will greatly affect the final retrieval effect. The same problem also exists in image search.

[0003] Therefore, how to improve the accuracy of image retrieval is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] In order to solve the above technical problems, the present application at least provides an image retrieval method, device and storage medium.

[0005] In a first aspect, the present application provides an image retrieval method, comprising: performing feature extraction in multiple dimensions on each base image to obtain a feature vector database of multiple dimensions corresponding to each base image; receiving retrieval information, performing feature extraction in multiple dimensions on the retrieval information, and obtaining retrieval information vectors of multiple dimensions corresponding to the retrieval information; calculating the matching degrees between the retrieval information vectors of multiple dimensions and the feature vectors in the feature vector database of the corresponding dimensions, and filtering out the feature vectors whose matching degrees meet preset conditions from the feature vector database of the multiple dimensions to obtain matching feature vectors corresponding to each dimension; determining the base images associated with the matching feature vectors corresponding to each dimension to obtain image retrieval results.

[0006] In one embodiment, the features of multiple dimensions include at least two of the following: description text of the image, attribute information of target objects contained in the image, and association relationships between target objects in the image.

[0007] In one embodiment, the retrieval information includes retrieval text and / or a reference image; multiple-dimensional feature extraction is performed on the retrieval information to obtain multiple-dimensional retrieval information vectors corresponding to the retrieval information, including: performing intent recognition on the retrieval text to obtain the true intent, optimizing the retrieval text based on the true intent to obtain the true intent description text; extracting at least two of the image description text, attribute information of the target object, and the association relationship between each target object from the true intent description text to obtain multiple-dimensional information extraction results; and / or extracting the corresponding image description text, attribute information of the target object, and at least two of the association relationship between each target object from the reference image to obtain multiple-dimensional information extraction results; encoding the multiple-dimensional information extraction results to obtain multiple-dimensional retrieval information vectors corresponding to the retrieval information.

[0008] In one embodiment, the base image associated with the matching feature vector corresponding to each dimension is determined to obtain an image retrieval result, including: determining the base image associated with the matching feature vector corresponding to each dimension to obtain a candidate image; calculating the image score of the candidate image for each dimension; combining the image score of the candidate image for each dimension to calculate the total score corresponding to the candidate image; selecting the candidate image whose total score meets the preset conditions as the image retrieval result corresponding to the image retrieval instruction.

[0009] In one embodiment, calculating the image score of the candidate image for each dimension includes:

[0010] Obtain the matching degree corresponding to the matching feature vector associated with the candidate image;

[0011] All candidate images corresponding to each dimension are sorted in order according to the degree of matching to obtain the sorting results corresponding to each dimension; based on the ranking of the candidate images in each sorting result, the candidate images are scored to obtain the sorting scores of the candidate images for each dimension; and / or, based on the number of occurrences of the same candidate image in each dimension, the candidate images are scored to obtain the number scores of the candidate images for each dimension; based on the sorting scores and / or number scores of the candidate images for each dimension, the image scores of the candidate images are calculated.

[0012] In one embodiment, the image scores of the candidate images for each dimension are combined to calculate the total score corresponding to the candidate image, including: obtaining the weight parameters corresponding to each dimension; and performing weighted sum calculation on the image scores of the candidate images for each dimension based on the weight parameters corresponding to each dimension to obtain the total score corresponding to the candidate image.

[0013] In one embodiment, obtaining the weight parameters corresponding to each dimension includes: parsing the search information, and determining the search attention corresponding to each dimension based on the parsing results; and setting the weight parameters corresponding to each dimension based on the search attention corresponding to each dimension.

[0014] In one embodiment, the retrieval information contains different types of retrieval data; obtaining the weight parameters corresponding to each dimension includes: determining the data quality of different types of retrieval data in the retrieval information respectively; setting the weight parameters corresponding to the candidate images in each dimension based on the data quality of different types of retrieval data; wherein the data quality of any type of retrieval data is positively correlated with the weight parameters corresponding to the candidate images obtained by filtering the retrieval data of any type through matching calculation.

[0015] The second aspect of the present application provides an image retrieval device, which includes: a base database feature extraction module, which is used to perform feature extraction of multiple dimensions on each base database image, and obtain a feature vector database of multiple dimensions corresponding to each base database image; a retrieval feature extraction module, which is used to receive retrieval information, perform feature extraction of multiple dimensions on the retrieval information, and obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information; a matching vector selection module, which is used to calculate the matching degree between the retrieval information vectors of multiple dimensions and the feature vectors in the feature vector database of the corresponding dimensions, and respectively filter out feature vectors whose matching degrees meet preset conditions from the feature vector database of multiple dimensions to obtain matching feature vectors corresponding to each dimension; and a result generation module, which is used to determine the base database images associated with the matching feature vectors corresponding to each dimension, and obtain image retrieval results.

[0016] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-mentioned image retrieval method.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned image retrieval method when the program instructions are executed by a processor.

[0018] The above scheme obtains multi-dimensional retrieval information vectors corresponding to the retrieval information by performing multi-dimensional feature extraction on the retrieval information. It can perform multi-dimensional feature extraction on the input retrieval information to improve the expression ability of the retrieval information vector. Then, the multi-dimensional retrieval information vectors are matched with the multi-dimensional feature vector database respectively, and the feature vectors whose matching degree meets the preset conditions are respectively screened out from the multi-dimensional feature vector database to obtain the matching feature vectors corresponding to each dimension. The underlying library images associated with the matching feature vectors corresponding to each dimension are determined to obtain the image retrieval results, thereby improving the accuracy of the image retrieval results.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0021] Figure 1 is a schematic diagram of a solution implementation environment shown in an exemplary embodiment of the present application;

[0022] Figure 2 is a flowchart of an image retrieval method shown in an exemplary embodiment of the present application;

[0023] Figure 3 is a flowchart illustrating feature extraction of search information according to an exemplary embodiment of the present application;

[0024] Figure 4 is a block diagram of an image retrieval device shown in an exemplary embodiment of the present application;

[0025] Figure 5 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0028] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0029] The term "and / or" in this article is merely information describing the association of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] The image retrieval method provided in the embodiments of the present application is described below.

[0031] Please refer to Figure 1 , Figure 1 FIG1 is a schematic diagram of an exemplary embodiment of the present application showing a solution implementation environment, wherein the solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are in communication connection with each other.

[0032] The number of the terminal 110 may be one or more. The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0033] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0034] In one example, the server 120 contains an image database, which is used to store background images. The terminal 110 sends an image retrieval instruction carrying retrieval information to the server 120. The server 120 obtains the successfully matched background images from the image database based on the retrieval information, obtains the image retrieval results, stores the image retrieval results locally, transmits them back to the terminal 110, or transmits them to other terminals.

[0035] In one example, terminal 110 includes an image database for storing database images. A client running a target application, such as an application providing image retrieval functionality, is installed on terminal 110. The target application is used to query the database for database images. Server 120 may be a backend server for the target application, providing backend services for the client.

[0036] In the image retrieval method provided in the embodiment of the present application, the execution entity of each step can be the terminal 110, such as the client of the target application installed and running in the terminal 110, or the server 120, or the terminal 110 and the server 120 can interact and cooperate to execute, that is, part of the steps of the method are executed by the terminal 110 and the other steps are executed by the server 120.

[0037] See also Figure 2 , Figure 2 This is a flowchart of an image retrieval method shown in an exemplary embodiment of the present application. The image retrieval method can be applied to Figure 1It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0038] like Figure 2 As shown, the image retrieval method includes at least steps S210 to S240, which are described in detail as follows:

[0039] Step S210: performing feature extraction of multiple dimensions on each base image to obtain a feature vector database of multiple dimensions corresponding to each base image.

[0040] The base image refers to an image that can be retrieved. Multi-dimensional feature extraction is performed on the base image to obtain multi-dimensional feature vectors corresponding to the base image. The multi-dimensional feature vectors corresponding to each base image are combined to obtain a multi-dimensional feature vector database.

[0041] In some embodiments, the features of multiple dimensions include at least two of the descriptive text of the image, the attribute information of the target objects contained in the image, and the association relationship between each target object in the image. The descriptive text corresponding to the base image can be generated, and / or the attribute information of the target objects contained in the base image can be extracted, and / or the association relationship between each target object in the base image can be extracted to obtain the feature vectors of multiple dimensions corresponding to the base image.

[0042] Exemplarily, the steps of generating description text corresponding to the base library image include: obtaining a prompt word template matching the base library image; inputting the base library image and the prompt word template into a pre-trained multimodal vision model to obtain the description text of the base library image output by the multimodal vision model; inputting the description text of the base library image into a pre-trained text encoding model to obtain the description text vector of the base library image output by the text encoding model; and combining the description text vector of each base library image to construct an overall image vector database corresponding to each base library image.

[0043] The prompt word template is used in the next step to be combined with the base image that currently needs to be feature extracted and sent into the multimodal vision model for reasoning, so that the multimodal vision model can focus on the specific information of the base image.

[0044] For example, the scene type to which the image retrieval application scene belongs is obtained, prompt word templates corresponding to different scene types are pre-stored, prompt word templates matching the scene type are searched, and the query results are used as prompt word templates matching the base library images.

[0045] For example, the scene type of the image retrieval application scenario indicates that an image environment information query is required, and the prompt word template obtained by the query may be: "Briefly describe the content of this picture, especially the scene, environment, and weather information."

[0046] Then, the base image and the prompt word template are input into the pre-trained multimodal vision model together to obtain the description text of the base image output by the multimodal vision model.

[0047] Among them, the multimodal vision model can process multimodal data such as images and texts. It can be composed of a convolutional neural network (CNN), and / or a recurrent neural network (RNN), and / or a transformer network, and / or a vision transformer (ViT) network, etc. This application does not limit the specific implementation method of the multimodal vision model.

[0048] After obtaining the descriptive text of the underlying image using the multimodal vision model, the descriptive text of the underlying image is input into a pre-trained text encoding model. This outputs the descriptive text vector of the underlying image. Combining the descriptive text vectors of each underlying image, a database of overall image vectors corresponding to each underlying image is constructed. The text encoding model is used to encode the input text into a vector, and the overall image vector in the overall image vector database represents the descriptive text vector of each underlying image.

[0049] Optionally, the base image can be encoded to obtain an image vector, and the description text vector and image vector corresponding to the base image can be associated and stored to obtain an image overall vector database. The image overall vector database includes the description text vector and image vector for each base image.

[0050] Among them, a multimodal coding model can be used for image encoding. A multimodal coding model refers to a model that can process data of different modalities (such as images or text) and convert these data into feature representations that the model can understand. Of course, a conventional image coding model can also be used for image encoding, and this application does not limit this.

[0051] Exemplarily, the step of extracting attribute information of target objects contained in the base image includes: inputting the base image into a pre-trained target detection model to obtain the position information of each target object contained in the base image output by the target detection model; based on the position information of each target object contained in the base image, extracting the image region containing the target object from the base image to obtain an object subgraph; inputting the object subgraph into a pre-trained attribute recognition model to obtain the attribute information of the target object output by the attribute recognition model; converting the attribute information into a text description to obtain object attribute description text; inputting the object attribute description text into a text encoding model to obtain a description text vector of the target object output by the text encoding model; and combining the description text vectors of the target objects contained in each base image to construct an object vector database corresponding to each base image. The object vector in the object vector database is the description text vector of the target object.

[0052] The target detection model is used to perform target detection on the input image and obtain the location information of the target objects (such as vehicles, buildings, etc.) contained in the image. The target detection box can be used to identify the location information of the target object.

[0053] The attribute recognition model is used to extract attribute information from the input object subgraph, such as extracting the type, and / or posture, and / or color, and / or size, and / or shape of the target object to obtain the attribute information of the target object.

[0054] Using the target detection model and attribute recognition model, the descriptive text vectors of the target objects in the base image are extracted. According to the descriptive text vectors of the target objects contained in each base image, an object vector database corresponding to each base image is constructed.

[0055] Optionally, the object subgraph of the target object may be image-encoded to obtain an object subgraph vector, and the description text vector corresponding to the target object and the object subgraph vector may be associated and stored to obtain an object vector database. The object vector in the object vector database includes the description text vector and the object subgraph vector of the target object.

[0056] Exemplarily, the steps of extracting the association relationships between target objects in a base image include: inputting the base image into a pre-trained target detection model to obtain the location information of each target object contained in the base image output by the target detection model; transmitting the base image and the location information of each target object to an association determination module, which outputs the association relationship between the target objects in the base image; converting the association relationship into a text description to obtain an object relationship description text corresponding to the base image; inputting the object relationship description text corresponding to the base image into a text encoding model to obtain an object relationship vector for the base image output by the text encoding model; and combining the object relationship vectors corresponding to each base image to construct a relationship vector database corresponding to each base image. The relationship vector in the relationship vector database is the object relationship vector corresponding to each base image.

[0057] Association relations are used to describe the connection information between target objects in an image.

[0058] For example, based on the location information of each target object, the distance between each target object is calculated, and the association relationship between each target object is obtained based on the size of the distance between each target object. For example, if the distance between each target object is greater than a preset distance threshold, the association relationship between each target object is irrelevant; if the distance between each target object is not greater than the preset distance threshold, the association relationship between each target object is relevant.

[0059] Then, the association relationship is converted into a text description to obtain the object relationship description text corresponding to the base image, and then the object relationship description text corresponding to the base image is input into the text encoding model to obtain the object relationship vector of the base image output by the text encoding model. Combined with the object relationship vector corresponding to each base image, a relationship vector database corresponding to each base image is constructed.

[0060] Of course, in addition to the dimensions exemplified above, features of other dimensions may also be selected, such as image brightness information, and / or image quality information, and / or image acquisition location, etc. This application does not limit this.

[0061] Step S220: receiving the search information, performing feature extraction of multiple dimensions on the search information, and obtaining search information vectors of multiple dimensions corresponding to the search information.

[0062] The retrieval information refers to data describing the image to be retrieved. The retrieval information may be text data, and / or image data, and / or audio data, etc. This application does not limit this.

[0063] Perform feature extraction on the retrieval information in multiple dimensions to obtain retrieval information vectors in multiple dimensions corresponding to the retrieval information.

[0064] For example, the descriptive text corresponding to the image to be retrieved can be extracted from the retrieval information, and / or the attribute information of the target object contained in the image to be retrieved can be extracted from the retrieval information, and / or the association relationship between the target objects in the image to be retrieved can be extracted from the retrieval information to obtain a retrieval information vector of multiple dimensions corresponding to the retrieval information.

[0065] The dimensions of the features to be extracted from the search information are determined according to the dimensions of the feature vector database corresponding to the base library image.

[0066] Step S230: Calculate the matching degree between the retrieval information vectors of multiple dimensions and the feature vectors in the feature vector database of the corresponding dimensions, and select the feature vectors whose matching degree meets the preset conditions from the feature vector database of multiple dimensions to obtain the matching feature vector corresponding to each dimension.

[0067] After obtaining the retrieval information vectors of multiple dimensions corresponding to the retrieval information, the matching degree between the retrieval information vector of each dimension and the feature vector in the feature vector database of the corresponding dimension is calculated.

[0068] For example, based on step S210, an image overall vector database, and / or an object vector database and / or a relationship vector database are constructed, and the description text corresponding to the image to be retrieved is extracted from the retrieval information to obtain the image overall retrieval vector, and the matching degree of the image overall retrieval vector is calculated with each image overall vector in the image overall vector database; and / or, the attribute information of the target object contained in the image to be retrieved is extracted from the retrieval information to obtain the object description retrieval vector, and the matching degree of the object description retrieval vector is calculated with each object vector in the object vector database; and / or, the association relationship between each target object in the image to be retrieved is extracted from the retrieval information to obtain the relationship retrieval vector, and the matching degree of the relationship retrieval vector is calculated with each relationship vector in the relationship vector database.

[0069] Among them, the Euclidean distance, Manhattan distance, Chebyshev distance, etc. between vectors can be calculated to obtain the matching degree between vectors.

[0070] Then, feature vectors whose matching degree meets preset conditions are screened out from the feature vector database of multiple dimensions to obtain matching feature vectors corresponding to each dimension.

[0071] Among them, the feature vector with a matching degree greater than a preset matching degree threshold can be regarded as the feature vector whose matching degree meets the preset conditions; the feature vector with a matching degree within the preset threshold range can also be regarded as the feature vector whose matching degree meets the preset conditions; or the feature vectors can be sorted in descending order according to the size of the matching degree, and a preset number of feature vectors ranked at the top can be regarded as the feature vectors whose matching degree meets the preset conditions. This application does not limit this.

[0072] Step S240: Determine the base library images associated with the matching feature vectors corresponding to each dimension, and obtain image retrieval results.

[0073] The underlying image associated with each matching feature vector is determined, and the image retrieval result is obtained based on the underlying image associated with each matching feature vector.

[0074] For example, the base image associated with each matching feature vector can be directly used as the image retrieval result; or the base image associated with each matching feature vector can be scored, and the base images with the highest score or a preset number of base images with the top scores can be selected as the image retrieval result. This application does not limit this.

[0075] Next, some embodiments of the present application are described.

[0076] In some embodiments, the features of multiple dimensions include at least two of the following: description text of the image, attribute information of the target objects contained in the image, and association relationships between target objects in the image.

[0077] In some embodiments, the search information includes search text, and step S220 performs feature extraction of multiple dimensions on the search information to obtain search information vectors of multiple dimensions corresponding to the search information, including the following steps S2211 to S2213.

[0078] Step S2211: perform intent recognition on the search text to obtain the true intent, and optimize the search text based on the true intent to obtain the true intent description text.

[0079] The true intent of the retrieved text is extracted, and the retrieved text is optimized based on the true intent to obtain the true intent description text.

[0080] For example, a pre-trained intent recognition model is used to perform intent recognition on the retrieved text and generate a true intent description text, thereby obtaining the true intent description text output by the intent recognition model. The intent recognition model can be implemented based on a neural network model architecture such as a Large Language Model (LLM), CNN, or RNN.

[0081] For example, the search text is filled into the prompt template corresponding to the intent recognition model to obtain the input text corresponding to the intent recognition model, which is then input into the intent recognition model for intent recognition to obtain the true intent description text output by the intent recognition model.

[0082] For example, the prompt template corresponding to the intent recognition model can be:

[0083] {"You are an expert in intent recognition and analysis in a specific scenario. Your responsibility is to analyze the search text entered by the user to determine the user's true intent."

[0084] "The search text entered by the user is": ...

[0085] Analyze the user's true intent and perform more comprehensive and accurate supplementary optimization of the user's search text. Output in the following format and avoid irrelevant associations:

[0086] The user's actual intention description text is: ......}.

[0087] By identifying the true intent of the search text and optimizing and rewriting it according to the true intent, a more standardized and accurate true intent description text is obtained.

[0088] Step S2212: extract at least two of the following from the true intention description text: image description text, attribute information of the target object, and association relationship between target objects, and obtain information extraction results in multiple dimensions.

[0089] Perform multi-dimensional key information extraction on the true intention description text to obtain multi-dimensional information extraction results.

[0090] For example, a pre-trained information extraction model is used to extract multi-dimensional key information from the true intent description text, and the information extraction results of multiple dimensions are output by the information extraction model. The information extraction model can be implemented based on a neural network model architecture such as LLM, CNN, or RNN.

[0091] For example, the real intention description text is filled into the Prompt template corresponding to the information extraction model to obtain the input text corresponding to the information extraction model, which is then input into the information extraction model to extract key information in multiple dimensions, and the information extraction results in multiple dimensions are output by the information extraction model.

[0092] For example, the Prompt template corresponding to the information extraction model can be:

[0093] {“You are a key information extraction expert in a specific scenario domain. Your responsibility is to extract key information from the input true intent description text.”

[0094] "The actual intent description text entered is": ...

[0095] Extract the scene environment information from the true intention description text to obtain the image description text, the attribute information of the target object of interest, and the relationship between the target objects of interest, and output them in the following format. If no relevant aspects are involved, the corresponding item output is None.

[0096] {Image description text: ...

[0097] Target object's attribute information:

[0098] The relationship between each target object:...}}.

[0099] By performing multi-dimensional key information extraction on the text describing the true intent, we can avoid the problem of inaccurate key information extraction due to text defects in the original search text itself. In addition, multi-dimensional key information extraction can improve the information richness of image retrieval, facilitate subsequent image retrieval, and improve image retrieval accuracy.

[0100] Step S2213: Encode the information extraction results of multiple dimensions to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information.

[0101] The information extraction results of multiple dimensions are encoded into data that can be recognized by a computer to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information.

[0102] Exemplarily, a text encoding model is used to encode information extraction results of multiple dimensions corresponding to the retrieval information to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information.

[0103] Exemplarily, the information extraction results of multiple dimensions corresponding to the text encoding model and the multimodal visual model are respectively encoded to obtain the retrieval information vectors of multiple dimensions output by the text encoding model and the retrieval information vectors of multiple dimensions output by the multimodal visual model, thereby improving the feature expression dimension of the retrieval information vector.

[0104] For example, feature vectors of the image description text, the attribute information of the target object, and the association relationship between each target object are extracted through the text encoding model to obtain the first image description retrieval vector A1, the first object description retrieval vector B1, and the association retrieval vector C respectively; and feature vectors of the image description text and the attribute information of the target object are extracted through the multimodal vision model to obtain the second image description retrieval vector A2 and the second object description retrieval vector B2 respectively.

[0105] Since the search text entered by the user may have problems such as ambiguous descriptions and unclear references, resulting in poor quality of the original search text, directly using the original search text for image retrieval may affect the retrieval accuracy. Taking the above problems into consideration, this application optimizes and rewrites the original search text through intent recognition to obtain the true intent description text to improve the quality of the text description. By performing multi-dimensional feature extraction on the true intent description text, information expansion can be performed to improve the accuracy of subsequent image retrieval.

[0106] In some embodiments, the retrieval information includes a reference image, and step S220 performs feature extraction of multiple dimensions on the retrieval information to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information, including the following steps S2221 to S2222.

[0107] Step S2221: extract at least two of the corresponding image description text, the attribute information of the target object, and the association relationship between each target object from the reference image to obtain information extraction results in multiple dimensions.

[0108] Multi-dimensional key information extraction is performed on the reference image to obtain multi-dimensional information extraction results.

[0109] For example, a pre-trained image processing model is used to extract multi-dimensional key information from a reference image, obtaining multi-dimensional information extraction results output by the image processing model. The image processing model can be implemented based on a neural network model architecture such as Visual-Language Models (VLM), CNN, or RNN.

[0110] For example, scene information that can be referenced in the reference image is extracted to obtain image description text, and / or attribute information of target objects that can be referenced in the reference image is extracted, and / or association relationships between target objects that can be referenced in the reference image are extracted.

[0111] Step S2222: Encode the information extraction results of multiple dimensions to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information.

[0112] The encoding method is similar to step S2213 and will not be described here in detail.

[0113] In some embodiments, the retrieval information includes retrieval text and reference images. The information extraction results of multiple dimensions obtained in combination with step S2212 and step S2221 are encoded to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information.

[0114] For example, the true intention description text corresponding to the retrieved text is filled into the Prompt template corresponding to the image processing model to obtain the input text corresponding to the image processing model, and the input text corresponding to the image processing model and the reference image are input into the image processing model to extract key information of multiple dimensions from the reference image, and obtain the information extraction results of multiple dimensions output by the image processing model.

[0115] For example, the prompt template corresponding to the image processing model can be:

[0116] {“You are a key information extraction expert in a specific scenario. Your responsibility is to extract and analyze the corresponding content information from the reference image based on the true intention description text.”

[0117] "The actual intent description text entered is": ...

[0118] According to the true intention description text, the scene environment information in the reference image is extracted to obtain the image description text, the attribute information of the target object of interest, and the relationship between the target objects of interest. The output is in the following format. If the output does not involve relevant aspects, the corresponding item output is None.

[0119] {Image description text: ...

[0120] Target object's attribute information:

[0121] The relationship between each target object:...}}.

[0122] The information extraction results corresponding to the retrieval text and the information extraction results corresponding to the reference image are integrated, and the integrated information extraction results of multiple dimensions are encoded to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information.

[0123] In some embodiments, the search information contains at least a search text, see Figure 3 , Figure 3 FIG. 1 is a flowchart of an exemplary embodiment of the present application showing feature extraction of search information. Figure 3 As shown, the process includes the following steps S301 to S306.

[0124] Step S301: Obtain the search text in the search information.

[0125] Step S302: performing intent recognition on the search text to obtain the true intent, and optimizing the search text according to the true intent to obtain a true intent description text.

[0126] Step S303: extract information in multiple dimensions from the true intention description text.

[0127] Step S304: Determine whether the search information contains a reference image. If so, execute steps S305 to S306. If not, end the process.

[0128] Step S305: extracting information extraction results of multiple dimensions from the reference image.

[0129] Step S306: Integrate the information extraction result of the retrieved text and the information extraction result of the reference image to obtain a final information extraction result.

[0130] Optionally, if the reference image contains a target object that needs to be referenced, the image area containing the target object is extracted from the reference image based on the position information of the target object in the reference image to obtain an object retrieval sub-image, and the object retrieval sub-image is image encoded to obtain a sub-image retrieval vector, so as to realize image retrieval using the sub-image retrieval vector in subsequent image retrieval.

[0131] Optionally, the reference image may be encoded to obtain an image retrieval vector, so that the image retrieval vector can be used to implement image retrieval during subsequent image retrieval.

[0132] Then, the matching degrees between the retrieval information vectors of multiple dimensions and the feature vectors in the feature vector database of the corresponding dimensions are calculated, and the feature vectors whose matching degrees meet the preset conditions are respectively selected from the feature vector database of multiple dimensions to obtain the matching feature vectors corresponding to each dimension.

[0133] For example, the feature vector database includes an overall image vector database, which contains a description text vector corresponding to each base image. Retrieval information (such as retrieval text and / or reference image) is obtained to extract an image description retrieval vector corresponding to the image description text. A matching degree is calculated between the image description retrieval vector and the description text vector corresponding to each base image. Then, from the description text vectors contained in the overall image vector database, a description text vector whose matching degree meets a preset condition is selected to obtain a matching feature vector corresponding to the first dimension. For example, the top K description text vectors with the largest matching degrees are selected to obtain a matching feature vector corresponding to the first dimension.

[0134] Optionally, the overall image vector database further contains an image vector corresponding to each base image, and the retrieval information contains a reference image. An image retrieval vector obtained by image encoding the reference image is obtained, and an image description retrieval vector corresponding to the image description text extracted from the retrieval information is obtained. A matching degree is then calculated between the image retrieval vector and the image vector corresponding to each base image, and a matching degree is calculated between the image description retrieval vector and the description text vector corresponding to each base image. Then, from the description text vectors and image vectors contained in the overall image vector database, a description text vector and an image vector whose matching degree meets a preset condition are selected to obtain a matching feature vector corresponding to the first dimension, such as selecting the top K description text vectors and image vectors with the largest matching degrees to obtain a matching feature vector corresponding to the first dimension.

[0135] For another example, the feature vector database includes an object vector database, which contains description text vectors corresponding to target objects in each base image. Attribute information of the target object extracted from the retrieval information is obtained and encoded to obtain an object description retrieval vector. The object description retrieval vector is then matched against the description text vector corresponding to the target object in each base image. Then, from the description text vectors in the object vector database, description text vectors with matching degrees that meet preset conditions are selected to obtain matching feature vectors corresponding to the second dimension. For example, the top K description text vectors with the largest matching degrees are selected to obtain matching feature vectors corresponding to the second dimension.

[0136] Optionally, the object vector database further contains an object subgraph vector corresponding to the target object in each base image, and the retrieval information contains a reference image. A subgraph retrieval vector obtained by encoding the target object in the reference image is obtained, and an object description retrieval vector extracted based on other retrieval information is obtained. The object description retrieval vector is respectively matched with the description text vector corresponding to the target object in each base image, and the subgraph retrieval vector is respectively matched with the object subgraph vector corresponding to the target object in each base image. Then, from the description text vectors and object subgraph vectors contained in the object vector database, the description text vectors and object subgraph vectors whose matching degree meets a preset condition are selected to obtain a matching feature vector corresponding to the second dimension, such as selecting the top K description text vectors and object subgraph vectors with the largest matching degree to obtain a matching feature vector corresponding to the second dimension.

[0137] For another example, the feature vector database includes a relationship vector database, which contains object relationship vectors corresponding to the association relationships between target objects in each base image. The association relationships between target objects extracted from the retrieval information are obtained and encoded to obtain association retrieval vectors. The association retrieval vectors are then matched against the object relationship vectors corresponding to each target object in each base image. Then, from the object relationship vectors in the relationship vector database, object relationship vectors whose matching degrees meet preset conditions are selected to obtain matching feature vectors corresponding to the third dimension. For example, the top K object relationship vectors with the largest matching degrees are selected to obtain matching feature vectors corresponding to the third dimension.

[0138] Then, the base library images associated with the matching feature vectors corresponding to each dimension are determined to obtain the image retrieval results.

[0139] Exemplarily, the base images associated with the matching feature vectors corresponding to each dimension are determined to obtain candidate images, and candidate images that meet preset conditions are selected from the candidate images as image retrieval results.

[0140] For example, the final matching degree corresponding to each candidate image is obtained, where if the number of occurrences of the same candidate image in all dimensions is greater than or equal to 2, then a candidate image corresponds to multiple matching degrees. The average value of all matching degrees corresponding to the candidate image can be calculated, or the maximum value of all matching degrees corresponding to the candidate image can be selected, or all matching degrees corresponding to the candidate image can be weighted and summed according to the weight parameter corresponding to each dimension to obtain the final matching degree corresponding to the candidate image; if the number of occurrences of the same candidate image in all dimensions is 1, then the matching degree of the candidate image is directly used as the final matching degree. Then, the candidate image whose final matching degree meets the preset conditions is selected as the image retrieval result, such as selecting the candidate image with the highest final matching degree as the image retrieval result, or selecting the candidate image whose final matching degree is greater than a preset threshold as the image retrieval result, or selecting a preset number of candidate images with the largest final matching degree as the image retrieval result.

[0141] For another example, a preset number of candidate images with the highest matching degree in each dimension are obtained respectively, and these candidate images are used as image retrieval results.

[0142] For another example, an image score is performed on each candidate image, and the candidate image is selected based on the image score.

[0143] For example, in step S240 , determining the base library images associated with the matching feature vectors corresponding to each dimension to obtain the image retrieval results includes steps S241 to S244 .

[0144] Step S241: Determine the base images associated with the matching feature vectors corresponding to each dimension to obtain candidate images.

[0145] Each feature vector in the feature vector database is associated with a base image, and the base images associated with the matching feature vectors corresponding to each dimension are determined, and these associated base images are used as candidate images.

[0146] Step S242: Calculate the image score of the candidate image for each dimension.

[0147] Each candidate image corresponding to each dimension is scored separately to obtain an image score corresponding to each candidate image.

[0148] Exemplarily, the candidate images are scored based on the numerical value of the matching degree of the candidate image, and / or the matching degree relationship between the candidate image and other candidate images in the corresponding dimension, and / or the number of occurrences of the same candidate image in all dimensions to obtain an image score corresponding to each candidate image.

[0149] For example, calculating the image score of the candidate image for each dimension includes: obtaining the matching degree corresponding to the matching feature vector associated with the candidate image; sorting all the candidate images corresponding to each dimension in order according to the matching degree, and obtaining the sorting result corresponding to each dimension; scoring the candidate image based on the ranking of the candidate image in each sorting result, and obtaining the sorting score of the candidate image for each dimension.

[0150] Specifically, if all candidate images corresponding to each dimension are sorted in descending order according to the degree of matching, the higher the ranking of the candidate image in the sorting result, the higher the ranking score of the candidate image in the sorting result; conversely, if sorted in ascending order, the lower the ranking of the candidate image in the sorting result, the higher the ranking score of the candidate image in the sorting result.

[0151] Taking descending sorting as an example, if the sorting result of the current dimension contains K candidate images, the ranking score calculation method corresponding to any candidate image in the current dimension can be found in the following formula 1:

[0152] score = (K-i+1) / K (Formula 1)

[0153] Where score is the ranking score corresponding to the candidate image, and i is the ranking rank of the candidate image in the ranking result of the current dimension. If the candidate image ranks first, the ranking score of the candidate image is 1 point. If the candidate image ranks last, the ranking score of the candidate image is 1 / K points.

[0154] For another example, the candidate images are scored based on the number of times the same candidate image appears in each dimension, and the number of times the candidate image appears in each dimension is obtained.

[0155] Specifically, the higher the number of occurrences of the same candidate image in each dimension, the higher the number score of the candidate image; the lower the number of occurrences of the same candidate image in each dimension, the lower the number score of the candidate image.

[0156] Based on the ranking score and / or frequency score of the candidate images for each dimension, the image score of the candidate images for each dimension is calculated. For example, for each dimension, the ranking score and frequency score are weighted summed to obtain the image score of the candidate images for each dimension, or the maximum or minimum value of the ranking score and frequency score is selected as the image score of the candidate images for each dimension, which is not limited in this application.

[0157] Step S243: combining the image scores of the candidate images in each dimension, and calculating the total score corresponding to the candidate images.

[0158] After obtaining the image score of the candidate image for each dimension, the total score corresponding to the candidate image is calculated based on the image score of each dimension.

[0159] For example, the image scores of each dimension can be weighted and summed to obtain the total score corresponding to the candidate image; the maximum or minimum value of the image scores of each dimension can also be selected to obtain the total score corresponding to the candidate image. This application does not limit this.

[0160] For example, the weight parameters corresponding to each dimension are obtained; based on the weight parameters corresponding to each dimension, the image scores of the candidate images for each dimension are weighted and summed to obtain the total score corresponding to the candidate image.

[0161] Among them, the weight parameters corresponding to each dimension can be pre-set based on experience. For example, the user sets the weight parameters corresponding to each dimension based on the attention paid to each dimension; the weight parameters corresponding to each dimension can also be flexibly calculated according to the current specific situation.

[0162] For example, the search information is parsed, and the search attention corresponding to each dimension is determined based on the parsing results; based on the search attention corresponding to each dimension, the weight parameters corresponding to each dimension are set.

[0163] For example, if the search information contains search text, the search text is parsed to determine the degree of association between the text content of the search text and information of different dimensions, and the search attention corresponding to each dimension is obtained. For example, by extracting keywords from the search text, the degree of association between keywords and different dimensions is calculated, and the search attention corresponding to each dimension is determined. For example, the features of multiple dimensions include the description text of the image, the attribute information of the target object contained in the image, and the association relationship between each target object in the image. If the extracted keywords are used to describe the image scene, then the degree of association between the keyword and the dimension of "image description text" is the highest; if the extracted keywords are used to describe the target object, then the degree of association between the keyword and the dimension of "attribute information of the target object" is the highest; if the extracted keywords are used to describe the relationship between the target objects, then the degree of association between the keyword and the dimension of "association between target objects" is the highest. By comprehensively considering the degree of association between each keyword and each dimension, the search attention corresponding to each dimension is obtained.

[0164] For example, if the search information contains a reference image, the reference image is parsed to determine the degree of correlation between the image content of the reference image and information of different dimensions, thereby obtaining the corresponding search attention of each dimension. For example, image information is extracted from the search information, and the extracted image information is categorized by dimension to determine the richness of the image information of each dimension. The richer the image information of the corresponding dimension, the higher the search attention of the dimension. Conversely, the less rich the image information of the corresponding dimension, the lower the search attention of the dimension.

[0165] Of course, a pre-trained retrieval information parsing model may also be used. The retrieval information parsing model may analyze the correlation between the input retrieval information and different dimensions, thereby obtaining the retrieval attention corresponding to each dimension.

[0166] Then, according to the search attention corresponding to each dimension, the weight parameters corresponding to each dimension are set. Specifically, the search attention is positively correlated with the weight parameter, the sum of the weight parameters of each dimension is 1, and the weight parameters of each dimension are not negative.

[0167] For another example, the retrieval information contains different types of retrieval data; obtaining the weight parameters corresponding to each dimension includes: determining the data quality of different types of retrieval data in the retrieval information respectively; setting the weight parameters of the matching feature vector corresponding to each dimension based on the data quality of different types of retrieval data; wherein the data quality of any type of retrieval data is positively correlated with the weight parameters corresponding to the candidate images obtained by screening the retrieval data of any type through matching degree calculation.

[0168] For example, the retrieval information contains two types of retrieval data: retrieval text and reference image, which determine the data quality of the retrieval text and reference image respectively. For example, the more words the retrieval text contains, the more detailed the text content description, the more key information, etc., the higher the data quality of the retrieval text; the clearer the reference image, the lower the occlusion rate of the target object contained, etc., the higher the data quality of the reference image.

[0169] According to the data quality of different types of retrieval data, the weight parameters of the matching feature vector corresponding to each dimension are set. Specifically, the higher the data quality of any type of retrieval data, the higher the weight parameter corresponding to the candidate image obtained by matching calculation and screening of the retrieval data of this type. Conversely, the lower the data quality of any type of retrieval data, the lower the weight parameter corresponding to the candidate image obtained by matching calculation and screening of the retrieval data of this type.

[0170] For example, the image description retrieval vector, and / or object description retrieval vector, and / or associated retrieval vector obtained by extracting the retrieval text are all used as text-related vectors. If the data quality of the retrieval text is higher, the weight parameter corresponding to the candidate image screened by the matching calculation with the text-related vector is higher. If the data quality of the retrieval text is lower, the weight parameter corresponding to the candidate image screened by the matching calculation with the text-related vector is lower. Similarly, the image description retrieval vector, and / or object description retrieval vector, and / or associated retrieval vector, and / or image retrieval vector, and / or sub-image retrieval vector obtained by extracting the reference image are all used as image-related vectors. If the data quality of the reference image is higher, the weight parameter corresponding to the candidate image screened by the matching calculation with the image-related vector is higher. If the data quality of the reference image is lower, the weight parameter corresponding to the candidate image screened by the matching calculation with the image-related vector is lower.

[0171] In the above embodiment, in addition to considering the attention paid to different dimensions to set the overall weight parameters for the corresponding dimensions, the weight parameters for each candidate image in the corresponding dimension can also be set based on the data quality of the corresponding type of search data involved in the matching calculation of the candidate images screened in the different dimensions. In other words, when calculating the weighted sum of the image scores of the candidate images for each dimension, in addition to considering the overall weight parameters of the dimension, the specific weight parameters of the candidate image in the corresponding dimension can also be considered.

[0172] For example, a first dimension and a second dimension coexist, and the retrieval information includes retrieval text and reference images. Candidate image matching is performed using the retrieval information, and the candidate images corresponding to the first dimension contain candidate image P1. Furthermore, the candidate images corresponding to the second dimension also contain candidate image P1. The calculated image score for candidate image P1 is a for the first dimension and b for the second dimension. In the first dimension, candidate image P1 is obtained by filtering based on the vector extracted from the retrieval text, and in the second dimension, it is obtained by filtering based on the vector extracted from the reference image. Based on the retrieval attention corresponding to each dimension, the overall weight parameter for the first dimension is w1, and the overall weight parameter for the second dimension is w2. Based on the data quality of the retrieval text, the weight parameter for the candidate images filtered out is w3, and based on the data quality of the reference image, the weight parameter for the candidate images filtered out is w4. The final total score for candidate image P1 is S = a × w1 × w3 + b × w2 × w4. Of course, if only w1 and w2 are considered, the final total score of the candidate image P1 is S=a×w1+b×w2; if only w3 and w4 are considered, the final total score of the candidate image P1 is S=a×w3+b×w4.

[0173] By setting the weight parameters used for weighted summation calculation of image scores for each dimension according to the retrieval attention corresponding to each dimension and / or the data quality of different types of retrieval data, the accuracy of the total score calculation is improved, making it easier for subsequent screening to obtain image retrieval results that better meet user needs and have higher quality.

[0174] According to the above weight parameters, the image scores of the candidate images in each dimension are weighted and summed up to obtain the total score corresponding to the candidate image.

[0175] Step S244: selecting candidate images whose total scores satisfy preset conditions as image retrieval results corresponding to the image retrieval instruction.

[0176] For example, the candidate image with the highest total score is selected as the image retrieval result, or the candidate image with a total score greater than a preset threshold is selected as the image retrieval result, or a preset number of candidate images with the largest total score are selected as the image retrieval result. This application does not limit this.

[0177] By combining the image scores of each dimension to select candidate images, more accurate image retrieval results can be obtained.

[0178] The image retrieval method provided in the present application performs multi-dimensional feature extraction on the retrieval information to obtain multi-dimensional retrieval information vectors corresponding to the retrieval information. The multi-dimensional feature extraction can be performed on the input retrieval information to improve the expression ability of the retrieval information vector. Then, the multi-dimensional retrieval information vectors are matched with the multi-dimensional feature vector database respectively, and the feature vectors whose matching degree meets the preset conditions are respectively screened out from the multi-dimensional feature vector database to obtain the matching feature vectors corresponding to each dimension, and the underlying library images associated with the matching feature vectors corresponding to each dimension are determined to obtain the image retrieval results, thereby improving the accuracy of the image retrieval results.

[0179] Figure 4 FIG. 1 is a block diagram of an image retrieval device according to an exemplary embodiment of the present application. Figure 6 As shown, the exemplary image retrieval apparatus 400 includes:

[0180] The base image feature extraction module 410 is used to extract features of multiple dimensions from each base image to obtain a feature vector database of multiple dimensions corresponding to each base image.

[0181] The retrieval feature extraction module 420 is used to receive retrieval information, perform feature extraction on the retrieval information in multiple dimensions, and obtain retrieval information vectors in multiple dimensions corresponding to the retrieval information;

[0182] Matching vector selection module 430 is used to calculate the matching degree between the search information vectors of multiple dimensions and the feature vectors in the feature vector database of the corresponding dimensions, and select the feature vectors whose matching degree meets the preset conditions from the feature vector database of the multiple dimensions to obtain the matching feature vector corresponding to each dimension;

[0183] The result generation module 440 is used to determine the base library images associated with the matching feature vectors corresponding to each dimension, and obtain image retrieval results.

[0184] It should be noted that the image retrieval device provided in the above embodiment and the image retrieval method provided in the above embodiment are based on the same concept. The specific manner in which the various modules and units perform their operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the image retrieval device provided in the above embodiment can, as needed, allocate the above functions to different functional modules, i.e., divide the internal structure of the device into different functional modules to perform all or part of the functions described above. This is not a limitation herein.

[0185] See also Figure 5 , Figure 5is a schematic diagram of the structure of an embodiment of an electronic device of the present application. Electronic device 500 includes memory 501 and processor 502. Processor 502 is configured to execute program instructions stored in memory 501 to implement the steps of any of the above-described image retrieval method embodiments. In a specific implementation scenario, electronic device 500 may include, but is not limited to, a microcomputer and a server. Furthermore, electronic device 500 may also include mobile devices such as laptops and tablet computers, without limitation herein.

[0186] Specifically, processor 502 is used to control itself and memory 501 to implement the steps of any of the above-mentioned image retrieval method embodiments. Processor 502 may also be referred to as a central processing unit (CPU). Processor 502 may be an integrated circuit chip with signal processing capabilities. Processor 502 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. In addition, processor 502 may be implemented by an integrated circuit chip.

[0187] See also Figure 6 , Figure 6 The computer-readable storage medium 600 stores program instructions 610 that can be executed by a processor, and the program instructions 610 are used to implement the steps of any of the above-mentioned image retrieval method embodiments.

[0188] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0189] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0191] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in either hardware or software functional units. If the integrated units are implemented as software functional units and sold or used as standalone products, they may be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product, stored on a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

Claims

1. An image retrieval method, characterized in that: The method comprises: Performing feature extraction on each base image in multiple dimensions to obtain a feature vector database corresponding to each base image in multiple dimensions; wherein the features in the multiple dimensions include at least the association relationship between target objects in the image, and at least one of the following: descriptive text of the image and attribute information of the target objects contained in the image; Receiving search information, the search information including a search text, performing intent recognition on the search text to obtain a true intent, and optimizing and rewriting the search text based on the true intent to obtain a true intent description text; Performing feature extraction of multiple dimensions on the true intention description text to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information; Calculating the matching degrees between the retrieval information vectors of the multiple dimensions and the feature vectors in the feature vector database of the corresponding dimensions, and filtering out the feature vectors whose matching degrees meet the preset conditions from the feature vector database of the multiple dimensions to obtain the matching feature vectors corresponding to each dimension; Determine the base library images associated with the matching feature vectors corresponding to each dimension, and obtain image retrieval results.

2. The method according to claim 1, characterized in that The retrieval information includes a retrieval text and a reference image; and the step of extracting features of multiple dimensions from the retrieval information to obtain retrieval information vectors of multiple dimensions corresponding to the retrieval information includes: Performing intent recognition on the search text to obtain a true intent, optimizing the search text based on the true intent to obtain a true intent description text; extracting from the true intent description text at least one of an image description text and attribute information of a target object, and associations between target objects, to obtain information extraction results in multiple dimensions; Extracting at least one of corresponding image description text, attribute information of target objects, and association relationships between target objects from the reference image to obtain information extraction results in multiple dimensions; The information extraction results of the multiple dimensions are encoded to obtain retrieval information vectors of the multiple dimensions corresponding to the retrieval information.

3. The method according to claim 1, characterized in that Determining the base library images associated with the matching feature vectors corresponding to each dimension to obtain image retrieval results includes: Determine the base images associated with the matching feature vectors corresponding to each dimension to obtain candidate images; Calculating an image score for each dimension of the candidate image; Based on the image scores of each dimension of the candidate image, a total score corresponding to the candidate image is calculated; The candidate images whose total scores meet the preset conditions are selected as the image retrieval results corresponding to the image retrieval instruction.

4. The method according to claim 3, characterized in that Calculating the image score of the candidate image for each dimension includes: Obtaining a matching degree corresponding to a matching feature vector associated with the candidate image; Sort all candidate images corresponding to each dimension in order according to the degree of matching, to obtain a sorting result corresponding to each dimension; score the candidate image based on its ranking in each sorting result, to obtain a sorting score for the candidate image for each dimension; and / or, scoring the candidate images based on the number of occurrences of the same candidate image in each dimension, to obtain a number score of the candidate images for each dimension; The image score of the candidate image is calculated based on the ranking score and / or the frequency score of the candidate image for each dimension.

5. The method according to claim 3, characterized in that The step of combining the candidate images with the image scores of each dimension to calculate the total score corresponding to the candidate images includes: Obtaining the weight parameters corresponding to each dimension; Based on the weight parameters corresponding to each dimension, a weighted sum calculation is performed on the image scores of the candidate images for each dimension to obtain a total score corresponding to the candidate image.

6. The method according to claim 5, characterized in that The obtaining of the weight parameters corresponding to each dimension includes: Analyze the search information, and determine the search attention corresponding to each dimension based on the analysis result; Based on the retrieval attention corresponding to each dimension, the weight parameters corresponding to each dimension are set.

7. The method according to claim 5, characterized in that The search information contains different types of search data; obtaining the weight parameters corresponding to each dimension includes: respectively determining the data quality of different types of search data in the search information; Based on the data quality of the different types of retrieval data, the weight parameters corresponding to the candidate images in each dimension are set; wherein the data quality of any type of retrieval data is positively correlated with the weight parameters corresponding to the candidate images obtained by screening the retrieval data of any type through matching calculation.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 7.

Images