Self-adaptive multi-level image searching method and system based on query difficulty

Abstract

The invention relates to the field of information retrieval, and provides a self-adaptive multi-level image searching method and system based on query difficulty. The method comprises the following steps: receiving a query text input by a user, and performing complexity evaluation on the query text to obtain a complexity label; performing feature coding on the query text by using a multi-modal embedding model to obtain a query feature vector; performing similarity matching on the query feature vector and a pre-constructed image feature index to obtain a preliminary retrieval result set; according to the complexity label, a corresponding retrieval strategy is selected to carry out enhancement processing on the preliminary retrieval result set, and a candidate image set is obtained; and fusing and sorting the images in the candidate image set, and screening to obtain a final retrieval result. According to the method, the retrieval strategy can be adaptively selected according to the query complexity, and the retrieval precision and efficiency are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information retrieval, and particularly relates to a multi-level text-to-image search method and system based on query difficulty self-adaptation. BACKGROUND

[0002] At present, the text-to-image search technology has been widely applied in the image retrieval field, and a mainstream implementation scheme thereof is based on vector similarity calculation of a multi-modal embedding model. Specifically, a system uses a multi-modal model such as CLIP to extract features of all images in an image library in advance, encodes each image into a high-dimensional feature vector, and constructs a vector index; when a user inputs a text query, the same model is used to encode the text into a feature vector, and then an approximate nearest neighbor search algorithm such as FAISS or HNSW is used to quickly search for the most similar image vector to the text vector in the vector index, so as to return a semantically relevant image result. In the processing of a simple query, for example, a query containing a single subject or a simple attribute description, the method can quickly and accurately return relevant images.

[0003] However, the existing text-to-image search technology has obvious limitations. First, all queries use a unified single retrieval strategy, lacking the ability to perceive the complexity of the query, and for a query containing multiple entities, complex spatial relationships or rich detail descriptions, a single vector retrieval cannot capture all the fine-grained semantic information and multi-entity co-occurrence relationships in the text, resulting in a significant decrease in retrieval accuracy; second, the system cannot dynamically adjust the allocation of computing resources according to the query difficulty, which causes a waste of computing power when using a complex model for a simple query, and cannot meet the accuracy requirements when using a simple model for a complex query, making it difficult to balance between retrieval efficiency and accuracy; third, when processing a long text query, the traditional method directly encodes the entire text, which can cause the loss of key detail information or the averaging of semantics, making the retrieval result deviate from the user's true intention; in addition, the existing method generally uses the whole image matching method for similarity calculation, ignoring the feature information of local regions and key entities in the image, and for query scenes that need to match multiple specific objects or pay attention to specific details, the retrieval accuracy is insufficient. SUMMARY

[0004] In view of the deficiencies of the prior art, the purpose of the application is to provide a multi-level text-to-image search method and system based on query difficulty self-adaptation.

[0005] The application provides a multi-level text-to-image search method based on query difficulty self-adaptation, comprising:

[0006] S1, receiving a query text input by a user, evaluating the complexity of the query text, and obtaining a complexity label;

[0007] S2, feature coding is carried out on the query text using a multi-modal embedding model, and a query feature vector is obtained;

[0008] S3, similarity matching is carried out between the query feature vector and a pre-constructed image feature index, and a preliminary retrieval result set is obtained;

[0009] S4, according to the complexity label, a corresponding retrieval strategy is selected to perform enhancement processing on the preliminary retrieval result set, and a candidate image set is obtained;

[0010] S5, fusion sorting is performed on the images in the candidate image set, and a final retrieval result is obtained through screening.

[0011] According to the multi-level text-to-image search method based on query difficulty self-adaption provided by the application, step S1 specifically comprises:

[0012] S11, receiving a query text input by a user;

[0013] S12, performing semantic representation processing on the query text through a text encoder, and obtaining query semantic features;

[0014] S13, inputting the query semantic features into a pre-trained complexity classification model, and performing difficulty discrimination processing on the query semantic features through a classification decision layer, and obtaining the complexity label.

[0015] According to the multi-level text-to-image search method based on query difficulty self-adaption provided by the application, the complexity label in step S1 comprises: simple label, medium label and complex label.

[0016] According to the multi-level text-to-image search method based on query difficulty self-adaption provided by the application, when the complexity label is a medium label, step S4 further comprises:

[0017] S411, performing entity positioning processing on a plurality of images in the preliminary retrieval result set using a target detection model, and obtaining a set of entity region bounding boxes corresponding to the plurality of images;

[0018] S412, performing cropping processing on the plurality of images according to the set of entity region bounding boxes, and obtaining a set of entity subgraphs;

[0019] S413, performing feature extraction processing on a plurality of entity subgraphs in the set of entity subgraphs through the multi-modal embedding model, and obtaining a set of entity subgraph feature vectors;

[0020] S414, performing similarity retrieval processing on a plurality of entity subgraph feature vectors in the set of entity subgraph feature vectors and the image feature index respectively, and obtaining a plurality of entity subgraphs corresponding to similar image results;

[0021] S415, merge the preliminary search result set with the similar image result to obtain a candidate image set.

[0022] According to the multi-level text-to-image search method based on query difficulty self-adaptation provided by the application, step S415 further includes:

[0023] S4151, record the similarity scores of multiple images in the preliminary search result set as first type scores, and record the similarity scores of multiple images in the similar image result corresponding to the multiple entity subgraphs as second type scores;

[0024] S4152, construct a candidate image pool, traverse the preliminary search result set and the similar image result corresponding to the multiple entity subgraphs, and add the appearing images and corresponding scores to the candidate image pool;

[0025] S4153, for each image in the candidate image pool, extract the first type score and the second type score corresponding to the current image, and obtain a comprehensive score of the current image through weighted average operation;

[0026] S4154, sort the images in the candidate image pool in descending order according to the comprehensive scores, and select a preset number of images from the sorting result to form a candidate image set.

[0027] According to the multi-level text-to-image search method based on query difficulty self-adaptation provided by the application, in step S4, when the complexity label is a complex label, step S4 further includes:

[0028] S421, construct a prompt word template containing decomposition instructions, fill the query text into the prompt word template, and perform semantic analysis processing on the filled prompt word through a large language model to obtain multiple sub-query texts;

[0029] S422, perform feature encoding processing, similarity matching processing and multi-hop retrieval processing on multiple sub-query texts respectively to obtain multiple sub-query text corresponding sub-retrieval result sets;

[0030] S423, use a text embedding model to respectively perform vectorization processing on the query text and multiple sub-query texts to obtain an original query vector and multiple sub-query vectors;

[0031] S424, calculate the semantic similarity between multiple sub-query vectors and the original query vector, and perform normalization processing on the semantic similarity to obtain multiple sub-query text corresponding semantic weights;

[0032] S425, according to the semantic weight, weighted fusion is performed on the sub-retrieval result set corresponding to the multiple sub-query texts to obtain a candidate image set.

[0033] According to the method, the step S425 further comprises:

[0034] S4251, the plurality of sub-query text corresponding sub-retrieval result set is summarized, and a candidate image pool is obtained;

[0035] S4252, for each candidate image in the candidate image pool, the plurality of sub-query text corresponding sub-retrieval result set is traversed, and the similarity score of the current candidate image in the plurality of sub-retrieval result set is recorded;

[0036] S4253, the similarity score and the corresponding semantic weight are weighted and summed, and the total weighted score of the current candidate image is obtained;

[0037] S4254, according to the total weighted score, the plurality of candidate images in the candidate image pool are sorted and screened, and the candidate image set is obtained.

[0038] The application further provides a multi-level text search image system based on query difficulty adaptation, comprising:

[0039] The evaluation module is used for receiving the query text input by the user, evaluating the complexity of the query text, and obtaining a complexity label;

[0040] The encoding module is configured as a multi-modal embedding model, which is used for feature encoding of the query text to obtain a query feature vector;

[0041] The matching module is used for similarity matching of the query feature vector and a pre-constructed image feature index to obtain a preliminary retrieval result set;

[0042] The enhancement module is used for selecting a corresponding retrieval strategy according to the complexity label to enhance the preliminary retrieval result set to obtain a candidate image set;

[0043] The sorting module is used for fusion sorting of the images in the candidate image set to obtain a final retrieval result.

[0044] The third aspect of the application provides a multi-level text search image device based on query difficulty adaptation, comprising:

[0045] A memory and at least one processor, the memory has instructions stored therein;

[0046] At least one processor calls the instructions in the memory, so that a multi-level text search image device based on query difficulty adaptation executes a multi-level text search image method based on query difficulty adaptation as described in any one of the above.

[0047] The fourth aspect of the present application provides a computer readable storage medium, which stores instructions, and the instructions are executed by a processor to implement the query difficulty adaptive multi-level text-to-image search method according to any one of the above.

[0048] The present application provides a query difficulty adaptive multi-level text-to-image search method, system, device and storage medium. First, the complexity classifier is constructed to evaluate the real-time complexity of the query text, and different retrieval strategies are adaptively selected according to the evaluation results, so that the system can flexibly allocate computing resources according to different difficulty queries, avoiding the problem of over-consumption of computing power for simple queries and insufficient processing depth for complex queries, and significantly improving the overall retrieval efficiency and resource utilization. When processing medium complexity queries, the target detection model is used to perform entity positioning and region cropping on the preliminary retrieval results, extract the local features of the key entities and perform secondary similarity retrieval, breaking through the limitations of traditional whole image matching, and accurately capturing the detailed information and multi-entity spatial relationship in the image, so that the query accuracy of queries containing multiple specific objects or specific detail requirements is greatly improved. For complex queries, the long text is automatically decomposed into multiple semantically complete sub-queries using a large language model, solving the problems of semantic loss and averaging in traditional methods, and through parallel processing of each sub-query and weighted fusion based on semantic similarity, the complete semantics of the original query is maintained, and the retrieval accuracy and processing efficiency are improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings are included to provide a further understanding of the embodiments of the present application, and are incorporated herein and constitute a part of the detailed description. It should be apparent to those skilled in the art that the accompanying drawings are only some embodiments of the present application described in the specification. Other drawings can also be obtained by those skilled in the art according to the drawings.

[0050] Figure 1 A query difficulty adaptive multi-level text-to-image search method is provided for the embodiments of the present application.

[0051] Figure 2 A query difficulty adaptive multi-level text-to-image search system structure is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0052] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions of the present application will be described clearly and completely in conjunction with the drawings below. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0053] In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concepts disclosed in the present application.

[0054] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for description purposes and cannot be understood as indicating or implying relative importance. The terms "mounting", "connecting", "connecting" should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication between two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0055] The exemplary embodiments will be described in detail hereinbelow, examples of which are shown in the accompanying drawings. In the following description, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of methods and systems consistent with some aspects of the present application as detailed in the appended claims.

[0056] In order to better understand the present application, the technical terms appearing in the embodiments of the present application will be explained first.

[0057] Text-to-Image: refers to the process of retrieving images from a large-scale image database that are semantically associated with the query content according to the text query input by the user.

[0058] Multimodal embedding model: refers to a deep learning model that can map data of different modalities (such as text and image) into the same vector space, so that semantically similar text and image have similar vector distances in the space. For example, the CLIP (Contrastive Language-Image Pre-training) model.

[0059] Multi-hop query: refers to a multi-stage retrieval strategy. In this invention, it specifically refers to a process of first performing "text-to-image search" to obtain preliminary results, then performing "image-to-image search" or entity refinement search on the preliminary results, and fusing the results of multiple searches.

[0060] Complexity classifier: a trained machine learning model (such as a text classification model) used to automatically determine the semantic complexity of an input text query and output a corresponding strategy selection label.

[0061] Large language model (LLM): refers to a large-scale pre-trained language model based on the Transformer architecture, such as the GPT series, ChatGLM, LLaMA, etc., which has strong language understanding, generation and reasoning capabilities.

[0062] The following describes an embodiment of the present invention in conjunction with the drawings.

[0063] As shown in Figure 1 The present invention provides a multi-level text-to-image search method based on query difficulty adaptation, which includes:

[0064] S1, receiving a user input query text, performing complexity evaluation on the query text, and obtaining a complexity label.

[0065] Specifically, step S1 includes:

[0066] S11, receiving a user input query text; S12, performing semantic representation processing on the query text through a text encoder to obtain query semantic features.

[0067] Further, the application first receives the query text input by the user through the interactive interface, which is a natural language form of image retrieval requirement description; then inputs the received query text into a text encoder, which is based on a Transformer architecture and contains multiple layers of self-attention mechanism and feedforward neural network layers. Specifically, the text encoder first performs word segmentation processing on the query text, divides the text into a plurality of word token sequences, then assigns each word token a corresponding word embedding vector and position encoding vector, and after processing by multiple Transformer encoding layers, the context information of each word token is fully fused, and finally the hidden state vectors of all word tokens are aggregated into a fixed-dimensional query semantic feature vector through a pooling operation, which contains the overall semantic information of the query text.

[0068] S13, input the query semantic feature into a pre-trained complexity classification model, perform difficulty discrimination processing on the query semantic feature through a classification decision layer, and obtain the complexity label.

[0069] Among them, the complexity label in step S1 includes: simple label, medium label and complex label.

[0070] Further, in step S13, the application inputs the query semantic feature vector into the complexity classification model, which is a pre-trained classifier. The complexity classification model contains a feature processing layer and a classification decision layer. The feature processing layer performs nonlinear transformation and feature extraction on the input query semantic feature. The classification decision layer contains a fully connected layer and a softmax activation function. Specifically, the classification decision layer outputs probability distribution values of three categories, respectively corresponding to simple label, medium label and complex label. The application selects the category with the highest probability value as the final complexity label. The query corresponding to the simple label contains a single subject or simple attribute description. The query corresponding to the medium label contains multiple entities and their spatial relationships. The query corresponding to the complex label contains rich scene elements, artistic style and detail description.

[0071] S2, use a multi-modal embedding model to encode the query text and obtain a query feature vector.

[0072] In step S2, this invention uses a multimodal embedding model to encode the query text. This model is a pre-trained CLIP model, which includes a text encoding branch and an image encoding branch. This invention calls the text encoding branch, inputting the query text into its text encoder. The text encoder, based on the Transformer architecture, performs lexicalization on the query text and then maps the text into a 512-dimensional feature space through multi-layer self-attention calculation and feedforward network transformation, outputting a query feature vector. The obtained query feature vector and image feature vector reside in the same semantic space; semantically similar text and images have relatively close vector distances in this space.

[0073] S3. Perform similarity matching between the query feature vector and the pre-constructed image feature index to obtain a preliminary retrieval result set.

[0074] In step S13, the present invention inputs the query feature vector into a pre-constructed image feature index for retrieval. This image feature index is constructed using the FAISS library and employs an inverted file index structure and product quantization technology. Specifically, the present invention performs an approximate nearest neighbor search in the index, calculating the cosine similarity between the query feature vector and all image feature vectors in the index. The cosine similarity is obtained by dividing the vector dot product by the product of the vector magnitudes, and its value ranges from -1 to 1, with a larger value indicating higher semantic similarity. After calculating the similarity, the present invention sorts the images according to the similarity values, selecting the Top K images with the highest similarity to form a preliminary retrieval result set, with each image corresponding to a similarity score.

[0075] S4. Based on the complexity label, select the corresponding retrieval strategy to enhance the preliminary retrieval result set to obtain a candidate image set.

[0076] Wherein, when the complexity label is a medium label, step S4 further includes:

[0077] S411. Use an object detection model to perform entity localization processing on multiple images in the preliminary retrieval result set to obtain a set of entity region bounding boxes corresponding to multiple images.

[0078] In step S411, the present application inputs each image in the preliminary search result set into the target detection model YOLOv7, which includes a backbone network, a neck network and a detection head. The backbone network performs multi-scale feature extraction on the input image. The neck network fuses feature information of different scales through a feature pyramid network. The detection head performs convolution operation on the fused feature map and predicts the coordinates of the bounding box and the class confidence. The present application performs forward inference on each image, and the detection model outputs the coordinates of all detected objects in the image, each bounding box being represented by four parameters of top-left corner coordinates, width and height. The detection model also outputs the class label and confidence score corresponding to the bounding box. Finally, the present application screens the bounding boxes with a confidence higher than a threshold value, and groups all the screened bounding boxes to form the entity region bounding box set corresponding to the image.

[0079] In step S412, the present application performs cropping operation on the original image according to the coordinate parameters of each bounding box in the entity region bounding box set. Specifically, the starting position of cropping is determined according to the top-left corner coordinates of the bounding box, and the range of the cropping region is determined according to the width and height parameters. The pixel data of the rectangular region is extracted from the original image to generate a small image containing a single entity. After performing the cropping operation on all the bounding boxes of each image in the preliminary search result set, all the small images obtained by cropping are collected to form the entity sub-image set.

[0080] In step S412, the present application performs cropping operation on the original image according to the coordinate parameters of each bounding box in the entity region bounding box set. Specifically, the starting position of cropping is determined according to the top-left corner coordinates of the bounding box, and the range of the cropping region is determined according to the width and height parameters. The pixel data of the rectangular region is extracted from the original image to generate a small image containing a single entity. After performing the cropping operation on all the bounding boxes of each image in the preliminary search result set, all the small images obtained by cropping are collected to form the entity sub-image set.

[0081] In step S413, the present application inputs each entity sub-image in the entity sub-image set into the image encoding branch of the multi-modal embedding model, which is based on the Vision Transformer architecture. Specifically, the image encoding branch first divides the entity sub-image into multiple image blocks, each with a size of 32x32 pixels. Then, each image block is flattened into a one-dimensional vector and position encoding is added. Subsequently, through self-attention calculation and feedforward network transformation of multiple layers of Transformer encoder, the features of all image blocks are aggregated into a 512-dimensional feature vector. After performing the above encoding process on each entity sub-image in the entity sub-image set, all the output feature vectors are finally grouped to form the entity sub-image feature vector set.

[0082] In step S413, the present application inputs each entity sub-image in the entity sub-image set into the image encoding branch of the multi-modal embedding model, which is based on the Vision Transformer architecture. Specifically, the image encoding branch first divides the entity sub-image into multiple image blocks, each with a size of 32x32 pixels. Then, each image block is flattened into a one-dimensional vector and position encoding is added. Subsequently, through self-attention calculation and feedforward network transformation of multiple layers of Transformer encoder, the features of all image blocks are aggregated into a 512-dimensional feature vector. After performing the above encoding process on each entity sub-image in the entity sub-image set, all the output feature vectors are finally grouped to form the entity sub-image feature vector set.

[0083] In step S414, the present application performs similarity retrieval processing on the entity sub-image feature vectors in the entity sub-image feature vector set and the image feature index respectively, to obtain similar image results corresponding to the entity sub-images.

[0084] In step S414, the present application inputs each entity subgraph feature vector in the entity subgraph feature vector set into the image feature index for retrieval. For each entity subgraph feature vector, the present application calculates the cosine similarity between it and all image feature vectors in the index, selects the TopM images with the highest similarity scores, and forms the similar image results corresponding to the entity subgraph. After performing the retrieval operation on all feature vectors in the entity subgraph feature vector set, the present application obtains the similar image results corresponding to multiple entity subgraphs.

[0085] S415, merging the preliminary retrieval result set and the similar image results to obtain a candidate image set.

[0086] In step S415, the present application further includes:

[0087] S4151, recording the similarity scores of multiple images in the preliminary retrieval result set as first type scores, and recording the similarity scores of multiple images in the similar image results corresponding to multiple entity subgraphs as second type scores.

[0088] In step S4151, the present application marks the similarity score of each image in the preliminary retrieval result set as a first type score, which is derived from the cosine similarity calculation between the query feature vector and the image feature vector in the aforementioned step S3. The present application marks the similarity score of each image in the similar image results corresponding to multiple entity subgraphs as a second type score, which is derived from the cosine similarity calculation between the entity subgraph feature vector and the image feature vector in the aforementioned step S414.

[0089] S4152, constructing a candidate image pool, traversing the preliminary retrieval result set and the similar image results corresponding to multiple entity subgraphs, and adding the images and corresponding scores appearing in the candidate image pool.

[0090] Further, in step S4152, the present application aims to create an empty candidate image pool data structure. Specifically, the present application traverses the preliminary retrieval result set, adds each image and its first type score in the preliminary retrieval result set to the candidate image pool, and then traverses the similar image results corresponding to multiple entity subgraphs, adds each image and its second type score in the similar image results to the candidate image pool. If a certain image appears in both the preliminary retrieval result set and the similar image results, the image is associated with both the first type score and the second type score in the candidate image pool.

[0091] S4153, for each image in the candidate image pool, extracting the first type score and the second type score corresponding to the current image, and obtaining the comprehensive score of the current image through weighted average operation.

[0092] Further, in step S4153, the application retrieves all scores associated with each image in the candidate image pool. If the image is only associated with the first type of score, the first type of score is extracted; if the image is only associated with the second type of score, the second type of score is extracted; if the image is associated with both the first type of score and the second type of score, both types of scores are extracted. Finally, the application performs a weighted average operation on the extracted scores, and the calculation formula is to multiply each score by the corresponding weight and sum the results, and the sum of the weights is 1, to obtain the comprehensive score of the image.

[0093] S4154, according to the comprehensive score, the images in the candidate image pool are arranged in descending order, and a preset number of images are selected from the sorting result to form a candidate image set.

[0094] In step S4154, the application arranges all images in the candidate image pool in descending order according to the comprehensive score, and the image with the highest comprehensive score is arranged at the front. The application selects a preset number K of images from the front end of the sorting result to form a candidate image set.

[0095] In step S4, when the complexity label is a complex label, step S4 further includes:

[0096] S421, constructing a prompt word template containing decomposition instructions, filling the query text into the prompt word template, and performing semantic analysis processing on the filled prompt word through a large language model to obtain a plurality of sub-query texts.

[0097] In step S421, the application first constructs a prompt word template, which contains fixed instruction text and query text placeholders. The instruction text is "Please decompose the following long query into several short, independent and core semantic sub-queries. Output the sub-queries directly, separated by semicolons, and query:". Subsequently, the application fills the query text into the placeholder position to form a complete prompt word.

[0098] After obtaining the prompt word, the application inputs the prompt word into the large language model. The model encodes the prompt word based on the Transformer decoder architecture and generates output text word by word through the autoregressive method. Finally, the application analyzes the output text and divides it into multiple sub-query texts according to the semicolon separator.

[0099] S422, performing feature encoding processing, similarity matching processing and multi-hop retrieval processing on the plurality of sub-query texts respectively to obtain a plurality of sub-retrieval result sets corresponding to the plurality of sub-query texts.

[0100] In step S422, the application performs feature coding processing on each subquery text, inputs the subquery text into the text coding branch of the multi-modal embedding model, and outputs the subquery feature vector corresponding to the subquery text. Specifically, the application performs similarity matching processing on each subquery feature vector, inputs the subquery feature vector into the image feature index for retrieval, calculates the cosine similarity and returns the TopK images to form the preliminary retrieval result of the subquery text; subsequently, the application performs multi-hop retrieval processing on the preliminary retrieval result, and the specific process is similar to steps S411 to S415. The entity is positioned using the target detection model, the entity subgraph is cropped, the entity subgraph feature is extracted, the secondary retrieval is performed, the results are merged and sorted, and finally the sub-retrieval result set corresponding to the subquery text is obtained. Finally, the application performs the above processing procedure on all subquery texts in parallel to obtain a plurality of sub-retrieval result sets corresponding to the subquery texts.

[0101] S423, using a text embedding model to vectorize the query text and the plurality of subquery texts respectively, to obtain an original query vector and a plurality of subquery vectors.

[0102] Further, in step S423, the application uses a text embedding model Sentence-BERT to vectorize the query text. The Sentence-BERT is based on the BERT model and can map text into a fixed-dimensional semantic vector through a twin network structure and contrastive learning training. Specifically, the application inputs the query text into the Sentence-BERT model, the model tokenizes and encodes the text, and aggregates the hidden states of all tokens into a 768-dimensional original query vector through average pooling operation. After performing the same vectorization processing on each subquery text, a plurality of subquery vectors are obtained.

[0103] S424, by calculating the semantic similarity between the plurality of subquery vectors and the original query vector, and normalizing the semantic similarity, obtaining the semantic weight corresponding to the plurality of subquery texts.

[0104] In step S424, the application first calculates the cosine similarity between each subquery vector and the original query vector to obtain P similarity values. Subsequently, the P similarity values are subjected to softmax normalization processing, the exponential value of each similarity value is calculated, and then each exponential value is divided by the sum of all exponential values to obtain P normalized values, the sum of which is 1. After calculation, the normalized values are taken as the semantic weights corresponding to the plurality of subquery texts.

[0105] S425, according to the semantic weight, weighting and fusing the sub-retrieval result set corresponding to the plurality of subquery texts to obtain a candidate image set.

[0106] Step S425 further includes:

[0107] S4251. Summarize the sub-retrieval result sets corresponding to multiple sub-query texts to obtain a candidate image pool; S4252. For each candidate image in the candidate image pool, traverse the sub-retrieval result sets corresponding to multiple sub-query texts and record the similarity score when the current candidate image appears in multiple sub-retrieval result sets.

[0108] Further, in steps S4251 to S4252, the present invention first summarizes all images in the sub-retrieval result sets corresponding to multiple sub-query texts to create a candidate image pool, which contains all images appearing in the sub-retrieval result sets. Then, for each candidate image in the candidate image pool, the present invention iterates through the sub-retrieval result sets corresponding to multiple sub-query texts; specifically, for the p-th sub-retrieval result set, the present invention checks whether the current candidate image appears in the result set. If it appears, the similarity score of the candidate image in the result set is recorded; if it does not appear, the score is recorded as 0. After performing the above checking and recording operations on all P sub-retrieval result sets, multiple similarity scores are obtained.

[0109] S4253. Perform a weighted summation operation on the similarity score and the corresponding semantic weight to obtain the total weighted score of the current candidate image; S4254. Sort and filter multiple candidate images in the candidate image pool according to the total weighted score to obtain the candidate image set.

[0110] In step S4253, the present invention multiplies the similarity score of the current candidate image in each sub-retrieval result set with the corresponding semantic weight, and then sums all the products to obtain the total weighted score of the candidate image; subsequently, in step S4254, the present invention calculates the total weighted score of all candidate images in the candidate image pool, sorts them in descending order according to the total weighted score, and selects the TopK images with the highest total weighted scores to form a candidate image set.

[0111] S5. The images in the candidate image set are fused, sorted, and filtered to obtain the final search results.

[0112] In step S5, the present invention sorts the images in the candidate image set according to the comprehensive score or the total weighted score, and selects a preset number of images with the highest scores as the final retrieval results and returns them to the user.

[0113] like Figure 2 As shown, the present invention also provides a multi-level text-to-image search system based on adaptive query difficulty, comprising:

[0114] Evaluation module 100: Used to receive query text input by the user, evaluate the complexity of the query text, and obtain a complexity label;

[0115] Encoding module 200, configured as a multimodal embedding model, is used to encode the query text to obtain a query feature vector;

[0116] Matching module 300: used to perform similarity matching between the query feature vector and the pre-constructed image feature index to obtain a preliminary retrieval result set;

[0117] Enhancement module 400: used to select the corresponding retrieval strategy based on the complexity label to enhance the preliminary retrieval result set and obtain a candidate image set;

[0118] Sorting module 500: used to merge and sort the images in the candidate image set, and filter to obtain the final retrieval result.

[0119] This invention also provides a multi-level text-to-image search device based on adaptive query difficulty, comprising:

[0120] A memory and at least one processor, wherein the memory stores instructions;

[0121] At least one of the processors invokes the instructions in the memory to cause a multi-level text-to-image search device that adapts to query difficulty to perform a multi-level text-to-image search method that adapts to query difficulty as described above.

[0122] The present invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement a multi-level text-to-image search method based on query difficulty adaptation as described in any of the preceding claims.

[0123] This invention significantly improves retrieval accuracy and resource utilization efficiency by adaptively evaluating and selecting strategies based on the complexity of the query text. In simple query scenarios, it provides rapid responses and accurate results, while invoking deeper retrieval mechanisms in complex query scenarios. For queries containing multiple entities, fine-grained entity-level matching of preliminary search results accurately captures local features and entity relationships in images, greatly improving retrieval accuracy for multi-entity queries. For complex long-text queries, semantic decomposition and weighted fusion mechanisms avoid the loss of key information and semantic averaging, resulting in retrieval results that better match user intent.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-level text-to-image search method based on adaptive query difficulty, characterized in that, include: S1. Receive the query text input by the user, evaluate the complexity of the query text, and obtain the complexity label; S2. Use a multimodal embedding model to encode the query text to obtain a query feature vector; S3. Perform similarity matching between the query feature vector and the pre-constructed image feature index to obtain a preliminary retrieval result set; S4. Based on the complexity label, select the corresponding retrieval strategy to enhance the preliminary retrieval result set to obtain a candidate image set; S5. The images in the candidate image set are fused, sorted, and filtered to obtain the final search results.

2. The multi-level text-to-image search method based on adaptive query difficulty as described in claim 1, characterized in that, Step S1 specifically includes: S11. Receive the query text input by the user; S12. The query text is semantically represented by a text encoder to obtain query semantic features; S13. Input the query semantic features into a pre-trained complexity classification model, and perform difficulty discrimination processing on the query semantic features through the classification decision layer to obtain the complexity label.

3. The multi-level text-to-image search method based on adaptive query difficulty as described in claim 1, characterized in that, The complexity labels in step S1 include: simple label, medium label, and complex label.

4. The multi-level text-to-image search method based on adaptive query difficulty as described in claim 3, characterized in that, When the complexity label is a medium label, step S4 further includes: S411. Use an object detection model to perform entity localization processing on multiple images in the preliminary retrieval result set to obtain a set of entity region bounding boxes corresponding to multiple images; S412. Crop multiple images according to the set of entity region bounding boxes to obtain a set of entity sub-images; S413. The multimodal embedding model is used to perform feature extraction processing on multiple entity subgraphs in the entity subgraph set to obtain a set of entity subgraph feature vectors; S414. Perform similarity retrieval processing on the multiple entity subgraph feature vectors in the entity subgraph feature vector set and the image feature index respectively to obtain similar image results corresponding to multiple entity subgraphs; S415. The preliminary search result set and the similar image results are merged to obtain a candidate image set.

5. The multi-level text-to-image search method based on adaptive query difficulty as described in claim 4, characterized in that, Step S415 further includes: S4151. Record the similarity scores of multiple images in the preliminary search result set as the first type of score, and record the similarity scores of multiple images in the similar image results corresponding to multiple entity sub-images as the second type of score. S4152. Construct a candidate image pool by traversing the preliminary retrieval result set and the similar image results corresponding to multiple entity subgraphs, and adding the appearing images and their corresponding scores to the candidate image pool. S4153. For each image in the candidate image pool, extract the first type score and the second type score corresponding to the current image, and obtain the comprehensive score of the current image by weighted average calculation. S4154. Sort the images in the candidate image pool in descending order according to the comprehensive score, and select a preset number of images from the sorting results to form a candidate image set.

6. The multi-level text-to-image search method based on adaptive query difficulty as described in claim 3, characterized in that, In step S4, when the complexity label is a complex label, step S4 further includes: S421. Construct a prompt word template containing decomposition instructions, fill the query text into the prompt word template, and perform semantic parsing processing on the filled prompt words through a large language model to obtain multiple sub-query texts; S422. Perform feature encoding, similarity matching and multi-hop retrieval processing on multiple sub-query texts respectively to obtain the sub-retrieval result set corresponding to multiple sub-query texts; S423. Use a text embedding model to vectorize the query text and multiple subquery texts respectively to obtain the original query vector and multiple subquery vectors; S424. By calculating the semantic similarity between multiple sub-query vectors and the original query vector, and normalizing the semantic similarity, the semantic weights corresponding to the multiple sub-query texts are obtained. S425. Based on semantic weights, the sub-retrieval result sets corresponding to multiple sub-query texts are weighted and fused to obtain a candidate image set.

7. The multi-level text-to-image search method based on adaptive query difficulty as described in claim 6, characterized in that, Step S425 further includes: S4251. Summarize the sub-retrieval result sets corresponding to multiple sub-query texts to obtain a candidate image pool; S4252. For each candidate image in the candidate image pool, traverse the sub-retrieval result sets corresponding to multiple sub-query texts and record the similarity score when the current candidate image appears in multiple sub-retrieval result sets. S4253. Perform a weighted summation operation on the similarity score and the corresponding semantic weight to obtain the total weighted score of the current candidate image; S4254. Sort and filter multiple candidate images in the candidate image pool according to the total weighted score to obtain the candidate image set.

8. A multi-level text-to-image search system based on adaptive query difficulty, characterized in that, include: Evaluation module: Used to receive query text input by the user, evaluate the complexity of the query text, and obtain a complexity label; An encoding module, configured as a multimodal embedding model, is used to encode the query text to obtain a query feature vector; Matching module: used to perform similarity matching between the query feature vector and the pre-built image feature index to obtain a preliminary retrieval result set; Enhancement module: used to select the corresponding retrieval strategy based on the complexity label to enhance the preliminary retrieval result set and obtain a candidate image set; Sorting module: used to merge and sort the images in the candidate image set, and filter to obtain the final search results.

9. A multi-level text-to-image search device based on adaptive query difficulty, characterized in that, include: A memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause a multi-level text-to-image search device that adapts to query difficulty to perform a multi-level text-to-image search method that adapts to query difficulty as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a processor, implement a multi-level text-to-image search method based on query difficulty adaptation as described in any one of claims 1 to 7.

Images