A zero-shot text-to-image retrieval method and system based on scale perception interaction

Abstract

The application discloses a zero-shot text-to-image retrieval method and system based on scale perception interaction, and belongs to the technical fields of artificial intelligence, image retrieval and human-computer interaction. The method comprises the following steps: after an initial text query is acquired, initial retrieval is performed based on a frozen model, a retrieval complexity score is calculated by using similarity entropy and visual diversity, and then a scale perception progressive questioning plan is generated by adaptively allocating global scene-level and local target-level questioning rounds; after a user's answer is acquired to update a dialogue history, an initial query is taken as a semantic anchor point, a sliding window context is selected from the last k rounds of question and answer, a rewritten query is generated, and finally a cross-modal similarity is calculated based on a frozen text encoder to output a ranking result. By quantifying the distinguishing difficulty of a candidate set, the granularity of the question is dynamically adjusted, and discriminative semantic clues are gradually supplemented in multiple rounds of interaction, thereby directly reducing the ambiguity of the candidate image and improving the retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, image retrieval, and human-computer interaction, and in particular to a zero-sample text-to-image retrieval method and system based on scale-aware interaction. Background Technology

[0002] With the development of image acquisition technology, the scale of image data and its application fields continue to grow, placing higher demands on efficient retrieval. Existing single-turn text-to-image retrieval methods struggle to handle complex compositions, dense multi-scale objects, and difficulties in distinguishing similar scenes; a single user query often fails to fully express the global scene and local relationships. Some existing methods focus on retrieval tasks that require both reference images and modified text as input, or rely on trained large models to handle multimodal dialogues, whose applicable conditions differ from zero-shot image retrieval based solely on text queries. Although visual language pre-trained models improve retrieval performance, their fine-tuning depends on extensive annotation and has limited generalization; existing general interactive methods lack joint modeling of the candidate set retrieval state and the multi-scale characteristics of images, making it easy to lose key clues in multiple rounds of interaction.

[0003] Therefore, the key to improving the accuracy and practicality of zero-shot text-to-image retrieval is to adaptively clarify user intent based on initial candidate results without changing the parameters of the underlying retrieval model. Summary of the Invention

[0004] This invention provides a zero-sample text-to-image retrieval method and system based on scale-aware interaction to solve the problems of insufficient semantics in existing single-round text retrieval, lack of state-driven scale modeling in general interactive retrieval, and difficulty in adapting frozen image-text retrieval models to long dialogue inputs.

[0005] In a first aspect, embodiments of the present invention provide a zero-sample text-to-image retrieval method based on scale-aware interaction, comprising:

[0006] An initial text query is obtained. Based on a visual language pre-trained retrieval model with frozen parameters, cross-modal similarity calculation is performed between the initial text query and images in the image library to obtain a candidate image set, the similarity score and visual features corresponding to each candidate image in the candidate image set.

[0007] The retrieval uncertainty is determined based on the similarity score, the candidate set diversity is determined based on the visual features, and the retrieval complexity score is calculated based on the retrieval uncertainty and the candidate set diversity.

[0008] Based on the retrieval complexity score, determine the global scenario-level questioning rounds and the local target-level questioning rounds, and generate a scale-aware progressive questioning plan;

[0009] The system generates clarification questions according to the scale-aware progressive questioning plan, obtains answers to the clarification questions, and updates the dialogue history based on the clarification questions and the answers.

[0010] Using the initial text query as the semantic anchor, the most recent k rounds of question and answer are selected from the dialogue history to form a sliding window context, and a rewritten query is generated based on the semantic anchor and the sliding window context;

[0011] A text encoder based on parameter freezing calculates the cross-modal similarity between the rewritten query and the image to be retrieved, and outputs the image retrieval ranking results based on the cross-modal similarity.

[0012] Optionally, an initial text query is obtained, and a visual language pre-trained retrieval model with frozen parameters is used to perform cross-modal similarity calculations between the initial text query and images in the image database to obtain a candidate image set, the similarity score corresponding to each candidate image in the candidate image set, and visual features, including:

[0013] Obtain the initial text query, call the visual language pre-trained retrieval model with frozen parameters, encode the images in the image library to obtain image features, and encode the initial text query to obtain initial text features;

[0014] Calculate the similarity between the initial text features and the image features;

[0015] According to the order of similarity from high to low, a preset number of images are selected as the candidate image set, and the similarity score and visual features corresponding to each candidate image in the candidate image set are retained.

[0016] Optionally, the retrieval uncertainty is determined based on the similarity score, the candidate set diversity is determined based on the visual features, and a retrieval complexity score is calculated based on the retrieval uncertainty and the candidate set diversity, including:

[0017] Temperature normalization is performed on the similarity scores corresponding to the candidate image set to obtain the candidate image probability distribution;

[0018] The normalized entropy is calculated based on the probability distribution of the candidate images, and the normalized entropy is used as the retrieval uncertainty.

[0019] The visual features of each candidate image in the candidate image set are normalized.

[0020] Calculate the feature similarity between the visual features of any two candidate images, and determine the diversity of the candidate set based on the feature similarity;

[0021] The retrieval complexity score is obtained by weighted summation of the retrieval uncertainty and the diversity of the candidate set.

[0022] Optionally, based on the retrieval complexity score, determine the global scene-level questioning rounds and the local target-level questioning rounds, and generate a scale-aware progressive questioning plan, including:

[0023] Get the total number of preset interaction rounds;

[0024] The global scenario-level question rounds are determined based on the retrieval complexity score, and the local target-level question rounds are determined based on the preset total number of interaction rounds and the global scenario-level question rounds.

[0025] A scale-aware progressive questioning plan is generated based on the global scenario-level questioning rounds and the local target-level questioning rounds.

[0026] Optionally, a retrieval clarification question is generated according to the scale-aware progressive questioning plan, an answer is obtained for the retrieval clarification question, and the dialogue history is updated based on the retrieval clarification question and the answer, including:

[0027] According to the scale-aware progressive questioning plan, scale control instructions are generated based on the questioning scale corresponding to the current round.

[0028] Based on the scale control instructions and the obtained dialogue history, the current round of retrieval clarification questions are generated;

[0029] Obtain the user's natural language answer to the current round of clarification question, combine the current round of clarification question with the natural language answer to form the current round of question and answer record, and append the current round of question and answer record to the dialogue history.

[0030] Optionally, using the initial text query as a semantic anchor, the most recent k rounds of question-and-answer sessions are selected from the dialogue history to form a sliding window context, and a rewritten query is generated based on the semantic anchor and the sliding window context, including:

[0031] Using the initial text query as a semantic anchor, the most recent k rounds of question and answer are selected from the dialogue history to form a sliding window context;

[0032] Based on the user's initial text query and the context of the sliding window, a descriptive short text is generated as the rewritten query.

[0033] Optionally, a parameter-frozen text encoder calculates the cross-modal similarity between the rewritten query and the image to be retrieved, and outputs the image retrieval ranking results based on the cross-modal similarity, including:

[0034] The rewritten query is text-encoded to obtain rewritten text features, and the pre-calculated features of the image to be retrieved are read.

[0035] Calculate the similarity between the rewritten text features and the image features to be retrieved;

[0036] The images to be retrieved are sorted from high to low according to the similarity, and the sorted image retrieval results are output.

[0037] Secondly, embodiments of the present invention provide a zero-shot text-to-image retrieval system based on scale-aware interaction, the system being used to execute the zero-shot text-to-image retrieval method based on scale-aware interaction described in any embodiment of the present invention, including:

[0038] The acquisition module is used to acquire an initial text query, and based on a visual language pre-trained retrieval model with frozen parameters, to perform cross-modal similarity calculation between the initial text query and images in the image library to obtain a candidate image set, the similarity score and visual features corresponding to each candidate image in the candidate image set;

[0039] The calculation module is used to determine the retrieval uncertainty based on the similarity score, determine the candidate set diversity based on the visual features, and calculate the retrieval complexity score based on the retrieval uncertainty and the candidate set diversity.

[0040] The determination module is used to determine the global scene-level questioning rounds and the local target-level questioning rounds based on the retrieval complexity score, and generate a scale-aware progressive questioning plan;

[0041] The update module is used to generate retrieval clarification questions according to the scale-aware progressive questioning plan, obtain answers to the retrieval clarification questions, and update the dialogue history based on the retrieval clarification questions and the answers.

[0042] The generation module is used to select the most recent k rounds of question and answer from the dialogue history to form a sliding window context, using the initial text query as the semantic anchor point, and to generate a rewritten query based on the semantic anchor point and the sliding window context.

[0043] The output module is used to calculate the cross-modal similarity between the rewritten query and the image to be retrieved based on the parameter-frozen text encoder, and output the image retrieval ranking results according to the cross-modal similarity.

[0044] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the scale-aware interaction-based zero-sample text-to-image retrieval method according to any embodiment of the present invention.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the zero-sample text-to-image retrieval method based on scale-aware interaction as described in any embodiment of the present invention.

[0046] This invention obtains an initial text query and performs an initial retrieval based on a frozen model. It then calculates the retrieval complexity score using similarity entropy and visual diversity, adaptively allocating global scene-level and local target-level question rounds to generate a scale-aware progressive questioning plan. After obtaining user responses to update the dialogue history, it uses the initial query as a semantic anchor, selects the most recent k rounds of question-and-answer to form a sliding window context, and generates a rewritten query. Finally, it calculates cross-modal similarity based on a frozen text encoder and outputs the ranking results. By quantifying the distinguishing difficulty of the candidate set and dynamically adjusting the coarseness and fineness of the questions, it gradually supplements discriminative semantic cues in multiple rounds of interaction, directly reducing the ambiguity of candidate images and improving retrieval accuracy. This invention solves the problems of insufficient semantics in existing single-round retrieval, lack of state-driven scale modeling in general interactive retrieval, and difficulty in adapting frozen models to long dialogue inputs. This invention achieves adaptive and progressive interaction optimization under zero-sample conditions without fine-tuning the target dataset, significantly improving the robustness and practicality of retrieval while maintaining good generalization ability and stable retrieval performance.

[0047] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1A flowchart of a zero-sample text-to-image retrieval method based on scale-aware interaction provided in Embodiment 1 of the present invention;

[0050] Figure 2 A schematic diagram illustrating the interactive process of a scale-aware interactive zero-sample text-to-image retrieval method provided in Embodiment 2 of the present invention;

[0051] Figure 3 This is a framework diagram of a zero-sample text-to-image retrieval system based on scale-aware interaction provided in Embodiment 3 of the present invention;

[0052] Figure 4 This is a detailed model framework diagram of a scale-aware interactive zero-shot text-to-image retrieval system provided in Embodiment 3 of the present invention;

[0053] Figure 5 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] Example 1:

[0057] Figure 1 This is a flowchart of a scale-aware interactive zero-shot text-to-image retrieval method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where a target image is searched from an image database using natural language descriptions. This method can be executed by a scale-aware interactive zero-shot text-to-image retrieval system. Figure 1 As shown, the method includes:

[0058] S110. Obtain the initial text query, and perform cross-modal similarity calculation between the initial text query and the images in the image library based on the visual language pre-trained retrieval model with parameter freezing, to obtain a candidate image set, the similarity score and visual features corresponding to each candidate image in the candidate image set.

[0059] S120. Determine the retrieval uncertainty based on the similarity score, determine the candidate set diversity based on the visual features, and calculate the retrieval complexity score based on the retrieval uncertainty and the candidate set diversity.

[0060] S130. Based on the retrieval complexity score, determine the global scene-level questioning rounds and the local target-level questioning rounds, and generate a scale-aware progressive questioning plan.

[0061] S140. Generate retrieval clarification questions according to the scale-aware progressive questioning plan, obtain answers to the retrieval clarification questions, and update the dialogue history based on the retrieval clarification questions and the answers.

[0062] S150. Using the initial text query as the semantic anchor, select the most recent k rounds of question and answer from the dialogue history to form a sliding window context, and generate a rewritten query based on the semantic anchor and the sliding window context.

[0063] S160. A text encoder based on parameter freezing calculates the cross-modal similarity between the rewritten query and the image to be retrieved, and outputs the image retrieval ranking result based on the cross-modal similarity.

[0064] In this embodiment, after obtaining the initial text query, an initial retrieval is performed based on a frozen model. Similarity entropy and visual diversity are used to calculate the retrieval complexity score, thereby adaptively allocating global scene-level and local target-level question rounds to generate a scale-aware progressive questioning plan. After obtaining user-answered dialogue history updates, the initial query is used as a semantic anchor point. The most recent k rounds of question-and-answer sessions are selected to form a sliding window context and a rewritten query is generated. Finally, cross-modal similarity is calculated based on the frozen text encoder to output the ranking results. By quantifying the distinguishing difficulty of the candidate set, the coarseness and fineness of the questions are dynamically adjusted. Discriminative semantic cues are gradually supplemented in multiple rounds of interaction, directly reducing the ambiguity of candidate images and improving retrieval accuracy. This invention solves the problems of insufficient semantics in existing single-round retrieval, lack of state-driven scale modeling in general interactive retrieval, and difficulty in adapting frozen models to long dialogue inputs. This invention achieves adaptive and progressive interaction optimization under zero-sample conditions without fine-tuning the target dataset, significantly improving the robustness and practicality of retrieval while maintaining good generalization ability and stable retrieval performance.

[0065] Example 2:

[0066] The technical solution in this embodiment is a further refinement based on the above embodiments.

[0067] In step S110, an initial text query is obtained. Based on a parameter-frozen visual language pre-trained retrieval model, cross-modal similarity calculation is performed between the initial text query and images in the image library to obtain a candidate image set, the similarity score corresponding to each candidate image in the candidate image set, and visual features, including:

[0068] Obtain the initial text query, call the visual language pre-trained retrieval model with frozen parameters, encode the images in the image library to obtain image features, and encode the initial text query to obtain initial text features;

[0069] The initial text query refers to the text entered by the user initially to describe the image they want to find. For example, a user entering "a white puppy running on the grass" is an initial text query, serving as the starting point and semantic anchor for the entire retrieval process. The parameter-frozen visual language pre-trained retrieval model refers to a deep learning model that has been trained on large-scale image and text data and whose parameters are no longer updated; it can simultaneously understand images and text and map them into a common mathematical space. "Frozen" means that no parameters of the model will be adjusted during subsequent retrieval processes, ensuring its stability and zero-sample capability. The image library refers to the collection of all images to be retrieved; the goal of this invention is to find one or more images from this image library that best match the user's text query. Image features refer to a high-dimensional vector (array of numbers) output after an image is input into the model's image encoder. This vector contains visual information from the image, such as color, shape, texture, objects, and scenes. Initial text features refer to a high-dimensional vector output after the user's initial text query is input into the model's text encoder; this vector represents the semantic information of the text.

[0070] Specifically, the first step is to establish an image library. And receive the initial query input from the user. The retrieval input of this invention is a text query. Images in the image library serve as the objects to be retrieved and candidate results in the retrieval state estimation. A combination of reference images and modified text is not a necessary condition. The image library can be applied to general imagery, e-commerce imagery, intelligent monitoring, industrial quality inspection, remote sensing interpretation, and other application scenarios. It is particularly suitable for image sets with multi-scale target distribution, complex spatial layout, and clear scene semantic hierarchy.

[0071] The system receives text strings input by the user from the user interface. For example, a red car is parked in the snow. [The string is used to represent this.] The text encoder is fed into the visual language pre-trained retrieval model with its parameters frozen. The encoder analyzes the semantics, syntactic structure, and context of each word, ultimately outputting a high-dimensional vector representing the entire query, i.e., the initial text features. .

[0072] For each image in image library I All of them are fed into the image encoder of the same visual language pre-trained retrieval model with frozen parameters. The encoder (usually a ViT, or Vision Transformer) segments the image into small blocks, analyzes the relationships between the blocks, and finally outputs a high-dimensional vector representing the content of the entire image, i.e., the image features. This step iterates through the entire image database, generating a unique image feature vector for each image in the database. In practical applications, for efficiency, these image features are usually pre-calculated and stored, so they do not need to be recalculated for each retrieval.

[0073] Calculate the similarity between the initial text features and the image features;

[0074] Similarity refers to the degree of matching between image features and text features in a shared semantic space. It is usually represented by a numerical value, such as a score between 0 and 1. The higher the score, the more relevant the image and text description are.

[0075] Specifically, the obtained initial text features Features of each image in the image library Perform a comparison. Calculate the cosine similarity:

[0076]

[0077] This formula calculates the cosine of the angle between two vectors. A larger value indicates more consistent directions and closer semantics. (Each image) They will all get a similarity score. . It is the text feature vector of the t-th round of rewriting query. This represents the L2 norm of a vector, which is its Euclidean length.

[0078] According to the order of similarity from high to low, a preset number of images are selected as the candidate image set, and the similarity score and visual features corresponding to each candidate image in the candidate image set are retained.

[0079] Specifically, all images are sorted according to their similarity scores. Sort the images from highest to lowest. Take the top C images to form a candidate image set. The value C is a preset parameter, such as 50 or 100. The system not only records the C images themselves, but also retains two corresponding key pieces of information:

[0080] Similarity score (i=1...C), used to calculate the uncertainty of subsequent retrieval.

[0081] Visual features (i=1...C), used for subsequent calculation of the diversity of the candidate set.

[0082] In this embodiment, the image library is encoded by calling a pre-trained model with frozen parameters, the similarity between the initial text and images is calculated, and a Top-C candidate set is selected, while retaining the similarity score and visual features. The frozen model can quickly obtain high-quality candidates without fine-tuning, directly providing necessary state information for subsequent complexity estimation and interactive questioning. The retention of the candidate set's scores and features allows for adaptive adjustment of the questioning strategy based on this information, thus laying a solid foundation for improving the accuracy and stability of zero-shot retrieval.

[0083] In step S120, retrieval uncertainty is determined based on the similarity score, candidate set diversity is determined based on the visual features, and a retrieval complexity score is calculated based on the retrieval uncertainty and the candidate set diversity, including:

[0084] Temperature normalization is performed on the similarity scores corresponding to the candidate image set to obtain the candidate image probability distribution;

[0085] Temperature normalization refers to using the Softmax function to convert a set of scores into a probability distribution, where the temperature parameter τ controls the degree of concentration of the distribution. The candidate image probability distribution refers to the probability values ​​obtained after processing the similarity scores through temperature-normalized Softmax, representing the likelihood that each candidate image is the target image.

[0086] Specifically, the similarity score of the Top-C candidate images , ,..., Perform a temperature-controlled Softmax transform:

[0087] ;

[0088] Where τ is a preset temperature parameter, such as τ = 0.07. The output is a set of non-negative probability values ​​that sum to 1. i represents the index of the candidate image currently being calculated, and j is used inside the summation symbol (∑) as an index variable for traversing all candidate images. This means starting from the first candidate image and continuing to the Cth candidate image, summing up the relevant values ​​of all images.

[0089] The normalized entropy is calculated based on the probability distribution of the candidate images, and the normalized entropy is used as the retrieval uncertainty.

[0090] Normalized entropy refers to the information entropy calculated from the probability distribution, then normalized by the maximum possible entropy logC, resulting in u. Its value ranges from [0,1]. A larger value indicates that the candidate image scores are closer and the discrimination is lower, suggesting that the initial query has a weaker ability to distinguish candidate images, requiring further discriminative clues through subsequent interactions. Retrieval uncertainty, i.e., normalized entropy u, is used to measure the current query's ability to distinguish candidate images.

[0091] Specifically, first calculate the information entropy. Then normalize by dividing by the maximum possible entropy logC:

[0092] ;

[0093] Among them, when all When the values ​​are equal (uniformly distributed), u=1, indicating that the search results are extremely uncertain; when a certain... When u = 1 and all others are 0, u = 0, indicating complete certainty. u represents the retrieval uncertainty.

[0094] The visual features of each candidate image in the candidate image set are normalized.

[0095] Normalization can refer to dividing each visual feature vector by its own L2 norm to make its length 1, so that the inner product can be used directly when calculating cosine similarity later.

[0096] Specifically, for each visual feature vector Perform L2 normalization:

[0097] ;

[0098] in, This represents calculating the length of the i-th image feature vector. In deep learning, feature vectors are typically L2 normalized, i.e., divided by their own length to make their length equal to 1, thus obtaining... =1. After normalization, the dot product of two vectors is equal to their cosine similarity.

[0099] Calculate the feature similarity between the visual features of any two candidate images, and determine the diversity of the candidate set based on the feature similarity;

[0100] Feature similarity refers to the cosine similarity between two visual feature vectors, used to measure the visual proximity of two images. Candidate set diversity describes the differences in visual structure within the candidate set; a higher value indicates greater visual differences and higher diversity among candidate images. This means the candidate set may contain multiple different categories of scenes, object combinations, or spatial layouts.

[0101] Specifically, for the normalized feature vectors Calculate the cosine similarity of all unordered pairs (i,j). Then calculate the average similarity, and subtract that average from 1 to get the diversity:

[0102] ;

[0103] In this context, if all candidate images are very similar, such as images consisting entirely of red cars, the mean cosine similarity is high, and d approaches 0; if candidate images contain various different scenes, such as cars, beaches, and cities, the mean similarity is low, and d approaches 1. d represents the diversity of the candidate set.

[0104] The retrieval complexity score is obtained by weighted summation of the retrieval uncertainty and the diversity of the candidate set.

[0105] The retrieval complexity score is used to guide the allocation of subsequent question rounds.

[0106] Specifically, two weighting coefficients, α and β, are used for linear weighting, where α and β are both positive numbers and usually satisfy α+β=1 or are adjusted as needed.

[0107] γ = αu + βd;

[0108] Here, γ represents the retrieval complexity score. When candidate image scores are similar (large u) and visual diversity is high (large d), γ is larger, indicating that the current retrieval task is difficult and more global scene-level questions are needed to differentiate them; conversely, when u is small and d is small, γ is smaller, and it is possible to get to the local detail questions more quickly.

[0109] In this embodiment, the similarity scores of candidate images are normalized by temperature and the normalized entropy is calculated as the retrieval uncertainty. At the same time, the visual features are normalized and pairwise similarities are calculated to obtain the diversity of the candidate set. The two are then weighted and fused into a retrieval complexity score. This quantifies the difficulty of distinguishing the current candidate set and provides a reliable state basis for the subsequent adaptive allocation of global and local question rounds. It can dynamically perceive the retrieval complexity, thereby guiding the questioning process more efficiently, effectively avoiding invalid questions, and significantly improving the accuracy and interaction efficiency of zero-shot retrieval.

[0110] In step S130, based on the retrieval complexity score, the global scene-level questioning rounds and the local target-level questioning rounds are determined, and a scale-aware progressive questioning plan is generated, including:

[0111] Get the total number of preset interaction rounds;

[0112] The preset total number of interaction rounds is the maximum number of question rounds T set in advance. For example, T=5 means that a maximum of 5 rounds of human-computer interaction will be performed.

[0113] Specifically, an integer T can be read from the system configuration or user settings, for example, T=5. This value is usually specified by the developer or user during method initialization, representing the maximum number of questions that can be asked during the entire interaction process.

[0114] The global scenario-level question rounds are determined based on the retrieval complexity score, and the local target-level question rounds are determined based on the preset total number of interaction rounds and the global scenario-level question rounds.

[0115] Among them, the global scene-level question rounds can refer to the number of rounds assigned to the overall scene category, the layout of main objects, and the semantic category of core areas, denoted as . Local goal-level questioning rounds can refer to the number of rounds assigned to fine-grained goal categories, attributes, quantities, spatial locations, and boundary relationship questions, denoted as... When the retrieval complexity score meets the higher complexity condition, the proportion of global scenario-level question rounds is increased; when the retrieval complexity score meets the lower complexity condition, the proportion of local target-level question rounds is increased. The higher complexity condition can refer to a situation where γ is greater than a certain threshold, indicating that the current retrieval is difficult and requires more global clarification. The lower complexity condition can refer to a situation where γ is less than a certain threshold, indicating that the current retrieval is relatively simple and can proceed more quickly to local details.

[0116] Specifically, calculate the total number of rounds of questioning. :

[0117] ;

[0118] Here, a and b are preset linear coefficients, for example, a=1, b=2, used to map the complexity score γ to the number of rounds. γ is the retrieval complexity score, typically ranging from 0 to 1. `clip` is a clipping function used to limit the global range of query rounds, ensuring it is not less than [value missing]. For example, 1, and not greater than For example, T-1, to avoid meaningless values.

[0119] Calculate the number of rounds of local questioning =T- That is, subtract the total number of rounds from the total number of rounds, and use the remainder for local questioning.

[0120] When γ is large (e.g., γ > 0.7), a + bγ increases, leading to Approaching the upper limit This increases the proportion of questions asked globally. Because the candidate set is difficult to distinguish, it is necessary to clarify it at the level of the larger scenario first.

[0121] When γ is small (e.g., γ < 0.3), a + bγ becomes smaller, leading to Approaching the lower limit This increases the proportion of questions asked in specific areas. Because the candidate set is already relatively clear, we can focus directly on the details.

[0122] A scale-aware progressive questioning plan is generated based on the global scenario-level questioning rounds and the local target-level questioning rounds.

[0123] Among them, the scale-aware progressive questioning plan can refer to an ordered sequence of questions, the preceding... The wheel is of GLOBAL scale, and the rear... The wheel is at the LOCAL scale. The GLOBAL scale corresponds to the overall scene category, the layout of major objects, and the semantics of the core region; the LOCAL scale corresponds to the fine-grained object category, attributes, quantity, spatial location, boundary relationships, and local structural details.

[0124] Specifically, create a list of length T, the first T... Each element is assigned the value "GLOBAL", then... Each element is assigned the value "LOCAL".

[0125] ;

[0126] This list is the question plan. The system will ask questions strictly according to the order of this list:

[0127] Round 1 to Round 2 Each round asks questions about the overall scene, the main object, and the spatial structure (global scale). From round +1 to round T, each round asks questions about fine-grained targets, attributes, quantities, and positional relationships (local scale). The order of this list reflects a cognitive pattern from coarse to fine: first clarify the specific scenario, then ask specific details. Furthermore, the round allocation is adaptive, dynamically adjusting based on the current search difficulty.

[0128] In this embodiment, the questioning rounds at the global scene level and the local target level are adaptively determined based on the retrieval complexity score, and a progressive questioning plan from coarse to fine is generated. This allows the questioning order to be dynamically adjusted according to the difficulty of distinguishing candidate sets. When the complexity is high, the global scene is clarified first, and when the complexity is low, the local details are explored in depth. This directly improves the targeting and efficiency of the interaction. It avoids ambiguity caused by blind questioning and premature refinement, and further enhances the accuracy and stability of zero-shot retrieval in multi-scale scenarios.

[0129] In step S140, a retrieval clarification question is generated according to the scale-aware progressive questioning plan, an answer to the retrieval clarification question is obtained, and the dialogue history is updated based on the retrieval clarification question and the answer, including:

[0130] According to the scale-aware progressive questioning plan, scale control instructions are generated based on the questioning scale corresponding to the current round.

[0131] The question scale corresponding to the current round can refer to the scale type specified in the t-th round of the scale-aware progressive questioning plan, i.e. The value can be either GLOBAL or LOCAL. Scale control instructions can refer to a natural language instruction used to guide large language models (such as GPT, DeepSeek, LLaMA, etc.) to ask questions at the current scale. For example, the GLOBAL instruction asks for the overall scene category and the layout of major objects; the LOCAL instruction asks for the attributes, quantity, and positional relationships of fine-grained objects.

[0132] Specifically, from the previously generated question plan Extract the scale label of round t. The system internally maintains two predefined instruction templates: GLOBAL instructions, such as asking for the overall scene category, main object layout, and core region semantics to avoid prematurely delving into local details; and LOCAL instructions, such as asking for questions about fine-grained object categories, attributes, quantities, spatial locations, and boundary relationships to supplement discriminative information missed in the initial query. The value corresponds to the instruction text, denoted as g( ).

[0133] For example, if t=1 and =GLOBAL, then g( = "Please inquire about the overall scene category..."; if t=4 and =LOCAL, then g( = "Please focus on fine-grained goals...".

[0134] Based on the scale control instructions and the obtained dialogue history, the current round of retrieval clarification questions are generated;

[0135] The obtained dialogue history can refer to all question and answer records up to the previous round. The current round of retrieval clarification questions can refer to the natural language questions generated by the large language model in round t, aimed at eliminating ambiguity. It is used to obtain discriminative semantic cues that can distinguish the current set of candidate images.

[0136] Specifically, the following information is pieced together to form a prompt for a large language model: the obtained dialogue history. This includes the initial query and all previous questions and answers. Scale control command g( This constrains the granularity of the question. The large language model is then invoked, and the above prompts are input... , obtain the output text = ( g( Outputting natural language problems For example, is the scene you are looking for indoors or outdoors? (GLOBAL) Or what color is the main object in the picture? (LOCAL)

[0137] Obtain the user's natural language answer to the current round of clarification question, combine the current round of clarification question with the natural language answer to form the current round of question and answer record, and append the current round of question and answer record to the dialogue history;

[0138] Natural language answers can refer to the natural language responses given by users to the current question. The current round of question and answer records can refer to a tuple consisting of the question and answer from the current round. , Conversation history can refer to all accumulated question-and-answer records. .

[0139] Specifically, the system will generate The information is displayed to the user, who then enters a natural language response after reading it. For example, yes, I'm looking for seashells on the beach. In offline verification, simulated answers can be generated using pre-set human responses or pre-trained visual language models. = ( , This is used to verify the interaction process and retrieval results. In actual retrieval, the input remains the user's initial text query and subsequent responses. For example, a pre-set response can be a set of fixed question-and-answer pairs manually written by researchers to simulate user responses. For instance, for the question "Is the scene you are looking for indoors or outdoors?", the pre-set answer is "indoors".

[0140] The advantages are complete controllability and suitability for testing specific logical branches. The disadvantages are a lack of flexibility and inability to cover all possible dialogue paths. Pre-trained visual language models (VLMs) can automatically generate simulated responses using a pre-trained visual language model (such as BLIP, GPT-4V, LLaVA, etc.). This model receives a target image. and current issues As input, output a natural language response. .

[0141] = ( , );

[0142] in, This represents a pre-trained visual language model. The target image is the image the user is actually looking for, which is the known ground truth in offline experiments. The current round of the system raises the retrieval clarification question. The model answers questions based on the content of the target image. Its advantages include automation, scalability, the ability to simulate diverse responses, and closer resemblance to real user behavior. The disadvantage is that VLM's answers may not be perfect, but they are sufficient to verify the effectiveness of the interaction flow.

[0143] Package the questions and answers into a structured tuple, append it to the conversation history, and execute... This allows the search intent to gradually transform from a vague textual description into a complex semantic description containing scene, target, and layout information throughout multiple rounds of interaction. The initial query... It has always been preserved as the first record in the history of the dialogue.

[0144] For example, suppose t=1: initial dialogue history =[ ],in, =A photo of a beach. Round 1: System generated. =Are you looking for a tropical or a cool-weather beach? (User's answer) =Tropical style. Updated dialogue history =[ , ( , )].

[0145] In this embodiment, following a scale-aware progressive questioning plan, scale control instructions are generated based on the current round's scale, and clarification questions are generated by combining the dialogue history. User responses are then appended to the dialogue history. This achieves dynamic generation of targeted questions matching the current scale across multiple rounds of interaction, gradually supplementing discriminative semantic cues and directly reducing the ambiguity of candidate images. Furthermore, it adaptively adjusts the content and order of questions based on the retrieval status, avoiding irrelevant questions and further improving the accuracy and efficiency of zero-shot retrieval.

[0146] In step S150, using the initial text query as the semantic anchor, the most recent k rounds of question-and-answer sessions are selected from the dialogue history to form a sliding window context, and a rewritten query is generated based on the semantic anchor and the sliding window context, including:

[0147] Using the initial text query as a semantic anchor, the most recent k rounds of question and answer are selected from the dialogue history to form a sliding window context;

[0148] Based on the user's initial text query and the context of the sliding window, a descriptive short text is generated as the rewritten query.

[0149] In this context, the sliding window context can refer to a fragment of context consisting of the most recent k rounds of question and answer responses. The window slides as the rounds progress, discarding older questions and answers and adding new ones. The rewritten query can refer to a short text description generated by a large language model that integrates the original intent and the latest clues. This is used to replace lengthy conversation histories as retrieval input. Descriptive short text can refer to a natural language expression, such as "a picture of a tropical beach with white seashells on the sand," rather than a question-and-answer format like a conversation.

[0150] Specifically, always use the initial query Place this at the very beginning as the core information that cannot be discarded. Assume we are currently in round t, and there have been t rounds of questions and answers in the dialogue history. Set the sliding window size to k (e.g., k=3). Select the most recent k rounds: if t≤k, then select all existing question-and-answer pairs. If t > k, then only the k most recent rounds are selected. Semantic anchors The text is concatenated with the question-and-answer pairs in the window to form a text block, which serves as the input for the subsequent large language model.

[0151] Because CLIP-style or similar frozen text retrieval models are better suited to short descriptive text, directly inputting the complete dialogue history can lead to problems such as excessive length, semantic dilution, intent drift, or truncation. This invention models the rewriting process as a retrieval-compatible context selection problem for frozen text encoders, rather than a typical dialogue summarization problem, by constructing an LLM hint. The LLM hint typically includes role instructions, indicating that you are a retrieval query rewriting assistant. Based on the user's original query and recent conversations, generate a short, descriptive retrieval query.

[0152] By prompting the project, the LLM is forced to generate descriptive short text as the rewritten query. = ( , Short text can refer to text that is typically no longer than 77 tokens, which is the maximum input length for CLIP. It should be descriptive, using declarative sentences rather than interrogative sentences, such as a photo of a tropical beach with white seashells and coconut trees. It should be retrieval-compatible, avoiding redundant vocabulary and focusing on visually identifiable entities and attributes.

[0153] In this embodiment, the initial text query serves as the semantic anchor. The most recent k rounds of question-and-answer dialogues are selected from the dialogue history to form a sliding window context, and a descriptive short text is generated as the rewritten query. By preserving the original retrieval intent and highlighting recently added discriminative clues, the semantic dilution or truncation by the frozen text encoder caused by an excessively long complete dialogue history is directly avoided. The generated short text query is more suitable for the input requirements of the frozen retrieval model, further improving the accuracy and stability of zero-shot retrieval.

[0154] In step S160, based on the parameter-frozen text encoder, the cross-modal similarity between the rewritten query and the image to be retrieved is calculated, and the image retrieval ranking result is output according to the cross-modal similarity, including:

[0155] The rewritten query is text-encoded to obtain rewritten text features, and the pre-calculated features of the image to be retrieved are read.

[0156] Calculate the similarity between the rewritten text features and the image features to be retrieved;

[0157] The images to be retrieved are sorted from high to low according to the similarity, and the sorted image retrieval results are output.

[0158] The image to be retrieved is either all images in the image library or candidate images determined based on the results of the previous retrieval, such as the Top-C set from the previous round. The parameter-frozen text encoder is... , which is the part dedicated to processing text in the pre-trained vision-language model, and its parameters remain unchanged during the retrieval process. The rewritten text feature may refer to the high-dimensional vector obtained after inputting the rewritten query into the text encoder . The cross-modal similarity may refer to the cosine similarity between the rewritten text feature and the image feature. The image retrieval ranking result may refer to an image list arranged in descending order of similarity, that is, the final retrieval output or the result of the final round .

[0159] Specifically, the rewritten query of the current round is input into the frozen text encoder to obtain a text feature vector: , this encoder is the same model as the one used in the initial retrieval, with completely identical parameters that are not updated. In the initialization stage, the system has converted all images in the image library I into image feature vectors through the frozen image encoder and stored them in the memory or database. In each round of retrieval, it is only necessary to read these precomputed features without running the image encoder again, which greatly improves efficiency.

[0160] For each to-be-retrieved image , calculate its image feature and the text feature cosine similarity:

[0161]

[0162] This score reflects the matching degree between the image and the rewritten query in the semantic space. For all to-be-retrieved images, according to their similarity scores perform descending sorting to obtain a ranked list: ;

[0163] where I is the to-be-retrieved image set of the current round.

[0164] If it is the final round (t=T), the final ranking result is output as the output of the entire interactive retrieval. If it is an intermediate round (t<T), this ranking result can be used to update the candidate image set for the next round of questioning.

[0165] In this embodiment, the rewritten query is encoded based on the parameter-frozen text encoder, the similarity with the precomputed image features is calculated, and the retrieval result is output by sorting according to the scores. By using the frozen encoder, cross-modal matching can be efficiently completed without fine-tuning, which directly avoids the computational overhead caused by repeatedly extracting image features. While maintaining the zero-shot adaptation capability, the retrieval speed is greatly improved, and the practicability and deployability of the method are further enhanced.

[0166] In a specific embodiment, the specific algorithm steps for zero-sample text-to-image retrieval based on scale-aware interaction can be as follows:

[0167] Input: Image library User initial text query Total number of interaction rounds Candidate set size Sliding window size N refers to the integer used to define the size of the image library, i.e., the size of the image library. There are a total of N images available for retrieval.

[0168] Output: Image retrieval ranking results .

[0169] 1. Initialization: Load the frozen visual language pre-trained retrieval model, where the image encoder is denoted as... The text encoder is denoted as Setting up a large language model ; Define the visual language model .

[0170] 2: Using image encoders respectively and text encoder right Each candidate image in the dataset is encoded to obtain... and .

[0171] 3: Calculation Cross-modal similarity with each candidate image And select the Top-scoring candidate. Candidate image set .

[0172] 4: According to Top- Similarity score calculation of candidate images is an uncertainty index for retrieval. And calculate the candidate set diversity index based on the visual features of the candidate images. .

[0173] 5: Will and Weighted fusion yields the retrieval complexity score. .

[0174] 6: According to Calculate the number of global scenario-level question rounds Number of rounds of questioning at the local target level Generate a scale-aware progressive questioning plan .

[0175] 7: Initialize the dialogue history .

[0176] 8: for to do

[0177] 8.1 According to the questioning scale , generate a scale control instruction . 8.2 Input the previous round of dialogue history and the scale control instruction , to obtain the current round of retrieval clarification question . 8.3 According to the target image or the user's retrieval intent, call a response function to generate the current round of answer , the answer includes two cases: if in a real interaction scenario, it is a user natural language response; if in an offline simulation verification scenario, it is a simulated answer generated based on the target image and the question , that is . 8.4 Append the current question and answer to the dialogue history to obtain . 8.5 Take as the semantic anchor point, select the recent rounds of question and answer to form a sliding window context , if the current round t<k, select all t rounds of historical question and answer. Input the context and the initial query to obtain the rewritten query of the current round .

[0178] 8.6 Use the text encoder to encode the rewritten query to obtain text features .

[0179] 8.7 Calculate the cross-modal similarity between and each candidate image in , and obtain the current round of retrieval ranking results in descending order of similarity .

[0180] 9: end for

[0181] 10: return the final retrieval ranking result .

[0182] Further, Figure 2 ​​​The second embodiment of the present invention provides a schematic diagram of the interaction process effect of a zero-sample text-to-image retrieval method based on scale-aware interaction.

[0183] Example 3:

[0184] Figure 3 This is a framework diagram of a scale-aware interactive zero-shot text-to-image retrieval system provided in Embodiment 3 of the present invention. The system is used to execute the scale-aware interactive zero-shot text-to-image retrieval method described in any embodiment of the present invention. Figure 3 As shown, the system includes:

[0185] The acquisition module 210 is used to acquire an initial text query, and based on a visual language pre-trained retrieval model with frozen parameters, to perform cross-modal similarity calculation between the initial text query and the images in the image library, and to obtain a candidate image set, the similarity score and visual features corresponding to each candidate image in the candidate image set;

[0186] The calculation module 220 is used to determine the retrieval uncertainty based on the similarity score, determine the candidate set diversity based on the visual features, and calculate the retrieval complexity score based on the retrieval uncertainty and the candidate set diversity.

[0187] The determination module 230 is used to determine the global scene-level questioning rounds and the local target-level questioning rounds based on the retrieval complexity score, and generate a scale-aware progressive questioning plan;

[0188] The update module 240 is used to generate retrieval clarification questions according to the scale-aware progressive questioning plan, obtain answers to the retrieval clarification questions, and update the dialogue history based on the retrieval clarification questions and the answers.

[0189] The generation module 250 is used to select the most recent k rounds of question and answer from the dialogue history to form a sliding window context, using the initial text query as the semantic anchor point, and to generate a rewritten query based on the semantic anchor point and the sliding window context.

[0190] Output module 260 is used for a parameter-frozen text encoder to calculate the cross-modal similarity between the rewritten query and the image to be retrieved, and to output the image retrieval ranking result based on the cross-modal similarity.

[0191] also, Figure 4 This is a detailed model framework diagram of a scale-aware interactive zero-shot text-to-image retrieval system provided in Embodiment 3 of the present invention, as shown below. Figure 4 As shown, the various modules of the zero-sample text-to-image retrieval system based on scale-aware interaction of the present invention are illustrated in detail.

[0192] The scale-aware interaction-based zero-shot text-to-image retrieval system provided in this invention can execute the scale-aware interaction-based zero-shot text-to-image retrieval method provided in any embodiment of this invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0193] Example 4:

[0194] Figure 5 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0195] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 to enable the at least one processor 11 to perform the method provided by the present invention.

[0196] The processor 11 can perform various appropriate actions and processes based on a computer program stored in the read-only memory (ROM) 12 or a computer program loaded from the storage unit 18 into the random access memory (RAM) 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0197] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0198] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as zero-shot text-to-image retrieval methods based on scale-aware interaction.

[0199] In some embodiments, the scale-aware interaction-based zero-shot text-to-image retrieval method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the scale-aware interaction-based zero-shot text-to-image retrieval method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the scale-aware interaction-based zero-shot text-to-image retrieval method by any other suitable means (e.g., by means of firmware).

[0200] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a memory system, at least one input device, and at least one output device, and transferring data and instructions to the memory system, the at least one input device, and the at least one output device.

[0201] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0202] In the context of this invention, a computer-readable storage medium stores computer instructions that, when executed by a processor, implement the scale-aware interactive zero-sample text-to-image retrieval method provided by this invention. The computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD monitor)); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0204] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0205] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0206] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0207] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A zero-shot text-to-image retrieval method based on scale-aware interaction, characterized in that, include: An initial text query is obtained. Based on a visual language pre-trained retrieval model with frozen parameters, cross-modal similarity calculation is performed between the initial text query and images in the image library to obtain a candidate image set, the similarity score and visual features corresponding to each candidate image in the candidate image set. The retrieval uncertainty is determined based on the similarity score, the candidate set diversity is determined based on the visual features, and the retrieval complexity score is calculated based on the retrieval uncertainty and the candidate set diversity. Based on the retrieval complexity score, determine the global scenario-level questioning rounds and the local target-level questioning rounds, and generate a scale-aware progressive questioning plan; The system generates clarification questions according to the scale-aware progressive questioning plan, obtains answers to the clarification questions, and updates the dialogue history based on the clarification questions and the answers. Using the initial text query as the semantic anchor, the most recent k rounds of question and answer are selected from the dialogue history to form a sliding window context, and a rewritten query is generated based on the semantic anchor and the sliding window context; A text encoder based on parameter freezing calculates the cross-modal similarity between the rewritten query and the image to be retrieved, and outputs the image retrieval ranking results based on the cross-modal similarity.

2. The method according to claim 1, characterized in that, An initial text query is obtained. Based on a pre-trained visual language retrieval model with frozen parameters, cross-modal similarity calculation is performed between the initial text query and images in the image database to obtain a candidate image set, the similarity score corresponding to each candidate image in the candidate image set, and visual features, including: Obtain the initial text query, call the visual language pre-trained retrieval model with frozen parameters, encode the images in the image library to obtain image features, and encode the initial text query to obtain initial text features; Calculate the similarity between the initial text features and the image features; According to the order of similarity from high to low, a preset number of images are selected as the candidate image set, and the similarity score and visual features corresponding to each candidate image in the candidate image set are retained.

3. The method according to claim 1, characterized in that, The retrieval uncertainty is determined based on the similarity score, the candidate set diversity is determined based on the visual features, and the retrieval complexity score is calculated based on the retrieval uncertainty and the candidate set diversity, including: Temperature normalization is performed on the similarity scores corresponding to the candidate image set to obtain the candidate image probability distribution; The normalized entropy is calculated based on the probability distribution of the candidate images, and the normalized entropy is used as the retrieval uncertainty. The visual features of each candidate image in the candidate image set are normalized. Calculate the feature similarity between the visual features of any two candidate images, and determine the diversity of the candidate set based on the feature similarity; The retrieval complexity score is obtained by weighted summation of the retrieval uncertainty and the diversity of the candidate set.

4. The method according to claim 1, characterized in that, Based on the retrieval complexity score, the global scene-level questioning rounds and the local target-level questioning rounds are determined, and a scale-aware progressive questioning plan is generated, including: Get the total number of preset interaction rounds; The global scenario-level question rounds are determined based on the retrieval complexity score, and the local target-level question rounds are determined based on the preset total number of interaction rounds and the global scenario-level question rounds. A scale-aware progressive questioning plan is generated based on the global scenario-level questioning rounds and the local target-level questioning rounds.

5. The method according to claim 4, characterized in that, Generate clarification questions according to the scale-aware progressive questioning plan, obtain answers to the clarification questions, and update the dialogue history based on the clarification questions and answers, including: According to the scale-aware progressive questioning plan, scale control instructions are generated based on the questioning scale corresponding to the current round. Based on the scale control instructions and the obtained dialogue history, the current round of retrieval clarification questions are generated; Obtain the user's natural language answer to the current round of clarification question, combine the current round of clarification question with the natural language answer to form the current round of question and answer record, and append the current round of question and answer record to the dialogue history.

6. The method according to claim 1, characterized in that, Using the initial text query as a semantic anchor, the most recent k rounds of question-and-answer sessions are selected from the dialogue history to form a sliding window context, and a rewritten query is generated based on the semantic anchor and the sliding window context, including: Using the initial text query as a semantic anchor, the most recent k rounds of question and answer are selected from the dialogue history to form a sliding window context; Based on the user's initial text query and the context of the sliding window, a descriptive short text is generated as the rewritten query.

7. The method according to claim 1, characterized in that, A text encoder based on parameter freezing calculates the cross-modal similarity between the rewritten query and the image to be retrieved, and outputs the image retrieval ranking results based on the cross-modal similarity, including: The rewritten query is text-encoded to obtain rewritten text features, and the pre-calculated features of the image to be retrieved are read. Calculate the similarity between the rewritten text features and the image features to be retrieved; The images to be retrieved are sorted from high to low according to the similarity, and the sorted image retrieval results are output.

8. A zero-shot text-to-image retrieval system based on scale-aware interaction, characterized in that, The system is used to execute the zero-shot text-to-image retrieval method based on scale-aware interaction as described in any one of claims 1-7, comprising: The acquisition module is used to acquire an initial text query, and based on a visual language pre-trained retrieval model with frozen parameters, to perform cross-modal similarity calculation between the initial text query and images in the image library to obtain a candidate image set, the similarity score and visual features corresponding to each candidate image in the candidate image set; The calculation module is used to determine the retrieval uncertainty based on the similarity score, determine the candidate set diversity based on the visual features, and calculate the retrieval complexity score based on the retrieval uncertainty and the candidate set diversity. The determination module is used to determine the global scene-level questioning rounds and the local target-level questioning rounds based on the retrieval complexity score, and generate a scale-aware progressive questioning plan; The update module is used to generate retrieval clarification questions according to the scale-aware progressive questioning plan, obtain answers to the retrieval clarification questions, and update the dialogue history based on the retrieval clarification questions and the answers. The generation module is used to select the most recent k rounds of question and answer from the dialogue history to form a sliding window context, using the initial text query as the semantic anchor point, and to generate a rewritten query based on the semantic anchor point and the sliding window context. The output module is used to calculate the cross-modal similarity between the rewritten query and the image to be retrieved based on the parameter-frozen text encoder, and output the image retrieval ranking results according to the cross-modal similarity.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the zero-sample text-to-image retrieval method based on scale-aware interaction as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the zero-sample text-to-image retrieval method based on scale-aware interaction as described in any one of claims 1-7.

Images