Image retrieval method, apparatus and device

Abstract

The application provides an image retrieval method, device and equipment, and relates to the technical field of image processing. The method comprises the following steps: obtaining a text prompt word input by a user; disassembling the text prompt word into at least two word elements with independent semantics; determining the retrieval priority of each word element based on the user attention degree and the recognition accuracy corresponding to each word element; sequentially using each word element for image iterative retrieval based on the retrieval priority of each word element until all the word elements complete retrieval; and determining the candidate image set obtained in the last round of retrieval as the retrieval image corresponding to the text prompt word, wherein the next round of retrieval is based on the candidate image set obtained in the previous round of retrieval. The application can improve the retrieval accuracy of images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image retrieval method, apparatus, and device. Background Technology

[0002] With the development of multimodal large model technology, text-to-image retrieval has been widely used in fields such as security, transportation, e-commerce, and content management. Users typically search for target images by inputting natural language descriptions, which requires the retrieval system to accurately understand the semantics of the text and match it with massive amounts of image content in order to retrieve the target image.

[0003] Currently, when performing text-to-image search, the text prompts entered by the user are used to extract global semantic feature vectors through a pre-trained multimodal model (such as Contrastive Language-Image Pre-training (CLIP)). Then, the similarity between each image in the image database and the global semantic feature vector is calculated one by one, and finally, the images with the highest similarity are returned as the search results.

[0004] However, in the above methods, when the text prompts contain complex semantics, the retrieval accuracy is low when performing image retrieval using the extracted global semantic feature vector. Summary of the Invention

[0005] This invention provides an image retrieval method, apparatus, and device to address the shortcomings of low image retrieval accuracy in existing technologies and to improve image retrieval accuracy.

[0006] This invention provides an image retrieval method, comprising: Get the text prompts entered by the user; The text prompt word is broken down into at least two semantically independent lexical units; Based on the user attention and recognition accuracy corresponding to each of the aforementioned word elements, the retrieval priority of each of the aforementioned word elements is determined; Based on the retrieval priority of each term, image iterative retrieval is performed sequentially using each term until all terms have been retrieved. The candidate image set obtained in the last round of retrieval is determined as the retrieval image corresponding to the text prompt word. The subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

[0007] According to an image retrieval method provided by the present invention, determining the retrieval priority of each term based on the user attention and recognition accuracy corresponding to each term includes: For each of the aforementioned terms, a user attention score is determined based on the user's historical search behavior. Obtain the recognition accuracy score of the term; Based on the user attention score and the recognition accuracy score, the comprehensive score of the word element is determined; Based on the comprehensive score of each term, the retrieval priority of each term is determined.

[0008] According to an image retrieval method provided by the present invention, obtaining the recognition accuracy score of the word element includes: At least two sample images containing the visual features described by the word element are input into the recognition model to obtain the recognition result of each sample image output by the recognition model. Based on each recognition result and the real label of the corresponding sample image, the recognition accuracy of the word element is determined, and the recognition accuracy is determined as the recognition precision score of the word element. or, A sample image containing the visual features described by the lexical is input into a pre-trained offline recognition model to obtain the recognition confidence score output by the offline recognition model, and the recognition accuracy score of the lexical is determined based on the recognition confidence score.

[0009] According to an image retrieval method provided by the present invention, the step of sequentially performing image iterative retrieval using each of the aforementioned terms based on their retrieval priorities includes: According to the retrieval priority order of each term, each term is traversed sequentially; If the currently traversed word is the first word, a search is performed in the first image database to obtain the first candidate image set; if the currently traversed word is not the first word, a search is performed in the candidate image set obtained in the previous round to obtain the first candidate image set. If a target classification model corresponding to the currently traversed term exists in the shared parameter library, the images in the first candidate image set are filtered using the target classification model to obtain a second candidate image set; the shared parameter library includes multiple sample terms and classification models corresponding to each sample term. The second candidate image set is determined as the retrieval space for the next round of retrieval, so that the next term can be used for image retrieval.

[0010] According to an image retrieval method provided by the present invention, the step of filtering images in the first candidate image set using the target classification model to obtain a second candidate image set includes: Each image in the first candidate image set is input into the target classification model to obtain the matching confidence score output by the target classification model; Based on the comparison result between the matching confidence and the preset confidence, images that meet the preset confidence requirements are determined from the first candidate image set to obtain the second candidate image set.

[0011] According to an image retrieval method provided by the present invention, the number of images in the first candidate image set is related to the recognition accuracy of the currently traversed tokens.

[0012] According to an image retrieval method provided by the present invention, the method further includes: If no target classification model corresponding to the currently traversed word exists in the shared parameter library, the first candidate image set is determined as the second candidate image set.

[0013] According to an image retrieval method provided by the present invention, the classification model is trained based on the following method: Based on the user's historical search records within a preset time period, at least one sample word is determined; For each of the sample words, a search is performed in the second image database using the sample words to obtain the search image set corresponding to the sample words; Each image in the retrieved image set is input into the initial classification model to obtain the corresponding prediction result; Obtain feedback information from the users regarding each of the prediction results; Using the feedback information as training labels, the parameters of the initial classification model are adjusted to obtain the online trained classification model; The sample words and their corresponding trained classification models are associated and stored in the shared parameter library.

[0014] The present invention also provides an image retrieval device, comprising: The acquisition module is used to acquire text prompts input by the user; The decomposition module is used to decompose the text prompt word into at least two semantically independent word units; The determination module is used to determine the retrieval priority of each word element based on the user attention and recognition accuracy corresponding to each word element; The retrieval module is used to perform iterative image retrieval using each of the aforementioned terms based on their retrieval priorities, until all terms have been retrieved. The candidate image set obtained in the last round of retrieval is then determined as the retrieval image corresponding to the text prompt word. The subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image retrieval method as described above.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image retrieval method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image retrieval method as described above.

[0018] The image retrieval method, apparatus, and device provided by this invention acquire text prompts input by the user, decompose the text prompts into at least two semantically independent units, determine the retrieval priority of each unit based on the user attention and recognition accuracy corresponding to each unit, and iteratively retrieve images using each unit sequentially based on the retrieval priority until all units have been retrieved. The candidate image set obtained in the last round of retrieval is determined as the retrieved image corresponding to the text prompt. Each subsequent round of retrieval is based on the candidate image set obtained in the previous round. By decomposing text prompts containing complex semantics into multiple independent units and iteratively matching them in a progressively narrowing candidate image set according to retrieval priority, mutual interference caused by different semantic elements being compressed into the same global feature vector is avoided. This ensures that the final retrieved image simultaneously satisfies the semantic conditions described by each unit, thereby improving image retrieval accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of the image retrieval method provided in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the image retrieval architecture provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of the image retrieval device provided in an embodiment of the present invention.

[0023] Figure 4This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] With the development of multimodal large model technology, text-based image retrieval technology has been widely applied in fields such as security, transportation, e-commerce, and content management. Users often retrieve target images by inputting text prompts containing multiple attributes (such as color, action, object, and scene). For example, the global semantic feature vector of the user-input text prompt can be extracted using CLIP, and this global semantic feature vector can be compared with the image feature vector of each image in the image database to obtain the images with the highest similarity as the final retrieval results. However, when the user-input text prompt is a complex text containing multiple independent semantic elements, CLIP needs to compress the complex semantics of the entire text prompt into a single global semantic feature vector. The differences between different attributes are easily interfered with or diluted, resulting in retrieval results that often only match some semantic elements and cannot simultaneously satisfy all descriptive conditions. This leads to a significant deviation between the presented retrieval image and the user's actual retrieval intent, resulting in low image retrieval accuracy.

[0026] In view of the above problems, this invention proposes an image retrieval method. In this method, the text prompt words input by the user are decomposed into multiple semantically independent word units, and the word units are matched and filtered sequentially in a gradually narrowing candidate image set according to their importance, thereby improving the image retrieval accuracy.

[0027] The following is combined with Figure 1 and Figure 2 The image retrieval method provided by the embodiments of the present invention is described below. The embodiments of the present invention are applicable to scenarios involving text-based image retrieval, and are particularly applicable to scenarios involving image retrieval with complex text prompts containing multiple semantic elements, such as security monitoring, e-commerce search, and content management.

[0028] The subject executing this method can be an electronic device such as a camera, terminal device, computer, server, server cluster, or specially designed image retrieval device, or it can be an image retrieval device installed in the electronic device. The image retrieval device can be implemented by software, hardware, or a combination of both.

[0029] Figure 1 This is a schematic flowchart of the image retrieval method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Obtain the text prompts entered by the user.

[0030] Step 102: Decompose the text prompt word into at least two semantically independent lexical units.

[0031] In this step, after obtaining the text prompt word input by the user, a natural language processing model (such as dependency parsing or word segmentation tools) can be used to decompose it into two lexical units. A lexical unit represents the smallest semantic unit, which is an indivisible independent semantic word obtained after decomposition. For example, suppose the text prompt word is "a man wearing a black shirt and walking a dog," after decomposition, we get the lexical units {man, walking a dog, black shirt}, each with independent semantic meaning.

[0032] Step 103: Determine the retrieval priority of each word based on the user attention and recognition accuracy corresponding to each word.

[0033] In this step, for each lexical element, the user attention and recognition accuracy corresponding to that lexical element can be obtained. User attention characterizes the degree of importance a user attaches to the content described by that lexical element, and can be determined based on the user's historical search behavior, such as the frequency of clicks, browsing duration, or active feedback on search results containing that lexical element. Recognition accuracy characterizes the accuracy with which the retrieval system recognizes the visual features described by that lexical element, and can be determined in advance through offline testing or model evaluation of core lexical elements.

[0034] After obtaining the user attention and recognition accuracy corresponding to each term, these two metrics can be weighted and fused to obtain a comprehensive score for each term. The retrieval priority of each term is then determined according to its comprehensive score, from highest to lowest. A higher retrieval priority indicates that the term will be used more frequently in subsequent retrieval processes. Since terms with higher retrieval priority have higher user attention and recognition accuracy, subsequent iterative retrieval processes can prioritize using the terms that users are most interested in and that the system recognizes most accurately for image filtering, gradually narrowing the range of candidate images and improving retrieval efficiency and accuracy.

[0035] Step 104: Based on the retrieval priority of each word, perform iterative image retrieval using each word in turn until all words have been retrieved. The candidate image set obtained in the last round of retrieval is determined as the retrieval image corresponding to the text prompt word. The subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

[0036] In this step, after determining the retrieval priority of each term, image retrieval is performed sequentially according to the retrieval priority from high to low. In each round of retrieval, the retrieval space is the set of candidate images obtained in the previous round. If the current round is the first round of retrieval, the retrieval space is a pre-defined image library. By narrowing the retrieval scope round by round, images that do not match the current term can be gradually filtered out, ensuring that the final set of candidate images simultaneously satisfies the semantic conditions described by all terms, thus improving retrieval accuracy.

[0037] The image retrieval method provided in this invention obtains text prompts input by the user, decomposes the text prompts into at least two semantically independent units, determines the retrieval priority of each unit based on the user attention and recognition accuracy corresponding to each unit, and iteratively retrieves images using each unit sequentially based on the retrieval priority until all units have been retrieved. The candidate image set obtained in the last round of retrieval is determined as the retrieved image corresponding to the text prompt. Each subsequent round of retrieval is based on the candidate image set obtained in the previous round. Because text prompts containing complex semantics are decomposed into multiple independent units and iteratively matched in a progressively narrowing candidate image set according to retrieval priority, mutual interference caused by different semantic elements being compressed into the same global feature vector is avoided. This ensures that the final retrieved image simultaneously satisfies the semantic conditions described by each unit, thereby improving image retrieval accuracy.

[0038] For example, based on the above embodiments, when determining the retrieval priority of each word based on the user attention and recognition accuracy corresponding to each word, it can be done in the following way: For each term, based on the user's historical search behavior, the user attention score of the term is determined, and the recognition accuracy score of the term is obtained. Based on the user attention score and the recognition accuracy score, the comprehensive score of the term is determined, and based on the comprehensive score of each term, the retrieval priority of each term is determined.

[0039] Specifically, a user's historical search behavior includes interactions such as clicking, browsing, saving, sharing, and marking the accuracy of search results. Based on a user's historical search behavior within a preset historical time period, the number of interactions corresponding to each keyword can be counted, and a user attention score can be determined based on this number of interactions. For example, the percentage of interactions with a particular keyword out of the total interactions with all keywords can be used to determine the user attention score for that keyword. The higher the user attention score for a keyword, the more attention or importance the user attaches to the content described by that keyword.

[0040] The word recognition accuracy score characterizes the accuracy with which a retrieval system recognizes the visual features described by that word. In one possible implementation, for some frequently occurring words, at least two sample images containing the visual features described by the word can be input into the recognition model in advance through offline testing. The recognition results of each sample image output by the recognition model are obtained. Based on each recognition result and the true label of the corresponding sample image, the word recognition accuracy is determined, and this accuracy is defined as the word recognition accuracy score.

[0041] Specifically, after obtaining the recognition results of each sample image output by the recognition model, the recognition accuracy of the word can be calculated based on the consistency between each recognition result and the corresponding true label of the sample image. This recognition accuracy is then used as the recognition precision score of the word. Furthermore, the correspondence between the obtained words and their corresponding recognition precision scores can be stored. When the recognition precision score of a word is needed, it can be obtained directly by looking up this correspondence, thus enabling the rapid acquisition of the recognition precision score for each word.

[0042] For example, if the word "black top" is used, a batch of sample images containing and not containing a black top can be pre-collected. Each sample image is labeled with a ground truth label, indicating whether the image contains a black top. These sample images are then input into a recognition model, which outputs a prediction result for each image. This prediction result indicates whether the image contains a black top. The prediction result is compared with the ground truth label, and the proportion of correctly predicted images out of the total number of images is calculated. This proportion is used as the recognition accuracy score for the word "black top". Assuming there are 100 sample images, and the recognition model's prediction results match the ground truth label for 92 of them, the recognition accuracy score for the word "black top" is 92%.

[0043] To improve the efficiency of obtaining subsequent recognition accuracy scores, the correspondence between the word "black top" and the recognition accuracy score of 92% can be stored. In this way, during subsequent image retrieval, if the recognition accuracy score of the word "black top" needs to be obtained, the 92% can be obtained directly by looking up the correspondence, without having to repeat the model evaluation.

[0044] By determining the recognition accuracy of each word and then determining the recognition precision score of each word based on the recognition accuracy, the recognition precision score can objectively reflect the recognition model's ability to recognize the visual features described by different words, avoiding the bias caused by subjective estimation and improving the accuracy of word recognition precision scores.

[0045] In another possible implementation, a pre-trained offline recognition model can be used to evaluate each word. For example, sample images containing the visual features described by the word can be input into the pre-trained offline recognition model to obtain the recognition confidence score output by the model. The recognition accuracy score of the word can then be determined based on this confidence score. For instance, for the word "red car," sample images containing a red car can be input into the pre-trained offline recognition model. The model outputs a recognition confidence score for that sample image, which characterizes the model's certainty that the sample image contains a red car. Assuming the offline recognition model outputs a recognition confidence score of 0.93 for a given sample image, this confidence score can be used as the recognition accuracy score for the word "red car." When there are multiple sample images, the average recognition confidence score of each sample image can be taken and used as the recognition accuracy score for that word. In this way, the recognition accuracy score of each word can be quickly obtained directly using the output of the offline recognition model. For newly added words, this method can improve the efficiency of obtaining their recognition accuracy scores.

[0046] After obtaining the user attention score and recognition accuracy score for each word element, weight coefficients can be assigned to these scores respectively. The two scores are then weighted and summed according to their respective weight coefficients to obtain the comprehensive score for each word element. The weight coefficients can be set or dynamically adjusted based on the actual application scenario.

[0047] Furthermore, the retrieval priority of each term can be determined by ranking the overall scores from highest to lowest.

[0048] In this embodiment, the retrieval priority of words is determined by comprehensively considering two dimensions: user attention and recognition accuracy. This allows words that users pay more attention to and that the system recognizes more accurately to be used for retrieval first, thereby more effectively narrowing the range of candidate image sets in subsequent iterative retrieval processes and improving retrieval efficiency and accuracy.

[0049] For example, based on the above embodiments, when performing iterative image retrieval using each term sequentially according to its retrieval priority, each term can be traversed sequentially according to its retrieval priority. If the currently traversed term is the first term, a search is performed in the first image library to obtain a first candidate image set. If the currently traversed term is not the first term, a search is performed in the candidate image set obtained in the previous round of retrieval to obtain a first candidate image set. If a target classification model corresponding to the currently traversed term exists in the shared parameter library, the images in the first candidate image set are filtered using the target classification model to obtain a second candidate image set. The shared parameter library includes multiple sample terms and classification models corresponding to each sample term. The second candidate image set is determined as the retrieval space for the next round of retrieval, so that the next term can perform image retrieval.

[0050] Specifically, the first round of retrieval begins with the term with the highest retrieval priority, following the retrieval priority order of each term. The term with the highest retrieval priority is taken as the currently traversed term. Since this is the first term, it is used as the retrieval condition to search within a pre-defined first image database, obtaining a first set of candidate images most semantically relevant to that term. For example, the top K images can be retrieved from the first image database as the first candidate image set. The number of images in the first candidate image set is related to the recognition accuracy of the currently traversed term, or it can be dynamically adjusted by the user. For example, when the recognition accuracy of the term is high, a smaller K value, such as 500, can be set to reduce the computational load of subsequent searches and improve computational efficiency. When the recognition accuracy of the term is low, a larger K value, such as 2000, can be set to ensure recall. The specific value of K can be set according to the actual application scenario.

[0051] After retrieving the first candidate image set corresponding to the term with the highest retrieval priority, the shared parameter library can be queried to see if a target classification model corresponding to that term exists. If a target classification model exists, it is used to filter the images in the first candidate image set to remove images that do not contain the visual features described by the term, thus obtaining the second candidate image set.

[0052] The shared parameter library contains multiple sample words and pre-trained classification models corresponding to each sample word. These classification models are used to determine whether the input image contains the visual features described by the corresponding word.

[0053] In one possible implementation, when filtering images in the first candidate image set using the target classification model, each image in the first candidate image set can be input into the target classification model to obtain the matching confidence score output by the target classification model. In the first candidate image set, images with matching confidence scores lower than a preset confidence score are deleted to obtain the second candidate image set.

[0054] Specifically, each image in the first candidate image set is input into the target classification model. Through the forward computation of the target classification model, the matching confidence score of each image is output. The matching confidence score characterizes the probability or likelihood that an image contains the visual feature described by the given term. If the matching confidence score of an image is less than a preset confidence score, it indicates that the image is very likely not to contain the visual feature described by the given term; therefore, the image can be removed from the candidate image set.

[0055] For example, suppose the currently encountered term is "black top". The first candidate image set contains 100 images. These 100 images are sequentially input into the target classification model corresponding to the term "black top". The target classification model outputs a matching confidence score for each image. The preset confidence score is 0.8. For an image that actually contains a black top, the target classification model represents the probability that the image contains the visual feature of a black top. If the hypothetical output matching confidence score is 0.95, which is higher than the preset confidence score, the image is retained. For another image that actually contains a dark blue top, the target classification model outputs a matching confidence score of 0.15, which is lower than the preset confidence score, so the image is deleted from the first candidate image set. After the above filtering operation, a second candidate image set containing 80 images is finally obtained. These images are all determined to contain a black top, thus effectively eliminating images that do not match the current term.

[0056] In this embodiment, candidate images are filtered using a target classification model, and images with a matching confidence level lower than a preset confidence level are deleted. This effectively eliminates images that do not match the currently traversed terms, reducing interference in subsequent retrieval rounds and thus improving retrieval accuracy and efficiency. Furthermore, since image retrieval is performed for each term, and a corresponding target classification model exists, it is used to further filter images. This ensures that the candidate image set passed to the next round of retrieval has higher accuracy, allowing subsequent term retrieval to be performed on a higher-quality candidate image set, further improving retrieval performance. Moreover, since the classification models stored in the shared parameter library are trained online, they can be continuously optimized according to the user's personalized retrieval habits, and the trained classification models can be reused in different retrieval tasks containing the same terms, further reducing computational costs.

[0057] If no target classification model exists in the shared parameter library, the first candidate image set is directly used as the second candidate image set, and the second candidate image set is determined as the retrieval space for the next round of retrieval. That is, no additional filtering is performed on the first candidate image set, so iterative retrieval can still be carried out normally even if no target classification model exists, thus improving the robustness of image retrieval.

[0058] After the highest priority term has been retrieved, the next highest priority term is taken as the current term being traversed. Since this term is not the first term, it is used as the search condition to search the second candidate image set obtained in the previous round, resulting in the first candidate image set most semantically relevant to this term. Similarly, the shared parameter library is queried to see if a target classification model corresponding to the next highest priority term exists. If it does, the first candidate image set is filtered according to the target classification model to obtain a new second candidate image set, which is used as the search space for the next round. If no target classification model exists, the obtained first candidate image set is used as the new second candidate image set. This process is repeated until all terms have been retrieved, and the candidate image set obtained in the last round is determined as the search image corresponding to the text prompt word.

[0059] In this embodiment, iterative retrieval is performed using each term sequentially according to retrieval priority. In each round of retrieval, the target classification model corresponding to the term is used to filter the images in the retrieved candidate image set. This gradually narrows the range of the candidate image set with each retrieval round, ensuring that the images in the final candidate image set simultaneously satisfy the semantic conditions described by all terms. Furthermore, since the shared parameter library stores classification models corresponding to multiple sample terms, these models can be reused for different retrieval tasks, thereby improving retrieval accuracy while reducing computational costs.

[0060] For example, based on the above embodiments, the classification model in the shared parameter library is trained in the following manner: Based on the user's historical search records within a preset time period, at least one sample word is determined. For each sample word, a search is performed in the second image database to obtain the set of searched images corresponding to the sample word. Each image in the searched image set is input into the initial classification model to obtain the corresponding prediction results. Feedback information from the user on each prediction result is obtained. The feedback information is used as training labels to adjust the parameters of the initial classification model to obtain the online trained classification model. The sample word and the corresponding trained classification model are associated and stored in the shared parameter database.

[0061] Because the model trained on complete prompt words is highly coupled with the prompt words themselves, the retrieval model will fail if the prompt words change even slightly. To address this issue, in this embodiment of the invention, instead of training on complete prompt words, online training is performed on frequently used word units. The trained classification model can be used in any combination.

[0062] Specifically, the preset time period is a pre-defined time window, such as the last 30 days, used to limit the scope of statistical analysis of users' historical search behavior, thereby dynamically adapting to changes in users' interests. Historical search records include text suggestions entered by the user during the search process, clicks on search results, and positive or negative feedback on the results. By analyzing historical search records, the frequency of each word / character within the preset time period can be statistically analyzed. Words / characters with higher frequencies are identified as sample words / characters, representing semantic elements frequently searched by the user, such as "black shirt" or "dog walking."

[0063] After determining the sample terms, for each sample term, a search is performed in the second image library using that sample term to obtain a set of retrieved images semantically related to that sample term. The second image library provides the initial candidate images needed to train the classification model; it may or may not be the same as the aforementioned first image library.

[0064] Each image in the retrieved image set is input into the initial classification model. The initial classification model outputs a prediction result for each image, which instructs the initial classification model to determine whether the image contains the visual features described by the sample word. Furthermore, the prediction results can be pushed to the user's corresponding terminal device, where the user can evaluate the accuracy of the prediction results and obtain user feedback information for each prediction result. This feedback information includes positive and negative feedback; positive feedback indicates that the user approves of the initial classification model's prediction results, while negative feedback indicates that the user does not approve of the initial classification model's prediction results.

[0065] Using user feedback as training labels, the parameters of the initial classification model are adjusted to enable the model to more accurately identify the visual features described by the word units in the sample during subsequent predictions. After multiple rounds of training and adjustment, the trained classification model can be obtained.

[0066] After training each sample word in the above manner, the sample word and the corresponding trained classification model are associated and stored in the shared parameter library so that they can be directly called in subsequent retrieval processes.

[0067] For example, suppose a user frequently searches for images containing the word "black top" in their search history over the past 30 days. Statistical analysis shows that the term "black top" appears frequently, so it is selected as a sample term. After searching a second image database using "black top," 100 images related to "black top" are obtained, forming a search image set. These 100 images are sequentially input into an initial classification model, which outputs a prediction result for each image, such as predicting whether an image contains a black top. This prediction result is then pushed to the user. The user, upon viewing the image, discovers that the person in the image is actually wearing a dark blue top, not a black one, and thus inputs negative feedback. In the same way, feedback information corresponding to each of the 100 images can be obtained. This feedback information is used as training labels to adjust the parameters of the initial classification model, enabling the model to correctly distinguish whether a person is wearing a black top in subsequent classifications. After multiple parameter adjustments, the finally trained classification model is obtained. The word "black top" is associated with the trained classification model and stored in the shared parameter library. In subsequent retrieval processes, if the text prompt word entered by the user contains the word "black top", the corresponding classification model can be directly called from the shared parameter library for image filtering.

[0068] In this embodiment, high-frequency sample terms are determined by utilizing users' historical search behavior, and the classification model is trained online based on user feedback on the model's prediction results. This allows the trained classification model to more accurately identify the visual features that users are interested in. Furthermore, since the classification model is trained online at the term level, it has strong generalization ability. The same classification model can be reused for different text prompts containing the same term, thereby improving retrieval accuracy while reducing model training costs. Moreover, the trained classification model can be arbitrarily combined, thus completely solving the problem of image retrieval models failing when text prompts change, and enabling adaptive optimization based on individual user search habits.

[0069] Figure 2 This is a schematic diagram of the image retrieval architecture provided in an embodiment of the present invention, such as... Figure 2 As shown, the image encoding module extracts features from the input images in the image set, converting them into an image feature set. The text encoding module segments the input text prompts into word units; for example, "a man wearing a black shirt and walking a dog" is segmented into word units such as "man," "black shirt," and "walking dog," and each word unit's features are encoded to generate a text feature set. Following the retrieval priority order corresponding to each word unit, the image-text matching module calculates the similarity between the text features and the image features in the image feature set, selecting at least one image with the highest matching degree as a candidate image. After traversing all word units, the candidate image obtained in the last round is used as the final target image.

[0070] The image retrieval device provided by the present invention is described below. The image retrieval device described below can be referred to in correspondence with the image retrieval method described above.

[0071] Figure 3 This is a schematic diagram of the structure of the image retrieval device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the image retrieval device 300 includes: The acquisition module 11 is used to acquire the text prompt words input by the user; The decomposition module 12 is used to decompose the text prompt word into at least two lexical units with independent semantics; The determining module 13 is used to determine the retrieval priority of each word element based on the user attention and recognition accuracy corresponding to each word element; The retrieval module 14 is used to perform image iterative retrieval sequentially using each of the aforementioned word elements based on their retrieval priorities, until all word elements have been retrieved. The candidate image set obtained in the last round of retrieval is then determined as the retrieval image corresponding to the text prompt word. The subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

[0072] In one example embodiment, the determining module 13 is specifically used for: For each of the aforementioned terms, a user attention score is determined based on the user's historical search behavior. Obtain the recognition accuracy score of the term; Based on the user attention score and the recognition accuracy score, the comprehensive score of the word element is determined; Based on the comprehensive score of each term, the retrieval priority of each term is determined.

[0073] In one example embodiment, the acquisition module 11 is further configured to: At least two sample images containing the visual features described by the word element are input into the recognition model to obtain the recognition result of each sample image output by the recognition model. Based on each recognition result and the real label of the corresponding sample image, the recognition accuracy of the word element is determined, and the recognition accuracy is determined as the recognition precision score of the word element. or, A sample image containing the visual features described by the lexical is input into a pre-trained offline recognition model to obtain the recognition confidence score output by the offline recognition model, and the recognition accuracy score of the lexical is determined based on the recognition confidence score.

[0074] In one example embodiment, the retrieval module 14 is specifically used for: According to the retrieval priority order of each term, each term is traversed sequentially; If the currently traversed word is the first word, a search is performed in the first image database to obtain the first candidate image set; if the currently traversed word is not the first word, a search is performed in the candidate image set obtained in the previous round to obtain the first candidate image set. If a target classification model corresponding to the currently traversed term exists in the shared parameter library, the images in the first candidate image set are filtered using the target classification model to obtain a second candidate image set; the shared parameter library includes multiple sample terms and classification models corresponding to each sample term. The second candidate image set is determined as the retrieval space for the next round of retrieval, so that the next term can be used for image retrieval.

[0075] In one example embodiment, the retrieval module 14 is specifically used for: Each image in the first candidate image set is input into the target classification model to obtain the matching confidence score output by the target classification model; Based on the comparison result between the matching confidence and the preset confidence, images that meet the preset confidence requirements are determined from the first candidate image set to obtain the second candidate image set.

[0076] In one example embodiment, the number of images in the first candidate image set is related to the recognition accuracy of the currently traversed lexical unit.

[0077] In one example embodiment, the determining module 13 is further configured to determine the first candidate image set as the second candidate image set if there is no target classification model corresponding to the currently traversed word in the shared parameter library.

[0078] In one example embodiment, the classification model is trained in the following manner: Based on the user's historical search records within a preset time period, at least one sample word is determined; For each of the sample words, a search is performed in the second image database using the sample words to obtain the search image set corresponding to the sample words; Each image in the retrieved image set is input into the initial classification model to obtain the corresponding prediction result; Obtain feedback information from the users regarding each of the prediction results; Using the feedback information as training labels, the parameters of the initial classification model are adjusted to obtain the online trained classification model; The sample words and their corresponding trained classification models are associated and stored in the shared parameter library.

[0079] The apparatus of this embodiment can be used in any of the methods in the image retrieval method side embodiment. Its specific implementation process and technical effects are similar to those in the image retrieval method side embodiment. For details, please refer to the detailed description in the image retrieval method side embodiment, which will not be repeated here.

[0080] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an image retrieval method. This method includes: acquiring a text prompt word input by a user; decomposing the text prompt word into at least two semantically independent units; determining the retrieval priority of each unit based on the user attention and recognition accuracy corresponding to each unit; performing iterative image retrieval using each unit sequentially based on its retrieval priority until all units have been retrieved; and determining the candidate image set obtained in the last round of retrieval as the retrieval image corresponding to the text prompt word. The subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

[0081] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image retrieval method provided by the above methods. The method includes: obtaining a text prompt word input by a user; decomposing the text prompt word into at least two semantically independent word units; determining the retrieval priority of each word unit based on the user attention and recognition accuracy corresponding to each word unit; performing iterative image retrieval using each word unit sequentially based on the retrieval priority of each word unit until all word units have been retrieved; and determining the candidate image set obtained in the last round of retrieval as the retrieval image corresponding to the text prompt word, wherein the subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image retrieval method provided by the above methods. The method includes: acquiring a text prompt word input by a user; decomposing the text prompt word into at least two semantically independent word units; determining the retrieval priority of each word unit based on the user attention and recognition accuracy corresponding to each word unit; performing iterative image retrieval using each word unit sequentially based on the retrieval priority of each word unit until all word units have been retrieved; and determining the candidate image set obtained in the last round of retrieval as the retrieval image corresponding to the text prompt word, wherein the subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image retrieval method, characterized in that, include: Get the text prompts entered by the user; The text prompt word is broken down into at least two semantically independent lexical units; Based on the user attention and recognition accuracy corresponding to each of the aforementioned word elements, the retrieval priority of each of the aforementioned word elements is determined; Based on the retrieval priority of each term, image iterative retrieval is performed sequentially using each term until all terms have been retrieved. The candidate image set obtained in the last round of retrieval is determined as the retrieval image corresponding to the text prompt word. The subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

2. The image retrieval method according to claim 1, characterized in that, The step of determining the retrieval priority of each term based on the user attention and recognition accuracy corresponding to each term includes: For each of the aforementioned terms, a user attention score is determined based on the user's historical search behavior. Obtain the recognition accuracy score of the term; Based on the user attention score and the recognition accuracy score, the comprehensive score of the word element is determined; Based on the comprehensive score of each term, the retrieval priority of each term is determined.

3. The image retrieval method according to claim 2, characterized in that, The process of obtaining the recognition accuracy score of the word element includes: At least two sample images containing the visual features described by the word element are input into the recognition model to obtain the recognition result of each sample image output by the recognition model. Based on each recognition result and the real label of the corresponding sample image, the recognition accuracy of the word element is determined, and the recognition accuracy is determined as the recognition precision score of the word element. or, A sample image containing the visual features described by the lexical is input into a pre-trained offline recognition model to obtain the recognition confidence score output by the offline recognition model, and the recognition accuracy score of the lexical is determined based on the recognition confidence score.

4. The image retrieval method according to claim 1, characterized in that, The method of sequentially performing image iterative retrieval using each of the aforementioned terms based on their retrieval priority includes: According to the retrieval priority order of each term, each term is traversed sequentially; If the currently traversed word is the first word, a search is performed in the first image database to obtain the first candidate image set; if the currently traversed word is not the first word, a search is performed in the candidate image set obtained in the previous round to obtain the first candidate image set. If a target classification model corresponding to the currently traversed term exists in the shared parameter library, the images in the first candidate image set are filtered using the target classification model to obtain a second candidate image set; the shared parameter library includes multiple sample terms and classification models corresponding to each sample term. The second candidate image set is determined as the retrieval space for the next round of retrieval, so that the next term can be used for image retrieval.

5. The image retrieval method according to claim 4, characterized in that, The step of using the target classification model to filter images in the first candidate image set to obtain a second candidate image set includes: Each image in the first candidate image set is input into the target classification model to obtain the matching confidence score output by the target classification model; Based on the comparison result between the matching confidence and the preset confidence, images that meet the preset confidence requirements are determined from the first candidate image set to obtain the second candidate image set.

6. The image retrieval method according to claim 4, characterized in that, The number of images in the first candidate image set is related to the recognition accuracy of the currently traversed lexical units.

7. The image retrieval method according to any one of claims 4-6, characterized in that, The method further includes: If no target classification model corresponding to the currently traversed word exists in the shared parameter library, the first candidate image set is determined as the second candidate image set.

8. The image retrieval method according to any one of claims 4-6, characterized in that, The classification model was trained in the following manner: Based on the user's historical search records within a preset time period, at least one sample word is determined; For each of the sample words, a search is performed in the second image database using the sample words to obtain the search image set corresponding to the sample words; Each image in the retrieved image set is input into the initial classification model to obtain the corresponding prediction result; Obtain feedback information from the users regarding each of the prediction results; Using the feedback information as training labels, the parameters of the initial classification model are adjusted to obtain the online trained classification model; The sample words and their corresponding trained classification models are associated and stored in the shared parameter library.

9. An image retrieval device, characterized in that, include: The acquisition module is used to acquire text prompts input by the user; The decomposition module is used to decompose the text prompt word into at least two semantically independent word units; The determination module is used to determine the retrieval priority of each word element based on the user attention and recognition accuracy corresponding to each word element; The retrieval module is used to perform iterative image retrieval using each of the aforementioned terms based on their retrieval priorities, until all terms have been retrieved. The candidate image set obtained in the last round of retrieval is then determined as the retrieval image corresponding to the text prompt word. The subsequent round of retrieval is based on the candidate image set obtained in the previous round of retrieval.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image retrieval method as described in any one of claims 1 to 8.

Images