Image search method, device, equipment and medium based on large model

Abstract

The present disclosure proposes an image search method, apparatus, device and medium based on a large model, which relates to artificial intelligence fields such as computer vision and natural language processing, and is applicable to a large model, including: obtaining a search statement and a candidate image information library; obtaining a candidate image set of the search statement and a candidate feature question-answer pair set of each candidate image in the candidate image set from the candidate image information library; obtaining a target feature question-answer pair set of each candidate image from the candidate feature question-answer pair set of each candidate image through a pre-acquired large model; and obtaining a target search image of the search statement from each candidate image based on the target feature question-answer pair set.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, in particular to artificial intelligence fields such as computer vision and natural language processing, and is applicable to large models. Background Art

[0002] With the development of technology, people have more and more demands for image search. In related technologies, a large image model can be used to convert images into word vectors, and the search requirements input by users can be converted into text vectors. The user's search image can be obtained through parameters such as the similarity between the word vector of the image and the text vector of the search requirement.

[0003] However, in the process of converting the image into word vectors, image features may be lost, resulting in the loss of some detailed descriptions of the image, which in turn affects the accuracy of image search results to a certain extent. Summary of the invention

[0004] The present disclosure proposes a large model-based image search method, device, equipment and medium.

[0005] According to the first aspect of the present disclosure, a large model-based image search method is proposed, the method comprising: obtaining a search statement and a candidate image information library; obtaining a candidate image set of the search statement and a candidate feature question-answer pair set of each candidate image in the candidate image set from the candidate image information library; obtaining a target feature question-answer pair set of each candidate image from the candidate feature question-answer pair set of each candidate image through a pre-acquired large model; and obtaining a target search image of the search statement from each candidate image based on the target feature question-answer pair set.

[0006] According to a second aspect of the present disclosure, a large model-based image search device is proposed, which includes: a first acquisition module, used to acquire a search statement and a candidate image information library; a second acquisition module, used to acquire a candidate image set of the search statement and a candidate feature question-answer pair set of each candidate image in the candidate image set from the candidate image information library; a third acquisition module, used to acquire a target feature question-answer pair set of each candidate image from the candidate feature question-answer pair set of each candidate image through a pre-acquired large model; and a search module, used to acquire a target search image of the search statement from each candidate image based on the target feature question-answer pair set.

[0007] According to the third aspect of the present disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large model-based image search method proposed in the first aspect above.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute the large model-based image search method proposed in the first aspect above.

[0009] According to a fifth aspect of the present disclosure, a computer program product is proposed, including a computer program, which, when executed by a processor, implements the large model-based image search method proposed in the first aspect.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 A schematic diagram of a flow chart of an image search method based on a large model according to an embodiment of the present disclosure;

[0013] Figure 2 A schematic diagram of a flow chart of an image search method based on a large model according to another embodiment of the present disclosure;

[0014] Figure 3 A schematic diagram of a flow chart of an image search method based on a large model according to another embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram of the structure of an image search device based on a large model according to an embodiment of the present disclosure;

[0016] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] Data processing is a basic part of system engineering and automatic control. Data is a form of expression of facts, concepts or instructions, which can be processed by manual or automatic devices. After data is interpreted and given a certain meaning, it becomes information. Data processing is the collection, storage, retrieval, processing, transformation and transmission of data. The basic purpose of data processing is to extract and derive valuable and meaningful data for certain specific people from a large amount of data that may be disorganized and difficult to understand.

[0019] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Natural language processing is mainly used in machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, Chinese OCR, etc.

[0020] Computer Vision (CV) refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing. Various imaging systems are used to replace visual organs as input sensitive means, and computers are used to replace the brain to complete processing and interpretation. Computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. Computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data.

[0021] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial Intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. Since the birth of artificial intelligence, the theory and technology of artificial intelligence have become increasingly mature, and the application field has continued to expand. It can be imagined that the technological products brought by artificial intelligence in the future will be the "container" of human wisdom. Artificial Intelligence can simulate the information process of human consciousness and thinking.

[0022] Figure 1 FIG. 1 is a flow chart of an image search method based on a large model according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0023] S101, obtaining a search statement and a candidate image information library;

[0024] In the disclosed embodiment, a sentence input by a user for performing an image search may be marked as a search sentence, wherein the search sentence carries relevant information of image features that the user needs to search for.

[0025] Optionally, the information library used for image search can be marked as a candidate image information library, wherein image search can be performed in the candidate image information library using relevant information of image features carried in the search statement, thereby obtaining a search image that meets the user requirements corresponding to the search statement.

[0026] S102, obtaining a candidate image set of the search statement and a candidate feature question-answer pair set of each candidate image in the candidate image set from a candidate image information library.

[0027] In the disclosed embodiment, the search statement has a certain search range in the candidate image information library, and the set of images covered within the search range can be marked as the candidate image set of the candidate statement in the candidate image information library.

[0028] Optionally, a search strategy corresponding to the search statement may be obtained, and a search scope of the search statement in the candidate image information library may be determined based on the search strategy, thereby obtaining a set of candidate images of the search statement in the candidate image information library.

[0029] In the disclosed embodiment, the candidate image information library includes an image storage cluster and a feature question storage cluster, wherein the feature question can be understood as a targeted question raised for the image features in each image in the image storage cluster.

[0030] Optionally, the candidate image information library also includes a feature description storage cluster, wherein, for any image, feature description information corresponding to each feature question in the image can be obtained, and the information is used as the feature description of each feature question in the image and stored in the feature description storage cluster.

[0031] In this scenario, the feature question associated with the search statement can be obtained from the feature question storage cluster, and the feature description of the feature question in the candidate image can be obtained from the feature description storage cluster, and the information pair consisting of the feature question and the feature description can be marked as a candidate feature question-answer pair for the candidate image.

[0032] Among them, the search statement input by the user may have multiple associated feature questions in the feature question storage cluster. In this scenario, a set of candidate feature question-answer pairs for candidate images can be marked based on the set of multiple information pairs consisting of the multiple feature questions and the feature descriptions of the multiple feature questions.

[0033] It should be noted that the feature question storage cluster in the candidate image information library can be a cluster (MySQL), the feature description storage cluster can be a cluster (Elasticsearch), the image storage cluster can be a cluster (MinIo), or it can be other types of clusters that can realize feature question storage, feature description storage and image storage. No specific restrictions are made at this time.

[0034] S103, obtaining a target feature question-answer pair set for each candidate image from the candidate feature question-answer pair set of each candidate image through the pre-acquired large model;

[0035] In the disclosed embodiment, the set of candidate feature question-answer pairs and the search statement of each candidate image can be input into a pre-acquired large model, and the set of candidate feature question-answer pairs of each candidate image can be analyzed through the model capability of the large model, and then the target feature question-answer pairs can be obtained from the set of candidate feature question-answer pairs of each candidate image.

[0036] Among them, for any candidate image, the model capability of the large model can be used to obtain some feature question and answer pairs whose degree of correlation with the search statement meets the preset correlation conditions from the candidate feature question and answer pairs of the candidate image, and use the set composed of these some feature question and answer pairs as the target feature question and answer pair set of the search statement under the candidate image.

[0037] It should be noted that the big model proposed in the embodiment of the present disclosure can be the Wenxin Yiyan big model, or it can be other big models that have the ability to obtain target feature question and answer pairs of candidate images, and no specific limitation is made here.

[0038] S104, obtaining a target search image of the search statement from each candidate image according to the target feature question-answer pair set.

[0039] In the disclosed embodiment, for a target feature question-answer pair set of any candidate image, each target feature question and answer included in the target feature question-answer pair set may be analyzed.

[0040] Among them, the degree of correlation between the target feature question-answer pair of each candidate image and the search statement can be obtained, and the candidate images can be sorted according to the degree of correlation, so as to obtain at least one image in the candidate image set whose degree of correlation with the search statement meets the preset conditions, and the image is used as the target search image obtained by the search statement in the candidate image information library.

[0041] The image search method based on a large model proposed in the present disclosure obtains a search statement, and obtains a candidate image set of the search statement and a candidate feature question-answer pair set of each candidate image from a candidate image information library, obtains a target feature question-answer pair set of each candidate image from each candidate feature question-answer pair set through a large model, and then obtains a target search image of the search statement from the candidate image set according to the target feature question-answer pair set of each candidate image. In the present disclosure, the target feature question-answer pair set of the search statement under each candidate image is obtained through a large model, which improves the acquisition efficiency and accuracy of the target feature question-answer pair set. Compared with the image search based on the user demand conversion vector and the image conversion vector in the related art, the target search image is obtained based on the target feature question-answer pair set of the search statement under the candidate image through the large model, which avoids the influence of the loss of image details caused by the feature loss that may occur in the vector conversion process on the search results, improves the acquisition efficiency and accuracy of the target search image, reduces the operation difficulty of the image search method based on the large model, improves the practicality and applicability of the image search method based on the large model, and optimizes the user's image search experience.

[0042] In the above embodiment, the acquisition of the target search image can also be combined with Figure 2 Further understanding, Figure 2 FIG. 1 is a flow chart of an image search method based on a large model according to another embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0043] S201, obtaining a candidate image set of a search statement and a candidate feature question-answer pair set of each candidate image in the candidate image set from a candidate image information library.

[0044] As a possible implementation manner, a candidate image cluster in a candidate image information library may be obtained, and all images in the candidate image cluster may be used as a candidate image set.

[0045] In the embodiment of the present disclosure, a cluster for storing images in the candidate image information library may be marked as a candidate image cluster in the candidate image information library.

[0046] Among them, all images included in the candidate image cluster can be used as search objects of the search statement. In this scenario, the set composed of all images in the candidate image cluster can be marked as the candidate image set of the search statement in the candidate image information library.

[0047] Optionally, a candidate feature question cluster and a candidate feature description cluster are obtained from a candidate image information library.

[0048] In the disclosed embodiment, the cluster storing feature questions included in the candidate image information library may be marked as a candidate feature question cluster, and the cluster storing feature descriptions may be marked as a candidate feature description cluster.

[0049] Optionally, a first candidate feature question set associated with the search statement is obtained from the candidate feature question cluster, and a first candidate feature description of each first candidate feature question in the first candidate feature question set is obtained from the candidate feature description cluster.

[0050] In the disclosed embodiment, some feature questions associated with the search statement may be obtained from each candidate feature question in the candidate feature question cluster, and the set consisting of the some feature questions may be marked as the first candidate feature question set.

[0051] In this scenario, the feature descriptions of each first candidate feature question in the first candidate feature question set may be obtained from the candidate feature description cluster and marked as the first candidate feature description of each first candidate feature question.

[0052] In the disclosed embodiment, the candidate feature question clusters have their own question identification information, and the candidate feature description clusters have their own description identification information. In this scenario, the question identification information of the first candidate feature question can be compared with each description identification information to obtain the description identification information that matches the question identification information, and the feature description indicated by the description identification information can be determined as the first candidate feature description of the first candidate feature question.

[0053] Optionally, a set of candidate feature question-answer pairs for each candidate image is obtained based on each first candidate feature question and the first candidate feature description of each first candidate feature question.

[0054] In the disclosed embodiment, there may be multiple first candidate feature questions under the candidate image, wherein the multiple first candidate feature questions and the first candidate feature descriptions of the multiple first candidate feature questions may be paired respectively, and the information pair consisting of the first candidate feature question and its corresponding first candidate feature description is marked as a candidate feature question and answer pair.

[0055] Furthermore, a set consisting of candidate feature question-answer pairs of each of the multiple first candidate feature questions is marked as a candidate feature question-answer pair set of the candidate image, thereby obtaining a candidate feature question-answer pair set of each candidate image.

[0056] As another possible implementation, candidate indexes of the candidate feature description clusters may be obtained, and a target index matching the search statement may be obtained from the candidate indexes.

[0057] In the embodiment of the present disclosure, if there is index information in the candidate feature description cluster, the index information may be marked as a candidate index in the candidate feature description cluster.

[0058] In this scenario, based on the matching algorithm in the relevant technology, the search statement and each index information in the candidate index can be algorithmically processed to obtain the matching parameters of the search statement and each index information, and then the index information matching the search statement is obtained from each index information according to the matching parameters and marked as the target index.

[0059] It should be noted that a search statement may carry multiple search requirements of a user. In this scenario, the target index of the search statement in the candidate index may include multiple index information.

[0060] Optionally, a candidate image set associated with the target index is obtained from the candidate image cluster.

[0061] In the disclosed embodiment, the index information in the target index stores images associated with it in the candidate image cluster, wherein the images associated with the index information in the candidate image cluster may be marked as candidate images of the target index.

[0062] The index information includes index identification information, and the index identification information can be compared with the image identification information of each image in the candidate image cluster to obtain a candidate image set of the target index.

[0063] Optionally, a second candidate feature question set associated with the target index is obtained from the candidate feature question cluster, and a second candidate feature description of each second candidate feature question in the second candidate feature question set is obtained from the candidate feature description cluster.

[0064] In the disclosed embodiment, the index identification information included in the index information of the target index can be compared with the question identification information of each candidate feature question cluster, and then some candidate feature questions associated with the target index can be obtained from the candidate feature question cluster, and the set consisting of these some candidate feature questions can be marked as the second candidate feature question set.

[0065] Accordingly, the question identification information of each of the second candidate feature question sets can be compared with the description identification information of each of the candidate feature description clusters, and then the candidate feature descriptions of each second candidate feature question in the second candidate feature question set can be obtained from the candidate description question cluster and marked as the second candidate feature descriptions of each second candidate feature question.

[0066] Optionally, a set of candidate feature question-answer pairs for each candidate image is obtained based on each second candidate feature question and the candidate feature description of each second candidate feature question.

[0067] In the disclosed embodiment, there may be multiple second candidate feature questions under the candidate image, wherein the multiple second candidate feature questions and the second candidate feature descriptions of the multiple second candidate feature questions may be paired respectively, and the information pair consisting of the second candidate feature question and its corresponding second candidate feature description is marked as a candidate feature question and answer pair.

[0068] Furthermore, a set consisting of candidate feature question-answer pairs of each of the multiple second candidate feature questions is marked as a candidate feature question-answer pair set of the candidate image, thereby obtaining a candidate feature question-answer pair set of each candidate image.

[0069] S202, for any candidate image, input the search sentence and the candidate feature question-answer pair set of the candidate image into the big model, and obtain the target feature question-answer pair set associated with the search sentence in the candidate feature question-answer pair set through the big model.

[0070] In the disclosed embodiment, a large model can be used to obtain some feature question-answer pairs that are associated with the search statement from the candidate feature question-answer pair set of the candidate image, thereby obtaining a target feature question-answer pair set for the candidate image.

[0071] Optionally, for any candidate image, the set of candidate feature question and answer pairs and the search statement under the candidate image can be input into the big model, and the degree of association between each candidate feature question and answer pair set and the search statement can be obtained through the model capability of the big model, and then some feature question and answer pairs whose association degrees meet the preset association degree conditions are screened out from the candidate feature question and answer pair set to form the target feature question and answer pair set of the candidate image.

[0072] S203, obtaining a target search image of the search statement from each candidate image according to the target feature question-answer pair set.

[0073] Optionally, the number of elements in the target feature question-answer pair set of each candidate image is obtained.

[0074] In the disclosed embodiment, for any candidate image, the number of question-answer pairs in the target feature question-answer pair of the candidate image may be marked as the number of elements in the target feature question-answer pair set.

[0075] As a possible implementation manner, for any candidate image, in response to the number of elements being greater than or equal to a preset element number threshold, the candidate image is determined to be a target search image of the search statement.

[0076] In the disclosed embodiment, the number of elements in each target feature question-answer pair set can be compared with a preset element number threshold. For any candidate image, when the number of elements in the target feature question-answer pair set of the candidate image is greater than or equal to the element number threshold, it can be determined that the degree of adaptation between the candidate image and the search statement meets the preset conditions, and further it can be determined that the candidate image may be the required image of the search statement.

[0077] Further, the candidate image may be determined as a target search image of the search sentence.

[0078] As another possible implementation method, the number of elements in each target feature question-answer pair set is sorted from most to least to obtain a sorted candidate image queue, and a preset number of candidate images are obtained starting from the first one in the candidate image queue as target search images for the search statement.

[0079] In the disclosed embodiment, the number of elements in each target feature question and answer pair set can be sorted based on the order of most to least, to obtain a queue composed of each target feature question and answer pair set, and then the candidate images can be sorted according to the queue sorting of each target feature question and answer pair set in the queue, to obtain a sorted candidate image queue.

[0080] Optionally, there is a preset number of images returned to the user terminal that inputs the search statement, and starting from the first position in the candidate image queue, some candidate images of this number are obtained and determined as target search images of the search statement.

[0081] The image search method based on a large model proposed in the present disclosure obtains a set of candidate images of a search statement and a set of candidate feature question-answer pairs of each candidate image from a candidate image information library, obtains a set of target feature question-answer pairs of each candidate image from the set of candidate feature question-answer pairs of each candidate image through a large model, and then determines a target search image of the search statement from each candidate image according to each target feature question-answer pair set. In the present disclosure, a set of target feature question-answer pairs of a search statement under each candidate image is obtained through a large model, which improves the efficiency and accuracy of obtaining the set of target feature question-answer pairs, and determines the target search image from each candidate image according to the number of elements in the set of target feature question-answer pairs, which improves the accuracy of obtaining the target search image and optimizes the degree of adaptation between the target search image and the search statement.

[0082] In the above embodiment, the acquisition of the candidate image information library can be combined with Figure 3 Further understanding, Figure 3 FIG. 1 is a flow chart of an image search method based on a large model according to another embodiment of the present disclosure. Figure 3 As shown, the method includes:

[0083] S301, obtaining a user input object and an initial image cluster, an initial feature question cluster and an initial feature description cluster in an initial image information library.

[0084] In the disclosed embodiment, the image information library to be updated may be marked as the initial image information library, wherein the acquisition of the candidate image information library may be achieved by updating the initial image information library.

[0085] The cluster storing images in the initial image information library may be marked as an initial image cluster, the cluster storing feature questions may be marked as an initial feature question cluster, and the cluster storing feature descriptions may be marked as an initial feature description cluster.

[0086] In this scenario, the user's input information may be obtained and marked as a user input object, wherein the user input object may be text or an image, which is not specifically limited here.

[0087] S302, in response to the user input object being a user input sentence, extracting a first updated feature description of the user input sentence in each initial image of the initial image set through a large model to update the initial image information library to obtain an updated candidate image information library.

[0088] In the disclosed embodiment, when the user input object is text, the sentence input by the user can be marked as a user input sentence, wherein the user input sentence can be input into a big model, and the updated search sentence corresponding to the user input sentence can be obtained through the big model.

[0089] Optionally, the large model may be used to identify whether the user input sentence is a question-type sentence.

[0090] In the disclosed embodiment, the sentence type of the user input sentence can be identified through the model capability of the large model, thereby determining whether the user input sentence is a question-type sentence.

[0091] Among them, the punctuation marks in the user input sentence can be used to identify the question-type sentence through the model capability of the large model, and the question-type sentence can also be identified through the semantic information of the user input sentence.

[0092] Optionally, in response to the large model recognizing that the user input statement is a question-type statement, it is determined that the user input statement is an update search statement.

[0093] It can be understood that, in a scenario where the large model recognizes that the user input sentence is a question-type sentence, the user input sentence can be directly used as a question-type sentence used in image search and marked as an updated search sentence.

[0094] Optionally, in response to the big model recognizing that the user input sentence is not a question-style sentence, the big model rewrites the user input sentence into a question-style sentence to obtain a rewritten updated search sentence.

[0095] It can be understood that when the big model recognizes that the user input sentence is not a question-type sentence, the user input sentence is rewritten through the model capabilities of the big model, wherein the user input sentence can be rewritten in a question-type based on the relevant restriction conditions of the question-type sentence, thereby rewriting the user input sentence from other types to a question-type type, and marking the rewritten sentence as an updated search sentence.

[0096] It should be noted that after rewriting the user input sentence, the large model can return the rewritten sentence to the user end for confirmation. After receiving the confirmation instruction input by the user end, the rewritten question-like sentence can be determined as the updated search sentence used for image search.

[0097] In the disclosed embodiment, each initial image and an update search statement may be input into a large model, and the large model may output a first update feature description of the update search statement in each initial image.

[0098] Among them, the feature description of the updated search statement in each initial image can be extracted through the model capability of the large model, and the extracted feature description can be marked as the first updated feature description of the updated search statement in each initial image.

[0099] Optionally, for any initial image, a first association index is constructed between the initial image, the updated search statement and the corresponding first updated feature description, and the initial feature question cluster is updated according to the updated search statement, the initial feature description cluster is updated according to the first updated feature description, and the first initial index in the initial feature description cluster is updated according to the first association index, so as to update the initial image information library to obtain an updated candidate image information library.

[0100] In the embodiment of the present disclosure, the initial image information library includes an initial feature description cluster, wherein index information in the initial feature description cluster can be marked as a first initial index.

[0101] In this scenario, the first updated feature description of the updated search statement under any initial image can be obtained, and based on the index construction method in the related technology, an index between the updated search statement, the initial image and the first updated feature description can be constructed and marked as the first associated index.

[0102] Furthermore, the first associated index can be added to the first initial index to update the first initial index, the updated search statement can be added to the initial feature question cluster in the initial image information library to update the initial feature question cluster, and the first updated feature description can be added to the initial feature description cluster to update the initial feature description cluster.

[0103] Furthermore, based on the updated initial feature question cluster, the updated initial feature description cluster and the updated first initial index, the initial image information library is updated, and the updated initial image information library is marked as a candidate image information library.

[0104] S303, in response to the user input object being a user input image, extracting a second updated feature description of each initial feature question in the initial feature question cluster in the user input image through the large model to update the initial image information library to obtain an updated candidate image information library.

[0105] Optionally, each initial feature question and the user input image are input into the large model, and the large model outputs a second updated feature description of each initial feature question in the updated input image.

[0106] In the disclosed embodiment, each initial feature question and the user input image can be respectively input into the large model, and the feature description of each initial feature question in the user input image can be extracted from the user input image through the model capability of the large model, thereby obtaining a second updated feature description of each initial feature question in the user input image.

[0107] Optionally, for any initial feature question, a second association index is constructed between the user input image, the initial feature question and the corresponding second updated feature description.

[0108] In the disclosed embodiment, a second updated feature description of any initial feature question in a user input image can be obtained, and according to an index construction method in related technology, index information between the initial feature question, the user input image and the second updated feature description can be constructed and marked as a second associated index.

[0109] Optionally, the initial image cluster is updated according to the user input image, the initial feature description cluster is updated according to the second updated feature description, and the second initial index in the initial feature description cluster is updated according to the second associated index to update the initial image information library to obtain an updated candidate image information library.

[0110] In the embodiment of the present disclosure, the user input image can be added to the initial image cluster to update the initial image cluster, and the second updated feature description can be added to the initial feature description cluster to update the initial feature description cluster, and the second associated index can be added to the second initial index in the initial feature description cluster to update the second initial index.

[0111] Furthermore, based on the updated initial image cluster, the updated initial feature description cluster and the updated second initial index, the initial image information library is updated to obtain an updated candidate image information library.

[0112] The image search method based on a large model proposed in the present disclosure obtains a user input object and an initial image information library. When the user input object is a user input sentence, an updated search sentence corresponding to the user input sentence is obtained through the large model, and the first updated feature description of the updated search sentence in each image of the initial image cluster is extracted through the large model, and the corresponding first associated index is constructed, and then the initial image information library is updated according to the updated search sentence, the first updated feature description and the first associated index to obtain a candidate image information library. Correspondingly, when the user input object is a user input image, the second updated feature description of each question in the initial feature question cluster in the user input image is proposed through the large model, and the corresponding second associated index is constructed, and then the initial image information library is updated according to the user input image, the second updated feature description and the second associated index to obtain a candidate image information library. In the present disclosure, the first updated feature description of the user input sentence in each image of the initial image cluster is extracted by a large model, and the second updated feature description of each question of the initial feature question cluster is extracted by a large model in the user input image, thereby improving the extraction efficiency and accuracy of the feature description, thereby improving the update efficiency of the candidate image information library, and optimizing the accuracy of the information in the candidate image information library. The non-question-type user input sentence is rewritten by the large model to obtain an updated search sentence without the need for the user to modify it by himself, thereby improving the rewriting efficiency and accuracy of the question-type sentence and optimizing the user experience.

[0113] Corresponding to the large model-based image search methods proposed in the above-mentioned embodiments, an embodiment of the present disclosure further proposes an image search device based on a large model. Since the large model-based image search device proposed in the embodiment of the present disclosure corresponds to the intentionality prediction + model training method proposed in the above-mentioned embodiments, the implementation method of the large model-based image search method mentioned above is also applicable to the large model-based image search device proposed in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0114] Figure 4 FIG. 1 is a schematic diagram of a structure of an image search device based on a large model according to an embodiment of the present disclosure. Figure 4 As shown, the image search device 400 based on the large model includes a first acquisition module 41, a second acquisition module 42, a third acquisition module 43 and a search module 44, wherein:

[0115] A first acquisition module 41 is used to acquire a search statement and a candidate image information library;

[0116] A second acquisition module 42 is used to acquire a candidate image set of the search statement and a candidate feature question-answer pair set of each candidate image in the candidate image set from the candidate image information library;

[0117] The third acquisition module 43 is used to acquire a target feature question-answer pair set for each candidate image from the candidate feature question-answer pair set of each candidate image through the pre-acquired large model;

[0118] The search module 44 is used to obtain a target search image of the search statement from each candidate image according to the target feature question-answer pair set.

[0119] In the disclosed embodiment, the second acquisition module 42 is also used to: acquire a candidate image cluster in the candidate image information library, and use all images in the candidate image cluster as a candidate image set; acquire a candidate feature question cluster and a candidate feature description cluster in the candidate image information library; acquire a first candidate feature question set associated with the search statement from the candidate feature question cluster, and acquire the first candidate feature description of each first candidate feature question in the first candidate feature question set from the candidate feature description cluster; obtain a candidate feature question and answer pair set for each candidate image based on each first candidate feature question and the first candidate feature description of each first candidate feature question.

[0120] In the disclosed embodiment, the second acquisition module 42 is also used to: obtain a candidate index of the candidate feature description cluster, and obtain a target index matching the search statement from the candidate index; obtain a set of candidate images associated with the target index from the candidate image cluster; obtain a second candidate feature question set associated with the target index from the candidate feature question cluster, and obtain a second candidate feature description of each second candidate feature question in the second candidate feature question set from the candidate feature description cluster; obtain a set of candidate feature question-answer pairs for each candidate image based on each second candidate feature question and the second candidate feature description of each second candidate feature question.

[0121] In the disclosed embodiment, the third acquisition module 43 is also used to: for any candidate image, input the search statement and the candidate feature question and answer pair set of the candidate image into the big model, and obtain the target feature question and answer pair set associated with the search statement in the candidate feature question and answer pair set through the big model.

[0122] In the disclosed embodiment, the search module 44 is also used to: obtain the number of elements in the target feature question-answer pair set of each candidate image; for any candidate image, in response to the number of elements being greater than or equal to a preset element number threshold, determine that the candidate image is the target search image of the search statement.

[0123] In the disclosed embodiment, the search module 44 is also used to: sort the number of elements in each target feature question-answer pair set from most to least to obtain a sorted candidate image queue, and obtain a preset number of candidate images starting from the first one in the candidate image queue as target search images for the search statement.

[0124] In the disclosed embodiment, the first acquisition module 41 is also used to: acquire the user input object and the initial image cluster, the initial feature question cluster and the initial feature description cluster in the initial image information library; in response to the user input object being a user input statement, extracting the first updated feature description of the user input statement in each initial image of the initial image set through the large model to update the initial image information library to obtain an updated candidate image information library; in response to the user input object being a user input image, extracting the second updated feature description of each initial feature question in the initial feature question cluster in the user input image through the large model to update the initial image information library to obtain an updated candidate image information library.

[0125] In the disclosed embodiment, the first acquisition module 41 is also used to: obtain an updated search statement corresponding to a user input statement through a large model; input each initial image and an updated search statement into the large model, and output a first updated feature description of the updated search statement in each initial image through the large model; for any initial image, construct a first association index between the initial image, the updated search statement and the corresponding first updated feature description; update the initial feature question cluster according to the updated search statement, update the initial feature description cluster according to the first updated feature description, and update the first initial index in the initial feature description cluster according to the first association index, so as to update the initial image information library to obtain an updated candidate image information library.

[0126] In the disclosed embodiment, the first acquisition module 41 is also used to: identify whether the user input statement is a question-style statement through the big model; in response to the big model identifying that the user input statement is a question-style statement, determine that the user input statement is an update search statement; in response to the big model identifying that the user input statement is not a question-style statement, rewrite the user input statement in a question-style manner through the big model to obtain a rewritten update search statement.

[0127] In the disclosed embodiment, the first acquisition module 41 is further used to: input each initial feature question and the user input image into the large model, and output the second updated feature description of each initial feature question in the user input image through the large model; for any initial feature question, construct a second association index between the user input image, the initial feature question and the corresponding second updated feature description; update the initial image cluster according to the user input image, update the initial feature description cluster according to the second updated feature description, and update the second initial index in the initial feature description cluster according to the second association index, so as to update the initial image information library to obtain an updated candidate image information library.

[0128] The image search device based on a large model proposed in the present disclosure obtains a search statement, and obtains a candidate image set of the search statement and a candidate feature question-answer pair set of each candidate image from a candidate image information library, obtains a target feature question-answer pair set of each candidate image from each candidate feature question-answer pair set through a large model, and then obtains a target search image of the search statement from the candidate image set according to the target feature question-answer pair set of each candidate image. In the present disclosure, the target feature question-answer pair set of the search statement under each candidate image is obtained through a large model, which improves the acquisition efficiency and accuracy of the target feature question-answer pair set. Compared with the image search based on the user demand conversion vector and the image conversion vector in the related art, the target search image is obtained based on the target feature question-answer pair set of the search statement under the candidate image through the large model, which avoids the influence of the loss of image details caused by the feature loss that may occur in the vector conversion process on the search results, improves the acquisition efficiency and accuracy of the target search image, reduces the operation difficulty of the image search method based on the large model, improves the practicality and applicability of the image search method based on the large model, and optimizes the user's image search experience.

[0129] According to an embodiment of the present disclosure, the present disclosure also proposes an electronic device, a readable storage medium, and a computer program product.

[0130] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0131] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 509 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0132] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 506, such as various types of displays, speakers, etc.; a storage unit 509, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a large model-based image search method. For example, in some embodiments, the large model-based image search method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 509. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the large model-based image search method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform a large model-based image search method in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be presented to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To propose interactions with a user account, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can propose input to the computer. Other types of devices may also be used to propose interactions with the user account; for example, feedback proposed to the user account may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account may be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user account computer with a graphical user account interface or a web browser through which a user account can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0139] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0141] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A large model-based image search method, wherein: The method comprises: Obtaining a search statement and a candidate image information library; Acquire a candidate image cluster in the candidate image information library, and use all images in the candidate image cluster as a candidate image set; Acquire a candidate feature question cluster and a candidate feature description cluster in the candidate image information library; Acquire a first candidate feature question set associated with the search statement from the candidate feature question cluster, and acquire a first candidate feature description of each first candidate feature question in the first candidate feature question set from the candidate feature description cluster; According to each first candidate feature question and each first candidate feature description of each first candidate feature question, a candidate feature question-answer pair set of each candidate image is obtained; Obtain a target feature question-answer pair set for each candidate image from the candidate feature question-answer pair set of each candidate image through the pre-acquired large model; According to the target feature question-answer pair set, a target search image of the search sentence is obtained from each candidate image.

2. A large model-based image search method, wherein: The method comprises: Obtaining a search statement and a candidate image information library; Obtaining a candidate index of a candidate feature description cluster, and obtaining a target index matching the search statement from the candidate index; Obtaining a set of candidate images associated with the target index from the candidate image cluster; Acquire a second candidate feature question set associated with the target index from the candidate feature question cluster, and acquire a second candidate feature description of each second candidate feature question in the second candidate feature question set from the candidate feature description cluster; According to each second candidate feature question and each second candidate feature description of each second candidate feature question, a candidate feature question-answer pair set of each candidate image is obtained; Obtain a target feature question-answer pair set for each candidate image from the candidate feature question-answer pair set of each candidate image through the pre-acquired large model; According to the target feature question-answer pair set, a target search image of the search sentence is obtained from each candidate image.

3. The method according to claim 1 or 2, wherein: The method of obtaining a target feature question-answer pair set for each candidate image from a candidate feature question-answer pair set for each candidate image through the pre-acquired large model includes: For any candidate image, the search statement and the candidate feature question-answer pair set of the candidate image are input into the large model, and the target feature question-answer pair set associated with the search statement in the candidate feature question-answer pair set is obtained through the large model.

4. The method according to claim 1 or 2, wherein: The step of obtaining a target search image of the search statement from each candidate image according to the target feature question-answer pair set includes: Obtain the number of elements in the target feature question-answer pair set of each candidate image; For any candidate image, in response to the number of elements being greater than or equal to a preset element number threshold, the candidate image is determined to be the target search image of the search statement.

5. The method according to claim 4, wherein: The method further comprises: A sorted candidate image queue is obtained by sorting the number of elements in each target feature question-answer pair set from most to least, and a preset number of candidate images are obtained starting from the first position in the candidate image queue as the target search images of the search statement.

6. The method according to claim 1 or 2, wherein: The step of obtaining a candidate image information library comprises: Obtaining a user input object and an initial image cluster, an initial feature question cluster, and an initial feature description cluster in an initial image information library; In response to the user input object being a user input sentence, extracting a first updated feature description of the user input sentence in each initial image of the initial image set by the large model, so as to update the initial image information library and obtain the updated candidate image information library; In response to the user input object being a user input image, the large model is used to extract a second updated feature description of each initial feature question in the initial feature question cluster in the user input image to update the initial image information library to obtain the updated candidate image information library.

7. The method according to claim 6, wherein: In response to the user input object being a user input sentence, extracting, by the large model, a first updated feature description of the user input sentence in each initial image of the initial image set to update the initial image information library to obtain the updated candidate image information library, including: Obtaining, through the large model, an updated search statement corresponding to the user input statement; Inputting each initial image and the update search statement into the large model, and outputting a first update feature description of the update search statement in each initial image through the large model; For any initial image, construct a first association index between the initial image, the update search statement and the corresponding first update feature description; The initial feature question cluster is updated according to the updated search statement, the initial feature description cluster is updated according to the first updated feature description, and the first initial index in the initial feature description cluster is updated according to the first associated index, so as to update the initial image information library to obtain the updated candidate image information library.

8. The method according to claim 7, wherein: The step of obtaining, through the large model, an updated search statement corresponding to the user input statement includes: Identify whether the user input sentence is a question sentence by using the large model; In response to the large model recognizing that the user input sentence is a question sentence, determining that the user input sentence is the update search sentence; In response to the large model recognizing that the user input sentence is not a question-style sentence, the user input sentence is rewritten in a question-style manner by the large model to obtain the rewritten updated search sentence.

9. The method according to claim 6, wherein: In response to the user input object being a user input image, extracting, by the large model, a second updated feature description of each initial feature question in the initial feature question cluster in the user input image, so as to update the initial image information library to obtain the updated candidate image information library, comprising: Inputting each initial feature question and the user input image into the large model, and outputting a second updated feature description of each initial feature question in the user input image through the large model; For any initial feature question, construct a second association index between the user input image, the initial feature question and the corresponding second updated feature description; The initial image cluster is updated according to the user input image, the initial feature description cluster is updated according to the second updated feature description, and the second initial index in the initial feature description cluster is updated according to the second associated index, so as to update the initial image information library to obtain the updated candidate image information library.

10. An image search device based on a large model, wherein: The device comprises: A first acquisition module, used to acquire a search statement and a candidate image information library; The second acquisition module is used to acquire a candidate image cluster in the candidate image information library, and take all images in the candidate image cluster as a candidate image set; acquire a candidate feature question cluster and a candidate feature description cluster in the candidate image information library; acquire a first candidate feature question set associated with the search statement from the candidate feature question cluster, and acquire a first candidate feature description of each first candidate feature question in the first candidate feature question set from the candidate feature description cluster; obtain a candidate feature question-answer pair set for each candidate image according to each first candidate feature question and the first candidate feature description of each first candidate feature question; A third acquisition module is used to acquire a target feature question-answer pair set for each candidate image from the candidate feature question-answer pair set of each candidate image through a pre-acquired large model; The search module is used to obtain the target search image of the search statement from each candidate image according to the target feature question-answer pair set.

11. An image search device based on a large model, wherein: The device comprises: A first acquisition module, used to acquire a search statement and a candidate image information library; A second acquisition module is used to obtain a candidate index of a candidate feature description cluster, and obtain a target index matching the search statement from the candidate index; obtain a candidate image set associated with the target index from the candidate image cluster; obtain a second candidate feature question set associated with the target index from the candidate feature question cluster, and obtain a second candidate feature description of each second candidate feature question in the second candidate feature question set from the candidate feature description cluster; obtain a candidate feature question-answer pair set for each candidate image based on each second candidate feature question and the second candidate feature description of each second candidate feature question; A third acquisition module is used to acquire a target feature question-answer pair set for each candidate image from the candidate feature question-answer pair set of each candidate image through a pre-acquired large model; The search module is used to obtain the target search image of the search statement from each candidate image according to the target feature question-answer pair set.

12. The device according to claim 10 or 11, wherein: The third acquisition module is further used for: For any candidate image, the search statement and the candidate feature question-answer pair set of the candidate image are input into the large model, and the target feature question-answer pair set associated with the search statement in the candidate feature question-answer pair set is obtained through the large model.

13. The device according to claim 10 or 11, wherein: The search module is also used for: Obtain the number of elements in the target feature question-answer pair set of each candidate image; For any candidate image, in response to the number of elements being greater than or equal to a preset element number threshold, the candidate image is determined to be the target search image of the search statement.

14. The device according to claim 13, wherein: The search module is also used for: A sorted candidate image queue is obtained by sorting the number of elements in each target feature question-answer pair set from most to least, and a preset number of candidate images are obtained starting from the first position in the candidate image queue as the target search images of the search statement.

15. The device according to claim 10 or 11, wherein: The first acquisition module is further used for: Obtaining a user input object and an initial image cluster, an initial feature question cluster, and an initial feature description cluster in an initial image information library; In response to the user input object being a user input sentence, extracting a first updated feature description of the user input sentence in each initial image of the initial image set by the large model, so as to update the initial image information library and obtain the updated candidate image information library; In response to the user input object being a user input image, the large model is used to extract a second updated feature description of each initial feature question in the initial feature question cluster in the user input image to update the initial image information library to obtain the updated candidate image information library.

16. The device according to claim 15, wherein: The first acquisition module is further used for: Obtaining, through the large model, an updated search statement corresponding to the user input statement; Inputting each initial image and the update search statement into the large model, and outputting a first update feature description of the update search statement in each initial image through the large model; For any initial image, construct a first association index between the initial image, the update search statement and the corresponding first update feature description; The initial feature question cluster is updated according to the updated search statement, the initial feature description cluster is updated according to the first updated feature description, and the first initial index in the initial feature description cluster is updated according to the first associated index, so as to update the initial image information library to obtain the updated candidate image information library.

17. The device according to claim 16, wherein: The first acquisition module is further used for: Identify whether the user input sentence is a question sentence by using the large model; In response to the large model recognizing that the user input sentence is a question sentence, determining that the user input sentence is the update search sentence; In response to the large model recognizing that the user input sentence is not a question-style sentence, the user input sentence is rewritten in a question-style manner by the large model to obtain the rewritten updated search sentence.

18. The device according to claim 15, wherein: The first acquisition module is further used for: Inputting each initial feature question and the user input image into the large model, and outputting a second updated feature description of each initial feature question in the user input image through the large model; For any initial feature question, construct a second association index between the user input image, the initial feature question and the corresponding second updated feature description; The initial image cluster is updated according to the user input image, the initial feature description cluster is updated according to the second updated feature description, and the second initial index in the initial feature description cluster is updated according to the second associated index, so as to update the initial image information library to obtain the updated candidate image information library.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images