Image-text retrieval method and device, equipment, storage medium and program product

Abstract

The invention provides an image-text retrieval method and device, equipment, a storage medium and a program product, and is applied to the technical field of computer vision. The method comprises the following steps: acquiring an image description text, and determining a candidate image set based on the image description text; dividing the candidate images in the candidate image set into a plurality of class clusters, and determining a representative image of each class cluster; generating N scene questions according to the representative image, and determining scene answers of the N scene questions in a form of interactive questions and answers with a user; generating a scene graph based on the scene question, the scene answer and the image description text; performing similarity retrieval based on the scene graph, the scene question, the scene answer and the image description text to obtain a target image; wherein N is an integer greater than 0.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a graphic and text retrieval method, device, equipment, storage medium and program product. Background Art

[0002] In the field of modern information retrieval, interactive image and text retrieval has shown unique advantages. It breaks the limitations of traditional retrieval methods, which are relatively static and lack of interaction. By actively using the text descriptions and related pictures entered by users, a question-and-answer interactive model is established. The significance of this interaction is that it is no longer a unilateral matching of images and texts according to established rules, but can dynamically adjust the retrieval direction and content based on the user's real-time feedback of ideas, requirements and other information, so that the final retrieval results can more closely fit the user's true needs.

[0003] However, the interactive image-text search methods in the prior art often rely on users to give feedback frequently and make adjustments. For example, users may need to modify text descriptions or add more relevant pictures many times before they can gradually approach the ideal search results. Such frequent interactions will slow down the pace of the entire question-answering process, making the question-answering efficiency low. Summary of the invention

[0004] The present invention provides a picture and text search method, device, equipment, storage medium and program product, which are used to solve the problem of low efficiency of question and answer in the interactive picture and text search mode in the prior art.

[0005] The present invention provides an image-text retrieval method, comprising: obtaining image description text, and determining a candidate image set based on the image description text; dividing the candidate images in the candidate image set into multiple clusters, and determining a representative image of each cluster; generating N scene questions according to the representative images, and determining scene answers to the N scene questions in the form of interactive questions and answers with users; generating a scene graph based on the scene questions, the scene answers and the image description text; performing similarity retrieval based on the scene graph, the scene questions, the scene answers and the image description text to obtain a target image; wherein N is an integer greater than 0.

[0006] According to a picture and text retrieval method provided by the present invention, the candidate images in the candidate image set are divided into multiple clusters, and the representative image of each cluster is determined, including: dividing the candidate images in the candidate image set into multiple clusters by a clustering algorithm; calculating the similarity distribution between the candidate images in the clusters and the image description texts; determining the candidate image with the lowest entropy value in the similarity distribution as the representative image of the corresponding cluster; wherein one cluster corresponds to one representative image.

[0007] According to a graphic and text retrieval method provided by the present invention, generating N scene questions based on the representative image includes: generating first text information based on the representative image, and generating M scene questions based on the first text information; filtering the scene questions that meet a first condition among the M scene questions to obtain the N scene questions; wherein the first condition includes: the answer is contained in the image description text, or is related to the candidate image but unrelated to the target image, and M is an integer greater than or equal to N.

[0008] According to a picture-text retrieval method provided by the present invention, the first text information is generated according to the representative image, and M scenario questions are generated based on the first text information, including: generating subtitles for the representative image through a subtitle generation model, and converting the generated subtitles into the first text information; generating M scenario questions related to the first text information through a large language model.

[0009] According to a graphic and text retrieval method provided by the present invention, the filtering of the scenario questions that meet the first condition among the M scenario questions to obtain the N scenario questions includes: automatically answering the M scenario questions through a large language model in combination with context descriptions, and filtering the scenario questions with answers; determining the similarity distribution of the context descriptions, determining the similarity distribution of the context descriptions combined with the M scenario questions, and filtering the scenario questions whose relative entropy in the similarity distribution is greater than a preset threshold.

[0010] According to a graphic and text retrieval method provided by the present invention, the target image is obtained by performing a similarity search based on the scene graph, the scene question, the scene answer and the image description text, including: generating a text in a title format according to the N scene questions and the scene answer, splicing the text in the title format and the image description text to obtain a second text information; splicing the scene graph and the second text information to obtain target graphic and text information; and performing a similarity search based on the target graphic and text information to obtain the target image.

[0011] The present invention also provides an image-text retrieval device, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire image description text; the processing module is used to determine a candidate image set based on the image description text; the candidate images in the candidate image set are divided into multiple clusters, and a representative image of each cluster is determined; N scene questions are generated according to the representative images; the acquisition module is used to determine the scene answers to the N scene questions in the form of interactive questions and answers with the user; the processing module is used to generate a scene graph based on the scene questions, the scene answers and the image description text; similarity retrieval is performed based on the scene graph, the scene questions, the scene answers and the image description text to obtain a target image; wherein N is an integer greater than 0.

[0012] According to a graphic and text retrieval device provided by the present invention, the processing module is used to divide the candidate images in the candidate image set into multiple clusters through a clustering algorithm; calculate the similarity distribution between the candidate images in the clusters and the image description texts; determine the candidate image with the lowest entropy value in the similarity distribution as the representative image of the corresponding cluster; wherein one cluster corresponds to one representative image.

[0013] According to a graphic and text retrieval device provided by the present invention, the processing module is used to generate first text information according to the representative image, and generate M scenario questions based on the first text information; filter the scenario questions that meet the first condition among the M scenario questions to obtain the N scenario questions; wherein the first condition includes: the answer is contained in the image description text, or is related to the candidate image but not to the target image, and M is an integer greater than or equal to N.

[0014] According to a graphic and text retrieval device provided by the present invention, the processing module is used to generate subtitles for the representative image through a subtitle generation model, and convert the generated subtitles into the first text information; and generate M scenario questions related to the first text information through a large language model.

[0015] According to a graphic and text retrieval device provided by the present invention, the processing module is used to automatically answer the M scenario questions through a large language model in combination with context descriptions, and filter the scenario questions with answers; determine the similarity distribution of the context descriptions, determine the similarity distribution of the context descriptions combined with the M scenario questions, and filter the scenario questions whose relative entropy in the similarity distribution is greater than a preset threshold.

[0016] According to a graphic and text retrieval device provided by the present invention, the processing module is used to generate a text in a title format according to the N scene questions and the scene answers, splice the text in the title format and the image description text to obtain second text information; splice the scene graph and the second text information to obtain target graphic and text information; and perform similarity retrieval based on the target graphic and text information to obtain the target image.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned image and text retrieval methods is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image-text retrieval method described above is implemented.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image-text retrieval method described above is implemented.

[0020] The image-text retrieval method, apparatus, device, storage medium and program product provided by the present invention, on the one hand, can ensure the diversity of source images of scene problems, thereby improving the information coverage of the scene graph, because the candidate images in the candidate image set can be divided into multiple clusters and the representative image of each cluster can be determined; on the other hand, since image-text retrieval can be performed based on the scene graph, not only the number of user interactions can be reduced, but also the rich object attribute information and relationship modeling of the scene graph can be utilized to enhance the information carrying capacity of the prompt text, thereby optimizing the efficiency and accuracy of image-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a schematic diagram of the flow of the image-text retrieval method provided by the present invention; Figure 2 It is a schematic diagram of the representation form of the scenario problem provided by the present invention; Figure 3 It is a schematic diagram of the representation form of the scene graph provided by the present invention; Figure 4 It is a structural schematic diagram of the image and text retrieval device provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0025] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0026] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and order of execution.

[0027] The embodiments of the present application describe some exemplary embodiments for the purpose of explanation. It should be understood that the present application can be implemented in other ways that are not specifically shown in the drawings.

[0028] like Figure 1 As shown, the embodiment of the present application provides a method for image-text retrieval, which can be applied to an image-text retrieval device. The image-text retrieval method may include S101-S105: S101: An image-text retrieval device obtains image description text, and determines a candidate image set based on the image description text.

[0029] The image description text is a description text of the image that the user wants to retrieve, which is input by the user.

[0030] Optionally, the image-text retrieval device may include the image description text Input retrieval model, the retrieval model can be based on image description text Search the image database to find a series of candidate images that may match the description and obtain a candidate image set .

[0031] S102: The image-text retrieval device divides the candidate images in the candidate image set into a plurality of clusters, and determines a representative image for each cluster.

[0032] Optionally, the image and text retrieval device can divide the candidate images in the candidate image set into multiple clusters through a clustering algorithm; calculate the similarity distribution between the candidate images in the cluster and the image description text; determine the candidate image with the lowest entropy value in the similarity distribution as the representative image of the corresponding cluster; wherein one cluster corresponds to one representative image.

[0033] Specifically, the image-text retrieval device can divide the candidate images in the candidate image set into multiple clusters by using the K-Means clustering algorithm, that is, candidate images with different distribution characteristics are in different clusters. Afterwards, the image-text retrieval device can resample the multiple clusters, that is, calculate the similarity distribution between the candidate images in the cluster and the image description text, and determine the candidate image with the lowest entropy value in the similarity distribution as the representative image of the corresponding cluster. The representative image has a more specific and clear object body, which can improve the quality of the image description and the relevance to the target image.

[0034] K-Means clustering is an unsupervised learning algorithm, which aims to divide candidate images into different clusters so that the similarity between candidate images in the same cluster is as high as possible, while the similarity between different clusters is as low as possible.

[0035] It should be noted that the lower the entropy of the similarity distribution, the more concentrated and certain the distribution is. For example, if the similarity between the candidate images in a cluster and the user description is mostly high with only a few fluctuations, then the entropy of the similarity distribution will be relatively low. Such a candidate image is selected as the representative image because it best represents the features similar to the user description in this cluster, and it is most consistent with other images in terms of similarity to the user description.

[0036] S103. The image-text retrieval device generates N scenario questions according to the representative image, and determines scenario answers to the N scenario questions by interactive question-and-answer session with the user.

[0037] Wherein, N is an integer greater than 0.

[0038] Optionally, the image and text retrieval device can generate first text information based on the representative image, and generate M scenario questions based on the first text information; filter the scenario questions that meet the first condition among the M scenario questions to obtain the N scenario questions; wherein the first condition includes: the answer is contained in the image description text, or is related to the candidate image but not to the target image, and M is an integer greater than or equal to N.

[0039] Specifically, the image-text retrieval device can generate subtitles for the representative image through a subtitle generation model, and convert the generated subtitles into the first text information; generate M scene questions related to the first text information through a large language model, that is, use the large language model, through a chain of thought, constrain the language model to use diverse subtitles to adopt logical reasoning to generate relevant scene questions, including object subjects, relationships and attribute information, etc. After determining the M scene questions, the image-text retrieval device can use the large language model and combine the context description to automatically answer the M scene questions, and filter the scene questions that have answers; it can also determine the similarity distribution of the context description, determine the similarity distribution of the context description combined with the M scene questions, filter the scene questions whose relative entropy in the similarity distribution is greater than a preset threshold, and finally obtain N scene questions. Figure 2 Shown is an exemplary representation of a scenario problem.

[0040] It should be noted that for scene problems, the ideal situation is that it can cover all the scene information contained in the image. Only in this way can the image-related content be fully mined and assist subsequent operations such as image and text retrieval. The resampling operation in the above steps can make the scene problem meet this richness requirement.

[0041] At the same time, scenario questions should also have a certain degree of difficulty, that is, necessity. They should not be simple questions that can be easily answered based on previously existing text descriptions. Because questions that are too simple and lack challenge may not be able to truly dig deep into the complex information behind the image, it is not conducive to improving the quality and effect of image and text retrieval. Therefore, it is necessary to adopt an online automatic filtering method to screen relevant questions. The specific approach is to use a large language model and combine it with context descriptions to try to automatically answer scenario questions. If a question can be answered directly from the existing text description, it means that it does not meet the necessity requirements, so it will be filtered out, leaving only those questions that cannot be directly answered from the existing text description, so as to ensure that the scenario questions have sufficient depth and mining value.

[0042] In addition, there is a potential risk in this process, that is, there may be some questions that are related to the candidate image but are not actually closely related to the target image. In order to avoid this situation, this application adopts a filtering method based on KL divergence (Kullback-Leibler Divergence). The appropriateness of the question is judged by comparing the KL divergence in the filtered context dialogue similarity distribution and the context dialogue similarity distribution combined with the question. The KL divergence measures the difference between two probability distributions. If the KL divergence corresponding to a question is high, it means that the context dialogue similarity distribution involved is very different from the ideal context dialogue similarity distribution related to the target image. It is likely that the problem is not strongly related to the target image, so this type of problem should be filtered out.

[0043] After the above filtering operation, we finally get a series of scene questions that contain multiple aspects of information such as object subject, relationship and attribute, and retain the richness and necessity of the questions, and these scene questions have a strong connection with the target image. Such a set of questions can better serve the subsequent image and text retrieval process based on scene graphs, etc., and help improve the accuracy, comprehensiveness and overall effect of retrieval, making it more in line with the needs of accurately obtaining relevant information of the target image in practical applications.

[0044] After determining N scene questions, the image and text retrieval device can present them to the user in the form of interactive questions and answers, and the user needs to rely on his or her own observation and cognition of the relevant features of the target image to answer these questions step by step to obtain scene answers to the N scene questions. The user's answers can further supplement the information about the target image.

[0045] S104: The image-text retrieval device generates a scene graph based on the scene question, the scene answer and the image description text.

[0046] After obtaining the scene answer, the image-text retrieval device can generate prompts based on the scene graph, combining the scene question, the corresponding scene answer, and the image description text as input information of the large language model to construct the scene graph.

[0047] like Figure 3 As shown, it is an exemplary representation of the scene graph. The scene graph can be in JSON (JavaScriptObject Notation) format. JSON is a lightweight data exchange format that has the advantages of being easy to read, write, and convenient for machine parsing and generation. Using this format to construct the scene graph is conducive to the subsequent storage, transmission, sharing and further processing of the scene graph between different systems and modules, so that the scene graph can be better utilized in the entire process of image and text retrieval or other related applications to achieve purposes such as improving retrieval accuracy and enriching the cognition of target images.

[0048] In this way, multi-channel data related to the target image can be fully integrated, making the content input into the large language model more comprehensive and detailed, covering all aspects from system-generated questions, user feedback to existing descriptions and existing scene graphs.

[0049] S105 , the image-text retrieval device performs a similarity search based on the scene graph, the scene question, the scene answer, and the image description text to obtain a target image.

[0050] Optionally, the image and text retrieval device can generate a text in a title format based on the N scene questions and the scene answers, splice the text in the title format and the image description text to obtain a second text information; splice the scene graph and the second text information to obtain target image and text information; perform a similarity search based on the target image and text information to obtain the target image.

[0051] Specifically, the large language models or text encoders currently used are often pre-trained on large-scale image and text datasets. In such a pre-training process, the large amount of text data that these models are exposed to may be presented more in the form of titles. Therefore, in the process of learning and summarizing features, the models gradually develop better processing capabilities and biases for title-format texts, which means that when processing texts in this format, they can more effectively extract useful information from them, thereby achieving better performance.

[0052] In view of the above model characteristics, the content obtained through the scene question dialogue can be integrated into a title format text, and then spliced ​​with the previously existing image description text. The second text information after integration and splicing can better fit the distribution of training data that the retrieval model is exposed to during the pre-training stage.

[0053] When searching, the scene graph and the second text information that has been reconstructed into the title format can be spliced ​​together and used as a whole to calculate the search similarity. By calculating their similarity with various types of pictures in the database, the most similar picture can be found. This most similar picture is the target image to be found. This method of combining the scene graph and the well-formatted text description makes full use of the information advantages of both aspects, and can more comprehensively and accurately measure the similarity between pictures, thereby improving the accuracy and effectiveness of the search, and ultimately achieving high-quality image and text retrieval and accurately obtaining the target image.

[0054] In the embodiments of the present application, on the one hand, since the candidate images in the candidate image set can be divided into multiple clusters and the representative image of each cluster can be determined, the diversity of the source images of the scene problem can be ensured, thereby improving the information coverage of the scene graph; on the other hand, since image and text retrieval can be performed based on the scene graph, not only the number of user interactions can be reduced, but also the rich object attribute information and relationship modeling of the scene graph can be utilized to enhance the information carrying capacity of the prompt text and optimize the efficiency and accuracy of image and text retrieval.

[0055] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0056] The image and text retrieval method provided in the embodiment of the present application can be executed by an image and text retrieval device or a control module for image and text retrieval in the image and text retrieval device. In the embodiment of the present application, the image and text retrieval device performing the image and text retrieval method is taken as an example to illustrate the image and text retrieval device provided in the embodiment of the present application.

[0057] It should be noted that the embodiment of the present application can divide the functional modules of the image and text retrieval device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0058] like Figure 4 As shown, an embodiment of the present application provides a graphic retrieval device 400. The graphic retrieval device 400 includes: an acquisition module 401 and a processing module 402. The acquisition module 401 can be used to acquire image description text; the processing module 402 is used to determine a candidate image set based on the image description text; divide the candidate images in the candidate image set into multiple clusters, and determine the representative image of each cluster; generate N scene questions according to the representative images; the acquisition module 401 is used to determine the scene answers to the N scene questions in the form of interactive questions and answers with the user; the processing module 402 is used to generate a scene graph based on the scene questions, the scene answers and the image description text; perform similarity retrieval based on the scene graph, the scene questions, the scene answers and the image description text to obtain a target image; wherein, N is an integer greater than 0.

[0059] Optionally, the processing module 402 is used to divide the candidate images in the candidate image set into multiple clusters through a clustering algorithm; calculate the similarity distribution between the candidate images in the clusters and the image description texts; and determine the candidate image with the lowest entropy value in the similarity distribution as the representative image of the corresponding cluster; wherein one cluster corresponds to one representative image.

[0060] Optionally, the processing module 402 is used to generate first text information according to the representative image, and generate M scenario questions based on the first text information; filter the scenario questions that meet the first condition among the M scenario questions to obtain the N scenario questions; wherein the first condition includes: the answer is contained in the image description text, or is related to the candidate image but not to the target image, and M is an integer greater than or equal to N.

[0061] Optionally, the processing module 402 is used to generate subtitles for the representative image through a subtitle generation model, and convert the generated subtitles into the first text information; and generate M scenario questions related to the first text information through a large language model.

[0062] Optionally, the processing module 402 is used to automatically answer the M scenario questions through a large language model and in combination with the context description, and filter the scenario questions with answers; determine the similarity distribution of the context description, determine the similarity distribution of the context description combined with the M scenario questions, and filter the scenario questions whose relative entropy in the similarity distribution is greater than a preset threshold.

[0063] Optionally, the processing module 402 is used to generate text in a title format based on the N scenario questions and the scenario answers, splice the text in the title format and the image description text to obtain second text information; splice the scene graph and the second text information to obtain target graphic information; perform similarity retrieval based on the target graphic information to obtain the target image.

[0064] In the embodiments of the present application, on the one hand, since the candidate images in the candidate image set can be divided into multiple clusters and the representative image of each cluster can be determined, the diversity of the source images of the scene problem can be ensured, thereby improving the information coverage of the scene graph; on the other hand, since image and text retrieval can be performed based on the scene graph, not only the number of user interactions can be reduced, but also the rich object attribute information and relationship modeling of the scene graph can be utilized to enhance the information carrying capacity of the prompt text and optimize the efficiency and accuracy of image and text retrieval.

[0065] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the image-text retrieval method, which includes: obtaining an image description text, and determining a candidate image set based on the image description text; dividing the candidate images in the candidate image set into a plurality of clusters, and determining a representative image of each cluster; generating N scene questions according to the representative images, and determining the scene answers of the N scene questions in the form of interactive questions and answers with the user; generating a scene graph based on the scene questions, the scene answers and the image description text; performing a similarity search based on the scene graph, the scene questions, the scene answers and the image description text to obtain a target image; wherein N is an integer greater than 0.

[0066] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0067] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image-text retrieval method provided by the above methods, which includes: obtaining image description text, and determining a candidate image set based on the image description text; dividing the candidate images in the candidate image set into multiple clusters, and determining a representative image for each cluster; generating N scene questions based on the representative images, and determining scene answers to the N scene questions in the form of interactive questions and answers with the user; generating a scene graph based on the scene questions, the scene answers and the image description text; performing similarity retrieval based on the scene graph, the scene questions, the scene answers and the image description text to obtain a target image; wherein N is an integer greater than 0.

[0068] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image-text retrieval method provided by the above-mentioned methods, the method comprising: obtaining image description text, and determining a candidate image set based on the image description text; dividing the candidate images in the candidate image set into multiple clusters, and determining a representative image for each cluster; generating N scene questions based on the representative images, and determining scene answers to the N scene questions in the form of interactive questions and answers with the user; generating a scene graph based on the scene questions, the scene answers and the image description text; performing similarity retrieval based on the scene graph, the scene questions, the scene answers and the image description text to obtain a target image; wherein N is an integer greater than 0.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image and text retrieval, characterized in that: include: Acquire image description text, and determine a candidate image set based on the image description text; Dividing the candidate images in the candidate image set into a plurality of clusters, and determining a representative image of each cluster; Generate N scenario questions according to the representative image, and determine scenario answers to the N scenario questions by interactive question-and-answer session with the user; Generate a scene graph based on the scene question, the scene answer, and the image description text; Perform similarity retrieval based on the scene graph, the scene question, the scene answer, and the image description text to obtain a target image; Wherein, N is an integer greater than 0.

2. The image-text retrieval method according to claim 1, characterized in that: The step of dividing the candidate images in the candidate image set into a plurality of clusters and determining a representative image of each cluster includes: Dividing the candidate images in the candidate image set into a plurality of clusters by a clustering algorithm; Calculating the similarity distribution between the candidate images in the cluster and the image description text; The candidate image with the lowest entropy value in the similarity distribution is determined as the representative image of the corresponding cluster; Among them, one cluster corresponds to one representative image.

3. The image-text retrieval method according to claim 1, characterized in that: The generating N scene problems according to the representative image includes: Generate first text information according to the representative image, and generate M scenario questions based on the first text information; Filtering the scenario questions that meet the first condition among the M scenario questions to obtain the N scenario questions; The first condition includes: the answer is contained in the image description text, or is related to the candidate image but unrelated to the target image, and M is an integer greater than or equal to N.

4. The image-text retrieval method according to claim 3, characterized in that: The step of generating first text information according to the representative image, and generating M scenario questions based on the first text information, includes: Generate subtitles for the representative image using a subtitle generation model, and convert the generated subtitles into the first text information; M scenario questions related to the first text information are generated through a large language model.

5. The image-text retrieval method according to claim 3, characterized in that: The filtering the scenario problems satisfying the first condition among the M scenario problems to obtain the N scenario problems includes: Automatically answer the M scenario questions through a large language model and in combination with context descriptions, and filter the scenario questions that have been answered; Determine the similarity distribution of the context description, determine the similarity distribution of the context description combined with the M scenario questions, and filter out scenario questions whose relative entropy in the similarity distribution is greater than a preset threshold.

6. The image-text retrieval method according to claim 1, characterized in that: The performing similarity retrieval based on the scene graph, the scene question, the scene answer and the image description text to obtain a target image includes: Generate a text in a title format according to the N scenario questions and the scenario answers, and perform splicing processing on the text in the title format and the image description text to obtain second text information; Performing splicing processing on the scene graph and the second text information to obtain target graphic and text information; A similarity search is performed based on the target image and text information to obtain the target image.

7. A graphic and text retrieval device, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire image description text; The processing module is used to determine a candidate image set based on the image description text; divide the candidate images in the candidate image set into a plurality of clusters, and determine a representative image of each cluster; Generate N scenario questions according to the representative image; The acquisition module is used to determine the scenario answers to the N scenario questions in the form of interactive question-and-answer with the user; The processing module is used to generate a scene graph based on the scene question, the scene answer and the image description text; perform similarity retrieval based on the scene graph, the scene question, the scene answer and the image description text to obtain a target image; Wherein, N is an integer greater than 0.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image and text retrieval method as claimed in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image-text retrieval method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image-text retrieval method according to any one of claims 1 to 6 is implemented.

Images