Information retrieval method and device, equipment, medium and product

Abstract

The invention provides an information retrieval method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence such as deep learning, natural language processing, generative models and large language models. The method comprises the steps that picture demand information corresponding to a to-be-retrieved question is determined, and the picture demand information is used for representing the demand level of the to-be-retrieved question for picture answers; determining an answer type corresponding to the to-be-retrieved question based on the graph demand information; and performing retrieval in a data set corresponding to the answer type, and generating answer information of the to-be-retrieved question based on a retrieval result. The invention provides an information retrieval method. According to the method, the answer is answered in a mode corresponding to the requirement of the question, so that the answer content is more targeted, the answer quality is improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of artificial intelligence technology such as deep learning, natural language recognition, generative models, large language models, and more particularly to an information retrieval method, apparatus, device, medium, and product. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and big data technology, more and more users prefer to use the terminal's search engine to search for answers when they encounter problems. The answers obtained are generally in text form. For example, when users encounter medical and health problems, they will first choose to search online to obtain relevant medical and health science content. Summary of the invention

[0003] The present disclosure provides an information retrieval method, apparatus, device, medium and product.

[0004] According to a first aspect of the present disclosure, there is provided an information retrieval method, comprising: determining graph requirement information corresponding to a question to be retrieved, wherein the graph requirement information is used to characterize the level of requirement for image-type answers to the question to be retrieved; determining the type of answer corresponding to the question to be retrieved based on the graph requirement information; searching in a data set corresponding to the answer type, and generating answer information for the question to be retrieved based on the search results.

[0005] According to a second aspect of the present disclosure, there is provided an information retrieval device, comprising: a requirement determination module, configured to determine graph requirement information corresponding to a question to be retrieved, wherein the graph requirement information is used to characterize the requirement level of the question to be retrieved for image-type answers; a type determination module, configured to determine the type of answer corresponding to the question to be retrieved based on the graph requirement information; and an answer generation module, configured to search in a data set corresponding to the answer type, and generate answer information for the question to be retrieved based on the search results.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner in the first aspect.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method described in any implementation manner in the first aspect.

[0009] The information retrieval method provided by the disclosed embodiment first determines the image requirement information corresponding to the question to be retrieved, wherein the image requirement information is used to characterize the level of requirement for the image-type answer of the question to be retrieved; then determines the type of answer corresponding to the question to be retrieved based on the image requirement information; finally, searches in the data set corresponding to the answer type, and generates the answer letter of the question to be retrieved based on the search results. The information retrieval method in this embodiment determines the corresponding answer type according to the image requirement of the question to be retrieved, and retrieves the final answer in the data set of the answer type, thereby answering in a manner corresponding to the requirement of the question, making the answer content more targeted, improving the quality of the answer, and also improving the efficiency of information retrieval and the accuracy of the retrieval results, thereby improving the user experience.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0013] Figure 2 is a flow chart of a first embodiment of an information retrieval method according to the present disclosure;

[0014] Figure 3 is a flow chart of a second embodiment of the information retrieval method according to the present disclosure;

[0015] Figure 4 is a flow chart of a third embodiment of the information retrieval method according to the present disclosure;

[0016] Figure 5 is a flowchart of a fourth embodiment of the information retrieval method according to the present disclosure;

[0017] Figure 6 is a flowchart of a fifth embodiment of the information retrieval method according to the present disclosure;

[0018] Figure 7 is a flowchart of a sixth embodiment of the information retrieval method according to the present disclosure;

[0019] Figure 8is a schematic diagram of an application scenario of the information retrieval method according to the present disclosure;

[0020] Fig. 9 is a schematic structural diagram of an embodiment of an information retrieval device according to the present disclosure;

[0021] Fig.10 It is a block diagram of an electronic device used to implement the information retrieval method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the information retrieval method or information retrieval device of the present disclosure can be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send information, etc. Various client applications can be installed on the terminal devices 101, 102, 103.

[0027] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the above electronic devices. They may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0028] The server 105 can provide various services. For example, the server 105 can analyze and process the questions to be retrieved obtained from the terminal devices 101, 102, and 103, and generate processing results (such as answer information).

[0029] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0030] It should be noted that the information retrieval method provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the information retrieval device is generally disposed in the server 105 .

[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0032] Continue to refer Figure 2 , which shows a process 200 of a first embodiment of an information retrieval method according to the present disclosure. The information retrieval method comprises the following steps:

[0033] Step 201, determine the graph requirement information corresponding to the question to be searched.

[0034] In this embodiment, the execution subject of the information retrieval method (for example Figure 1 The server 105 shown in the figure will determine the image requirement information corresponding to the question to be retrieved, wherein the image requirement information is used to characterize the requirement level of the question to be retrieved for the image-type answer.

[0035] The above-mentioned execution entity will first obtain the questions input by the user on the terminal device, that is, the questions to be searched. The questions here can be questions about various aspects, such as medical questions, common sense questions, etc.

[0036] After obtaining the query to be searched input by the user, the above-mentioned execution subject will identify the query to be searched to determine the image requirement information corresponding to the query to be searched. The image requirement information here refers to the image requirement of the answer corresponding to the query to be searched, that is, whether the answer needs an image to supplement the explanation, so as to make the answer clearer, direct and easy to understand. Since the existing search results are generally in text form, the above-mentioned execution subject will determine whether the query to be searched requires an answer in the form of both pictures and text.

[0037] Specifically, the execution subject may first perform semantic analysis on the query to be searched to determine the graph requirement information corresponding to the query to be searched. Alternatively, the execution subject may also use a pre-trained requirement recognition model to recognize the query to be searched to determine the graph requirement information corresponding to the query to be searched.

[0038] In some optional implementations of this embodiment, the graph demand information includes: strong demand, weak demand or no demand.

[0039] That is, in this implementation, the graph requirements are pre-set into three levels, namely strong requirements, weak requirements and no requirements. For example, the user's question is "What does a boil look like?" In this case, it may be necessary to describe it through pictures, which is more intuitive and difficult to describe through text. In this case, it can be determined that the graph requirement corresponding to the question is a strong requirement. For another example, the user's question is "Symptoms of amoxicillin allergy". In this case, it can be described in words, such as a rash on the skin, or with relevant symptom pictures. In this case, it can be determined that the graph requirement corresponding to the question is a weak requirement. For another example, the user asks how to use a certain medicine. In this case, there may be no need for images to help understand, and a plain text answer can be output. In this case, it can be determined that there is no graph requirement.

[0040] Therefore, according to the analysis results of the search question, it is determined whether it is necessary to generate corresponding answer information in a graphic and textual manner.

[0041] Step 202: determine the answer type corresponding to the question to be retrieved based on the graph requirement information.

[0042] In this embodiment, the execution entity determines the answer type corresponding to the question to be retrieved based on the graph requirement information, that is, different graph requirement information corresponds to different answer types.

[0043] For example, picture-strong needs correspond to picture-type answers. Picture-type answers are mainly based on pictures, and can be accompanied by a small amount of picture description information, such as explaining what kind of picture it is, but picture-type answers are still mainly based on pictures.

[0044] For another example, picture-weak demand corresponds to picture-text type answers, which include pictures and large paragraphs of text content. The text here is not only a description of the picture, but also relevant descriptive information in other articles or journals.

[0045] For another example, picture-no requirement corresponds to a text-only answer, which means that this type of answer does not require pictures as supplementary information and can be clearly described using only text.

[0046] Step 203, searching in the data set corresponding to the answer type, and generating answer information of the question to be searched based on the search results.

[0047] In this embodiment, after determining the answer type corresponding to the question to be retrieved according to the graph requirement information, the above-mentioned execution entity will search in the data set corresponding to the answer type to obtain the corresponding search results, and generate the answer to the question to be retrieved based on the search results.

[0048] Here, the above-mentioned execution entity has pre-generated data sets corresponding to different answer types, that is, the picture type corresponds to the picture data set, the picture and text type corresponds to the picture and text data set, and the text type corresponds to the text data set, so as to search in the data set corresponding to the answer type to improve the pertinence of the search results and ensure the quality of the search results.

[0049] After obtaining the search results, the above-mentioned execution entity will also determine the matching degree between the search results and the question to be searched, thereby filtering the final answer from the search results based on the matching degree and displaying the answer to the user.

[0050] The information retrieval method provided by the disclosed embodiment first determines the image requirement information corresponding to the question to be retrieved, wherein the image requirement information is used to characterize the level of requirement for the image-type answer of the question to be retrieved; then determines the type of answer corresponding to the question to be retrieved based on the image requirement information; finally, searches in the data set corresponding to the answer type, and generates the answer letter of the question to be retrieved based on the search results. The information retrieval method in this embodiment determines the corresponding answer type according to the image requirement of the question to be retrieved, and retrieves the final answer in the data set of the answer type, thereby answering in a manner corresponding to the requirement of the question, making the answer content more targeted, improving the quality of the answer, and also improving the efficiency of information retrieval and the accuracy of the retrieval results, thereby improving the user experience.

[0051] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0052] Continue to refer Figure 3 , Figure 3 A process 300 of a second embodiment of an information retrieval method according to the present disclosure is shown. The information retrieval method comprises the following steps:

[0053] Step 301, using a pre-trained graph requirement recognition model to determine the graph requirement information corresponding to the question to be retrieved.

[0054] In this embodiment, the execution subject of the information retrieval method (for example Figure 1The server 105 shown in the figure will pre-train a graph requirement recognition model for identifying the level of requirement for image-type answers to the questions to be retrieved, and then use the graph requirement recognition model to determine the graph requirement level corresponding to the questions to be retrieved, where the graph requirement level includes strong requirement, weak requirement (or medium requirement) and no requirement.

[0055] Here, the executive body will first formulate the scope of graphic and text answers and construct user question samples. Then, it will use the ERNIE (Wenxin Big Model) teacher model to fine-tune and distill it to the TextCNN (convolutional neural network for text classification) student model to construct a lightweight graph demand recognition model, and then use the graph demand recognition model to classify user questions and identify questions with strong demand, weak demand, and no demand.

[0056] Therefore, the graph requirement recognition model can be used to accurately identify the requirement level of image answers for the questions to be retrieved.

[0057] Step 302, in response to determining that the image output requirement information is a strong requirement, determining that the answer type corresponding to the question to be retrieved is a picture type.

[0058] In this embodiment, if the image requirement information is determined to be a strong requirement, the execution subject can determine that the answer type corresponding to the question to be searched is a picture type. A strong requirement means that the question to be searched requires an answer in the form of a picture, and the answer of the picture type is mainly a picture, with only a small amount of picture description information, such as explaining what kind of picture it is, but the answer of the picture type is still mainly a picture.

[0059] Step 303, searching in the picture data set corresponding to the picture type to obtain the picture search result corresponding to the question to be searched.

[0060] In this embodiment, after determining that the answer type corresponding to the question to be searched is a picture type, the execution subject will search the picture data set corresponding to the picture type to obtain the corresponding picture search result. The question to be searched is searched in the picture data set to obtain the corresponding picture search result, which may include multiple picture type answers.

[0061] In some optional implementations of this embodiment, in response to not obtaining an image retrieval result in the image data set, a search is performed in a graphic data set corresponding to the graphic type and / or a text data set corresponding to the text type, and answer information is generated based on the graphic retrieval result and / or the text retrieval result.

[0062] In this implementation, since the answer type corresponding to the question to be retrieved is a picture type, the picture data set will be searched first. However, if the corresponding picture result is not retrieved in the picture data set, that is, the data is not recalled in the picture data set, then the execution subject will search in the picture and text data set or the text data set to obtain the corresponding picture and text search result or text search result, and generate the final answer based on the above search results. Therefore, when the most matching answer is not retrieved in the picture data set, the matching answer is retrieved from other data sets, avoiding the situation where the corresponding answer cannot be retrieved for the user, and improving the user experience.

[0063] In some optional implementations of this embodiment, a picture data set is generated through the following steps: obtaining a picture sample set; generating description information of picture samples in the picture sample set using a pre-trained picture description model; generating picture data based on picture titles and description information of the picture samples to obtain a picture data set.

[0064] In this implementation, the execution subject obtains a picture sample set including multiple picture samples. For each picture sample, the description information (caption) of the picture sample is generated using an existing picture description model, and each picture sample and its corresponding caption are taken as a picture data, thereby obtaining a picture data set including multiple picture data.

[0065] The above steps generate an image dataset, and a search is performed in the image dataset to obtain corresponding image results, thereby improving the accuracy of the image results.

[0066] Step 304, generating answer information corresponding to the question to be searched based on the preset template and the image search results.

[0067] In this embodiment, the execution subject generates answer information corresponding to the question to be searched based on the preset speech template and the image search results. Since the user inquiries of the strong demand type are more direct or obvious, a speech template can be preset, and the image in the image search results and the speech template can be combined to generate the final answer information.

[0068] For example, the script template is: The following pictures were found for you, xxx; the picture search results include: Picture 1, Picture 2 and Picture 3.

[0069] The answer generated based on the speech template and image retrieval results is:

[0070] We found the following pictures for you:

[0071] Image 1: X;

[0072] Image 2: X;

[0073] Image 3: X.

[0074] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiment, the information retrieval method in this embodiment, when the question to be retrieved is a picture-strong requirement, retrieves the final answer from the picture data set corresponding to the picture type, thereby displaying the answer information in the form of a picture, thereby improving the user experience.

[0075] Continue to refer Figure 4 , Figure 4 A process 400 of a third embodiment of an information retrieval method according to the present disclosure is shown. The information retrieval method comprises the following steps:

[0076] Step 401, using a pre-trained graph requirement recognition model to determine the graph requirement information corresponding to the question to be retrieved.

[0077] Step 401 is substantially the same as step 301 in the aforementioned embodiment, and reference may be made to the description of step 301 , which will not be repeated here.

[0078] Step 402, in response to determining that the image output requirement information is a weak requirement, determining that the answer type corresponding to the question to be retrieved is a graphic and text type.

[0079] In this embodiment, if the graph requirement is determined to be a weak requirement, the execution subject of the information retrieval method (e.g. Figure 1 The server 105 shown in the figure can determine that the answer type corresponding to the question to be retrieved is a picture-text type. The picture-text type answer includes pictures and large paragraphs of text content, where the text is not only a description of the picture, but also related description information in other articles or journals.

[0080] Step 403, searching in the image and text data set corresponding to the image and text type to obtain the image and text search result corresponding to the search question.

[0081] In this embodiment, after determining that the answer type corresponding to the question to be searched is a picture and text type, the execution subject will search the picture and text data set corresponding to the picture and text type to obtain the corresponding picture and text search result. That is, the question to be searched is searched in the picture and text data set to obtain the corresponding picture and text search result, which may include multiple pictures and text answers.

[0082] In some optional implementations of this embodiment, the image and text data set is generated through the following steps: obtaining an image and text sample set; extracting the image titles, article titles and article text contents of the image and text samples in the image and text sample set; generating image and text data based on the image titles, article titles and article text contents to obtain the image and text data set.

[0083] In this implementation, the execution subject obtains a picture and text sample set containing multiple picture and text samples. For each picture and text sample, the title of the picture, the title of the article, and the text content of the article in the picture and text sample are obtained, and each picture sample, picture title, article title, and article text content are taken as a picture and text data, thereby obtaining a picture and text data set containing multiple picture and text data.

[0084] Through the above steps, a picture and text data set is generated, and corresponding picture and text results are obtained by searching in the picture and text data set, thereby improving the accuracy of the picture and text results.

[0085] Step 404: determine the serial number information of the content referenced by the image and text search result.

[0086] In this embodiment, the execution subject determines the numbering information of the content referenced by the image and text search results. The document information and image information in the image and text search results need to be numbered so that they can be referenced and traced according to the specific serial numbers in the generated answers. Therefore, the execution subject adds numbers to the image information and document information in the image and text search results.

[0087] Step 405, generating answer information according to the image and text search results and number information.

[0088] In this embodiment, the execution entity combines the image information and document information in the image-text search result with the corresponding number information to generate a final answer.

[0089] from Figure 4 It can be seen that Figure 3 Compared with the corresponding embodiment, the information retrieval method in this embodiment, when the question to be retrieved is a graph-weak requirement, retrieves the final answer from the graph and text data set corresponding to the graph and text type, thereby displaying the answer information in the form of graph and text, improving the user experience.

[0090] Continue to refer Figure 5 , Figure 5 The flowchart 500 of the fourth embodiment of the information retrieval method according to the present disclosure is shown. The information retrieval method comprises the following steps:

[0091] Step 501, using a pre-trained graph requirement recognition model to determine the graph requirement information corresponding to the question to be retrieved.

[0092] Step 501 is substantially the same as step 301 in the aforementioned embodiment, and reference may be made to the description of step 301 , which will not be repeated here.

[0093] Step 502, in response to determining that the image output requirement information is a weak requirement, determining that the answer type corresponding to the question to be retrieved is a graphic and text type.

[0094] In this embodiment, if the graph requirement is determined to be a weak requirement, the execution subject of the information retrieval method (e.g. Figure 1 The server 105 shown in the figure can determine that the answer type corresponding to the question to be searched is a picture and text type. The picture and text type is a picture and text type + a text type. The text type answer is an auxiliary supplement of the picture and text type answer in terms of text knowledge, thereby ensuring the output of a picture and text answer while ensuring the quality of the text answer expression.

[0095] Step 503 , searching is performed simultaneously in the image-text data set corresponding to the image-text type and the text data set corresponding to the text type to obtain image-text search results and text search results corresponding to the search question.

[0096] In this embodiment, after determining that the answer type corresponding to the question to be searched is a picture and text type, the execution subject will search in the picture and text data set corresponding to the picture and text type and the text data set corresponding to the text type at the same time, thereby obtaining the corresponding picture and text search results and text search results. That is, the question to be searched is searched in the picture and text data set and the text data set at the same time, and the corresponding picture and text search results and text search results are obtained.

[0097] Step 504: determine the first numbering information of the content referenced by the image and text search results and the second numbering information of the content referenced by the text search results.

[0098] In this embodiment, the above-mentioned execution subject will determine the numbering information of the content referenced by the graphic and text search results, that is, the first numbering information. The numbering information of the content referenced by the text search results will also be determined, that is, the second numbering information. Whether it is the document information and image information in the graphic and text search results, or the document information in the text search results, it needs to be assigned a number so that it can be referenced and traced according to the specific serial number in the generated answer. Therefore, the above-mentioned execution subject will add numbers to the image information, document information in the graphic and text search results, and the text information in the text search results.

[0099] Step 505, generating answer information according to the image and text search results, the first number information and the text search results, the second number information.

[0100] In this embodiment, the execution entity combines the image information and document information in the image and text search results with the corresponding number information, and combines the text information in the text search results with the corresponding number, so as to generate a final answer.

[0101] from Figure 5 It can be seen that Figure 4Compared with the corresponding embodiment, the information retrieval method in this embodiment, when the question to be retrieved is a picture-weak requirement, searches for the final answer in the picture-text data set corresponding to the picture-text type and the text data set corresponding to the text type at the same time. The answer of the text type is an auxiliary supplement of the text knowledge of the picture-text type answer, thereby ensuring the output of answers with both pictures and texts while ensuring the quality of the expression of the text answer.

[0102] Continue to refer Figure 6 , Figure 6 The fifth embodiment of the information retrieval method according to the present disclosure is shown in process 600. The information retrieval method comprises the following steps:

[0103] Step 601, using a pre-trained graph requirement recognition model to determine the graph requirement information corresponding to the question to be retrieved.

[0104] Step 601 is substantially the same as step 301 in the aforementioned embodiment, and reference may be made to the description of step 301 , which will not be repeated here.

[0105] Step 602, in response to determining that the image output requirement information is not required, determining that the answer type corresponding to the question to be retrieved is a text type.

[0106] In this embodiment, if it is determined that the graph requirement is not required, the execution subject of the information retrieval method (for example Figure 1 The server 105 shown in the figure can determine that the answer type corresponding to the to-be-searched question is a text type. The text type answer only includes text content, and the text here can be related description information in other articles or journals.

[0107] Step 603, searching in the text data set corresponding to the text type to obtain a text search result corresponding to the question to be searched.

[0108] In this embodiment, after determining that the answer type corresponding to the question to be searched is a text type, the execution subject will search the text data set corresponding to the text type to obtain the corresponding text search result. That is, the question to be searched is searched in the text data set to obtain the corresponding text search result, and the text search result may include multiple text answers.

[0109] In some optional implementations of the present embodiment, a text sample set is obtained; text titles and text body contents of text samples in the text sample set are extracted; the text titles and text body contents are segmented, and text data is generated based on the segmented text to obtain text data.

[0110] In this implementation, the execution subject obtains a text sample set containing multiple text samples. For each text sample, the text title and text body content are extracted, and the text title and text body content are processed to obtain text information of appropriate length, and the segmented text information is used as a text data, thereby obtaining a text data set containing multiple text data.

[0111] A text dataset is generated through the above steps, and corresponding text results are obtained by searching in the text dataset, thereby improving the accuracy of the text results.

[0112] Step 604: determine the serial number information of the content referenced by the text search result.

[0113] In this embodiment, the execution subject determines the numbering information of the content cited by the text search result. The text information in the text search result needs to be numbered so that it can be referenced and traced according to the specific serial number in the generated answer. Therefore, the execution subject adds numbers to the text information in the text search result.

[0114] Step 605, generating answer information according to the text search result and the number information.

[0115] In this embodiment, the execution entity combines the text information in the text search result with the corresponding number information to generate a final answer.

[0116] from Figure 6 It can be seen that Figure 5 Compared with the corresponding embodiment, the information retrieval method in this embodiment, when the question to be retrieved is a picture-no requirement, retrieves the final answer from the text data set corresponding to the text type, thereby displaying the answer information in the form of pure text, thereby improving the user experience.

[0117] Continue to refer Figure 7 , Figure 7 The sixth embodiment of the information retrieval method according to the present disclosure is shown in process 700. The information retrieval method comprises the following steps:

[0118] Step 701, using a pre-trained graph requirement recognition model to determine graph requirement information corresponding to the medical question to be retrieved.

[0119] In this embodiment, the problem to be searched is a medical problem, and the execution subject of the information search method (for example Figure 1 The server 105 shown in the figure will use the pre-trained graph requirement recognition model to determine the graph requirement information corresponding to the medical question to be retrieved.

[0120] With the booming development of the mobile Internet era, more and more users will first choose to obtain medical and health knowledge online and seek solutions when they encounter medical and health problems. However, there are some problems with the current medical knowledge online. On the one hand, the quality and comprehensiveness of online medical content information are often not guaranteed, and there are often unclear and indirect expressions, which makes it difficult for users to understand; on the other hand, if some medical questions are only answered with pure text answers, it is difficult to meet user needs, resulting in a poor user experience, such as those related to skin disease symptoms, acupuncture point locations, and the distinction between symptoms of different diseases.

[0121] Based on this, this embodiment will first train a graph requirement recognition model for identifying the level of requirement for image-based answers to the medical questions to be retrieved, and then use the graph requirement recognition model to determine the graph requirement level corresponding to the medical questions to be retrieved, where the graph requirement level includes strong requirement, weak requirement (or medium requirement) and no requirement.

[0122] For example, if the user's question is "What does a boil look like?", this situation may require a picture description, which is more intuitive and difficult to describe with words. In this case, it can be determined that the picture requirement corresponding to the question is a strong requirement.

[0123] For another example, the user's question is "Symptoms of amoxicillin allergy". This situation can be described in words, such as a rash on the skin, or accompanied by relevant symptom pictures. In this case, it can be determined that the picture requirement corresponding to the question is a weak requirement.

[0124] For example, if a user asks about the usage of a certain medicine, this situation may not require images to help understand, and outputting a plain text answer is sufficient. In this case, it can be determined that there is no demand for images.

[0125] Therefore, according to the analysis results of the search question, it is determined whether it is necessary to generate corresponding answer information in a graphic and textual manner.

[0126] Step 702: Determine the answer type corresponding to the medical question to be retrieved based on the graph requirement information.

[0127] In this embodiment, the execution entity will determine the answer type corresponding to the medical question to be retrieved based on the graph requirement information, that is, different graph requirement information corresponds to different answer types.

[0128] For example, picture-strong needs correspond to picture-type answers. Picture-type answers are mainly based on pictures, and can be accompanied by a small amount of picture description information, such as explaining what kind of picture it is, but picture-type answers are still mainly based on pictures.

[0129] For another example, picture-weak demand corresponds to picture-text type answers, which include pictures and large paragraphs of text content. The text here is not only a description of the picture, but also relevant descriptive information in other articles or journals.

[0130] For another example, picture-no requirement corresponds to a text-only answer, which means that this type of answer does not require pictures as supplementary information and can be clearly described using only text.

[0131] Step 703: input the medical question to be retrieved into the large language model, and use the large language model to search in the data set corresponding to the answer type, and output the answer information corresponding to the medical question to be retrieved.

[0132] In this embodiment, after determining the answer type corresponding to the medical question to be retrieved according to the graph requirement information, the execution subject will input the medical question to be retrieved into the large language model, and search the data set corresponding to the answer type through the large language model to retrieve the corresponding search results, and generate the answer to the medical question to be retrieved according to the search results. Thus, the answer to the question to be retrieved is generated through the large model, which improves the accuracy of the answer.

[0133] It should be noted that a big model is an artificial intelligence model designed to understand and generate human language. By training on large amounts of text data, it can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. A big model refers to a deep neural network with more than 1 billion parameters that can process massive amounts of data and complete various complex tasks. With the continuous improvement of computer hardware performance and the continuous optimization of deep learning algorithms, the development of big models is also getting faster and faster. The parameter scale of big models continues to expand, and the training time is getting longer and longer, but the performance is also improving. Big models are usually based on deep learning architectures, such as transformers, which enable impressive capabilities in various natural language processing tasks. Common big models can include but are not limited to ChatGPT, GPT-4, ERNIE, Lingyi Bot, and so on.

[0134] In some optional implementations of this embodiment, the data sets corresponding to the answer types include: a picture medical data set corresponding to the picture type, a picture-text medical data set corresponding to the picture-text type, and a text medical data set corresponding to the text type.

[0135] In this implementation, the image medical dataset can be derived from a medical article, which includes a title and a text. The text includes one or more images, and each image has a caption. Then, the existing image description model is used to generate caption descriptions for all images. The image medical dataset is constructed based on the images and their description information.

[0136] In addition, you can also use picture titles, article titles, and text content as graphic medical data to construct a graphic medical dataset.

[0137] You can also obtain text content through self-built medical articles and medical classics, authorized book guides, etc., and perform appropriate segmentation on longer articles to obtain better retrieval results. Then use the segmented text as text medical data to construct a text medical data set.

[0138] from Figure 7 It can be seen that Figure 6 Compared with the corresponding embodiment, the information retrieval method in this embodiment identifies the graphic requirements of the user's medical problems and determines whether it is suitable to answer them in a graphic and textual manner, that is, to answer with pure pictures, pure text or a combination of pictures and texts, and retrieves relevant picture data and text data in real time, thereby improving the retrieval efficiency of medical problems and the accuracy of retrieval results.

[0139] Continue to refer Figure 8 , which shows a schematic diagram of an application scenario of the information retrieval method according to the present disclosure. In this application scenario, it includes an offline part and an online part, as follows:

[0140] 1. Offline part:

[0141] First, obtain the image and text data: The source of the image and text data can be a medical article, which includes a title and a text. The text contains one or more pictures, and each picture has a picture title. Plain text data mainly comes from self-built medical articles and medical classics, authorized book guides, etc.

[0142] Then, data processing is performed: the existing LLM (picture description model) is used to generate the caption description of the picture. For plain text data, the longer content of the article is appropriately divided into chunks for better retrieval results.

[0143] There are two ways to build a database: the first is to use the image title and caption as searchable fields to produce image-caption evidence; the second is to use the image title, the original title of the atlas, and the original content of the atlas as searchable fields to produce image-text evidence.

[0144] Thus, we obtain graph granularity-ANN (Artificial Neural Network) and content / chunk granularity-ANN. Graph granularity-ANN includes all graph data, and content / chunk granularity-ANN includes text data and plain text data in graph and text data.

[0145] 2. Online part:

[0146] First, obtain the user's medical problem and perform intent recognition on the user's problem to determine whether the problem belongs to graph-strong demand, graph-medium weak demand, or graph-no demand. Here, a pre-trained user intent recognition model can be used to identify the user's graph demand.

[0147] Then the search is sorted according to the graph requirement level, specifically:

[0148] During retrieval: For image-strong requirements: prioritize the retrieval of offline generated image caption evidence. If there is recalled data, only use image caption evidence. If there is no recalled data, then retrieve image and text evidence or plain text evidence.

[0149] For the weak demand of "picture-text": retrieve both picture and text evidence and plain text evidence at the same time, and obtain only picture and text evidence or picture and text evidence + plain text evidence. The purpose of retrieving plain text evidence here is to supplement the picture and text evidence with text knowledge, so as to ensure the output of both picture and text answers while ensuring the quality of text answer expression.

[0150] For graphs - no requirement: only plain text evidence is retrieved.

[0151] When sorting, according to the image-strong requirement: if there is recalled image caption data, only image caption evidence is used; if there is no recalled image caption data, image and text evidence or image and text evidence + plain text evidence is used; otherwise, text evidence is used.

[0152] For weak requirements of picture-text evidence: give priority to using only picture-text evidence. If there is none, use picture-text evidence + plain text evidence.

[0153] For graphs - no requirement: only plain text evidence is used.

[0154] Finally, the answers are organized according to the search ranking results. For the RAG (Retrieval-Augmented Generation) model training samples, the prompt instructions of the large model must be tuned first, and auxiliary samples are produced based on the powerful large model, and then medical professionals are introduced to refine the annotations. The online refined annotation instruction set samples are used to train the ERNIE Speed ​​model through SFT (Supervised Fine-tuning), which has a faster response speed.

[0155] When constructing instructions, in addition to the already tuned template part, variable information that needs to be introduced includes: user questions, user conversation history, and retrieval evidence information (graphic evidence includes picture title, picture caption description, and the title and content of the text in which it is located; plain text evidence includes the title and content of the text). The document information and multiple picture information of the retrieval evidence need to be numbered so that they can be referenced and traced according to the specific serial number in the generated answer. The instruction set sample is constructed differently according to different user needs:

[0156] (1) For image-intensive needs: For this type of needs, users’ inquiries are more direct or obvious. Instruction set samples are constructed and trained using speech templates, such as “We found the following image for you, xxx.”

[0157] (2) For weak requirements in pictures: This type of requirement is more suitable for answering with a text answer and a picture caption. The text answer and the picture can be guaranteed to come from the same evidence or different evidence. When the answer is generated, the referenced document number and picture number will be given to facilitate locating the corresponding document reference source and picture for display.

[0158] (3) For figure-free requirements: For this type of requirements, the referenced document number will be given when the answer is generated.

[0159] The above information retrieval method is based on the RAG technical solution. The offline part introduces the construction of atlas evidence, the online search sorting priority strategy formulation, and the LLM-based answer organization generation, so as to generate answer information in a graphic and textual way.

[0160] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an information retrieval device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0161] like Fig. 9 As shown, the information retrieval device 900 of this embodiment includes: a demand determination module 901, a type determination module 902 and an answer generation module 903. The demand determination module 901 is configured to determine the image demand information corresponding to the question to be retrieved, wherein the image demand information is used to characterize the demand level of the question to be retrieved for the image type answer; the type determination module 902 is configured to determine the answer type corresponding to the question to be retrieved based on the image demand information; the answer generation module 903 is configured to search in the data set corresponding to the answer type, and generate the answer information of the question to be retrieved based on the search result.

[0162] In this embodiment, in the information retrieval device 900, the specific processing of the demand determination module 901, the type determination module 902 and the answer generation module 903 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0163] In some optional implementations of this embodiment, the graphic requirement information includes: strong requirement, weak requirement or no requirement; and the type determination module is further configured to: in response to determining that the graphic requirement information is a strong requirement, determine that the answer type corresponding to the question to be retrieved is a picture type; in response to determining that the graphic requirement information is a weak requirement, determine that the answer type corresponding to the question to be retrieved is a graphic and text type or a graphic and text type; in response to determining that the graphic requirement information is no requirement, determine that the answer type corresponding to the question to be retrieved is a text type.

[0164] In some optional implementations of this embodiment, the image requirement information is a strong requirement, and the answer generation module includes: an image retrieval submodule, configured to search in an image data set corresponding to the image type to obtain an image retrieval result corresponding to the question to be retrieved; a first answer generation submodule, configured to generate answer information based on a preset template and the image retrieval result.

[0165] In some optional implementations of this embodiment, the answer generation module also includes: a second answer generation sub-module, which is configured to search in a picture and text data set corresponding to the picture and text type and / or a text data set corresponding to the text type in response to not obtaining an image retrieval result in the picture data set, and generate answer information based on the picture and text retrieval result and / or the text retrieval result.

[0166] In some optional implementations of the present embodiment, the above-mentioned information retrieval device also includes a first generation module for generating a picture data set, and the first generation module is configured to: obtain a picture sample set; generate description information of picture samples in the picture sample set using a pre-trained picture description model; generate picture data based on the picture titles and description information of the picture samples to obtain a picture data set.

[0167] In some optional implementations of this embodiment, the graph requirement information is a weak requirement, and the answer type is a graph and text type; the answer generation module includes: a graph and text retrieval submodule, configured to search in a graph and text data set corresponding to the graph and text type, and obtain a graph and text retrieval result corresponding to the question to be retrieved; a first number determination submodule, configured to determine the number information of the content referenced by the graph and text retrieval result; and a third answer generation submodule, configured to generate answer information based on the graph and text retrieval result and the number information.

[0168] In some optional implementations of this embodiment, the graphic requirement information is a weak requirement, and the answer type is the graphic and text type; the answer generation module includes: a graphic and text retrieval submodule, which is configured to simultaneously search in the graphic data set corresponding to the graphic type and the text data set corresponding to the text type to obtain the graphic retrieval results and text retrieval results corresponding to the question to be searched; a second number determination submodule, which is configured to determine the first number information of the content referenced by the graphic and text retrieval results and the second number information of the content referenced by the text retrieval results; a fourth answer generation submodule, which is configured to generate answer information based on the graphic and text retrieval results, the first number information, the text retrieval results, and the second number information.

[0169] In some optional implementations of the present embodiment, the above-mentioned information retrieval device also includes a second generation module for generating a graphic and text data set, and the second generation module is configured to: obtain a graphic and text sample set; extract the picture title, article title and article text content of the graphic and text samples in the graphic and text sample set; generate graphic and text data based on the picture title, article title and article text content to obtain a graphic and text data set.

[0170] In some optional implementations of this embodiment, the graph requirement information is no requirement, and the answer generation module includes: a text retrieval submodule, configured to search in a text data set corresponding to the text type to obtain a text retrieval result corresponding to the question to be searched; a third number determination submodule, configured to determine the number information of the content referenced by the text retrieval result; and a fifth answer generation submodule, configured to generate answer information based on the text retrieval result and the number information.

[0171] In some optional implementations of the present embodiment, the above-mentioned information retrieval device also includes a third generation module for generating a text data set, and the third generation module is configured to: obtain a text sample set; extract text titles and text body contents of text samples in the text sample set; segment the text titles and text body contents, generate text data based on the segmented text, and obtain a text data set.

[0172] In some optional implementations of this embodiment, the requirement determination module is further configured to: use a pre-trained graph requirement recognition model to determine the graph requirement information corresponding to the question to be retrieved.

[0173] In some optional implementations of this embodiment, the answer generation module is further configured to: input the question to be retrieved into the large language model, and search in the data set corresponding to the answer type through the large language model, and output the answer information.

[0174] In some optional implementations of this embodiment, the question to be retrieved is a medical question to be retrieved, and the data sets corresponding to the answer type include: a picture medical data set corresponding to the picture type, a picture-text medical data set corresponding to the picture-text type, and a text medical data set corresponding to the text type.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0176] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0178] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as information retrieval methods. For example, in some embodiments, the information retrieval method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the information retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the information retrieval method in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] Cloud computing refers to a technology system that uses the network to access a shared pool of physical or virtual resources that is elastic and scalable. Resources may include servers, operating systems, networks, software, applications, or storage devices, and can be deployed and managed in an on-demand, self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as model training.

[0186] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0187] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0188] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An information retrieval method, comprising: Determine the image requirement information corresponding to the question to be searched, wherein the image requirement information is used to represent the level of requirement for the image answer of the question to be searched; Determine the answer type corresponding to the question to be retrieved based on the graph requirement information; A search is performed in the data set corresponding to the answer type, and answer information of the question to be searched is generated based on the search result.

2. According to the method of claim 1, the graph requirement information comprises: strong demand, weak demand, or no demand; as well as The determining the answer type corresponding to the to-be-searched question based on the graph requirement information includes: In response to determining that the image requirement information is the strong requirement, determining that the answer type corresponding to the question to be retrieved is a picture type; In response to determining that the image requirement information is the weak requirement, determining that the answer type corresponding to the question to be searched is an image-text type or an image-text and text type; In response to determining that the graph requirement information is no requirement, determining that the answer type corresponding to the question to be retrieved is a text type.

3. The method according to claim 2, wherein: The graph requirement information is the strong requirement, searching in the data set corresponding to the answer type, and generating the answer information of the question to be searched based on the search result, including: Searching in the image data set corresponding to the image type to obtain the image search result corresponding to the question to be searched; The answer information is generated according to a preset template and the image retrieval result.

4. The method according to claim 3, wherein: The step of searching in the data set corresponding to the answer type and generating the answer information of the question to be searched based on the search result also includes: In response to not obtaining the image retrieval result in the image dataset, a search is performed in the image-text dataset corresponding to the image-text type and / or the text dataset corresponding to the text type, and the answer information is generated based on the image-text retrieval result and / or the text retrieval result.

5. The method according to claim 3, wherein: The image dataset is generated by the following steps: Get a sample set of images; Generate description information of the image samples in the image sample set using a pre-trained image description model; The picture data is generated based on the picture title of the picture sample and the description information to obtain the picture data set.

6. The method according to claim 2, wherein: The graphic requirement information is the weak requirement, and the answer type is the graphic type; The step of searching in the data set corresponding to the answer type and generating the answer information of the question to be searched based on the search result includes: Searching in the image and text data set corresponding to the image and text type to obtain the image and text search result corresponding to the question to be searched; Determine the serial number information of the content cited by the image and text search result; The answer information is generated according to the graphic and text search results and the number information.

7. The method according to claim 2, wherein: The image requirement information is the weak requirement, and the answer type is the image and text type; The step of searching in the data set corresponding to the answer type and generating the answer information of the question to be searched based on the search result includes: At the same time, searching the image-text data set corresponding to the image-text type and the text data set corresponding to the text type to obtain the image-text search result and the text search result corresponding to the question to be searched; Determine the first number information of the content cited by the image and text search result and the second number information of the content cited by the text search result; The answer information is generated according to the image-text search result and the first number information and the text search result and the second number information.

8. The method according to claim 6 or 7, wherein: The image and text dataset is generated by the following steps: Get a sample set of images and texts; Extracting picture titles, article titles and article text contents of the picture and text samples in the picture and text sample set; The graphic and text data is generated based on the picture title, the article title and the article body content to obtain the graphic and text data set.

9. The method according to claim 2, wherein: The graph requirement information is that there is no requirement, searching in the data set corresponding to the answer type, and generating the answer information of the question to be searched based on the search result, including: Searching in the text data set corresponding to the text type to obtain a text search result corresponding to the question to be searched; Determining the serial number information of the content cited by the text search result; The answer information is generated based on the text search result and the number information.

10. The method according to claim 9, wherein: The text dataset is generated by the following steps: Get a text sample set; Extracting text titles and text body contents of text samples in the text sample set; The text title and the text body content are segmented, and text data is generated according to the segmented text to obtain the text data set.

11. The method according to claim 1, wherein: The step of determining the graph requirement information corresponding to the question to be searched includes: The graph requirement information corresponding to the question to be retrieved is determined using a pre-trained graph requirement recognition model.

12. The method according to claim 1, wherein: The step of searching in the data set corresponding to the answer type and generating the answer information of the question to be searched based on the search result includes: The question to be searched is input into a large language model, and the large language model is used to search in a data set corresponding to the answer type, and the answer information is output.

13. The method according to claim 2, wherein: The question to be retrieved is a medical question to be retrieved, and the data sets corresponding to the answer type include: a picture medical data set corresponding to the picture type, a picture and text medical data set corresponding to the picture and text type, and a text medical data set corresponding to the text type.

14. An information retrieval device, comprising: A requirement determination module is configured to determine the image requirement information corresponding to the to-be-searched question, wherein the image requirement information is used to represent the level of requirement of the to-be-searched question for an image-based answer; A type determination module, configured to determine the type of answer corresponding to the to-be-retrieved question based on the graph requirement information; The answer generation module is configured to search in the data set corresponding to the answer type and generate answer information of the question to be searched based on the search result.

15. The apparatus according to claim 14, wherein the graph requirement information comprises: strong, weak or no demand; and The type determination module is further configured to: In response to determining that the image requirement information is the strong requirement, determining that the answer type corresponding to the question to be retrieved is a picture type; In response to determining that the image requirement information is the weak requirement, determining that the answer type corresponding to the question to be searched is an image-text type or an image-text and text type; In response to determining that the graph requirement information is no requirement, determining that the answer type corresponding to the question to be retrieved is a text type.

16. The device according to claim 15, wherein: The graph requirement information is the strong requirement, and the answer generation module includes: The image retrieval submodule is configured to search in the image data set corresponding to the image type to obtain the image retrieval result corresponding to the question to be searched; The first answer generation submodule is configured to generate the answer information according to a preset template and the image retrieval result.

17. The device according to claim 16, wherein: The answer generation module also includes: The second answer generation submodule is configured to, in response to not finding the image retrieval result in the image dataset, search in the image-text dataset corresponding to the image-text type and / or the text dataset corresponding to the text type, and generate the answer information based on the image-text retrieval result and / or the text retrieval result.

18. The device according to claim 16, wherein: The apparatus further comprises a first generating module for generating the picture data set, wherein the first generating module is configured to: Get a sample set of images; Generate description information of the image samples in the image sample set using a pre-trained image description model; The picture data is generated based on the picture title of the picture sample and the description information to obtain the picture data set.

19. The device according to claim 15, wherein: The graphic requirement information is the weak requirement, and the answer type is the graphic type; The answer generation module comprises: The image and text retrieval submodule is configured to search in the image and text data set corresponding to the image and text type to obtain the image and text retrieval result corresponding to the question to be searched; A first serial number determination submodule is configured to determine serial number information of the content referenced by the graphic and text search result; The third answer generation submodule is configured to generate the answer information according to the image and text retrieval result and the numbering information.

20. The device according to claim 15, wherein: The image requirement information is the weak requirement, and the answer type is the image and text type; The answer generation module comprises: The image and text search submodule is configured to search the image and text data set corresponding to the image and text type and the text data set corresponding to the text type at the same time to obtain the image and text search results and text search results corresponding to the question to be searched; A second number determination submodule is configured to determine the first number information of the content referenced by the image-text search result and the second number information of the content referenced by the text search result; The fourth answer generation submodule is configured to generate the answer information according to the image-text search result and the first numbering information and the text search result and the second numbering information.

21. The device according to claim 19 or 20, wherein: The apparatus further comprises a second generating module for generating the graphic and text data set, wherein the second generating module is configured to: Get a sample set of images and texts; Extracting picture titles, article titles and article text contents of the picture and text samples in the picture and text sample set; The graphic and text data is generated based on the picture title, the article title and the article body content to obtain the graphic and text data set.

22. The device according to claim 15, wherein: The graph requirement information is no requirement, and the answer generation module includes: A text retrieval submodule is configured to search in the text data set corresponding to the text type to obtain a text retrieval result corresponding to the question to be searched; A third number determination submodule is configured to determine the number information of the content referenced by the text search result; The fifth answer generation submodule is configured to generate the answer information according to the text retrieval result and the number information.

23. The device according to claim 22, wherein: The apparatus further comprises a third generating module for generating the text data set, wherein the third generating module is configured to: Get a text sample set; Extracting text titles and text body contents of text samples in the text sample set; The text title and the text body content are segmented, and text data is generated according to the segmented text to obtain the text data set.

24. The device according to claim 14, wherein: The demand determination module is further configured to: The graph requirement information corresponding to the question to be retrieved is determined using a pre-trained graph requirement recognition model.

25. The device according to claim 14, wherein: The answer generation module is further configured to: The question to be searched is input into a large language model, and the large language model is used to search in a data set corresponding to the answer type, and the answer information is output.

26. The device according to claim 15, wherein: The question to be retrieved is a medical question to be retrieved, and the data sets corresponding to the answer type include: a picture medical data set corresponding to the picture type, a picture and text medical data set corresponding to the picture and text type, and a text medical data set corresponding to the text type.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method of any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images