Large model RAG recall strategy intelligent planning method and device, medium and equipment

Abstract

The invention relates to the technical field of intelligent question answering, in particular to a large model RAG recall strategy intelligent planning method and device, a medium and equipment, initial questions are converted into target questions by introducing cue words, and multi-dimensional processing of key information and vectors is performed on the target questions to obtain reference recall data based on a preset large model, so that the target questions can be called according to the reference recall data. And inputting the initial scene category, the preset recall path, the target problem, the reference recall data and the preset cue word into a preset large model, obtaining a target recall path, sequentially executing all data recall links in the target recall path according to a sequence, obtaining target recall data, and providing multi-dimensional reference information for the large model. According to the method, the large model is assisted to obtain the target recall path formed by the multiple data recall links arranged in sequence, so that the data recall process is more efficient, the method can adapt to various complex and changeable application scenes and problem types, and the accuracy and efficiency of the recall method are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent question answering technology, and in particular to a large-model RAG recall strategy intelligent planning method, device, medium and equipment. Background Art

[0002] With the rapid development of information technology, users' demand for information retrieval and question-answering systems is growing. Traditional information retrieval methods are often based on simple keyword matching. For example, the retrieval method based on inverted index can only mechanically match keywords and it is difficult to deeply understand the semantics and context of the question. When faced with complex natural language question-answering tasks, it is difficult to accurately understand user intentions and provide comprehensive and accurate answers, resulting in a significant reduction in the accuracy and relevance of the retrieval results.

[0003] With the development of artificial intelligence technology, large models can understand and generate natural language to a certain extent after pre-training on large amounts of text data, and have gradually become the focus of the field of natural language processing. However, the knowledge reserves of large models themselves have limitations. When faced with some problems that require specific domain knowledge or the latest information, it is difficult to provide comprehensive and accurate answers relying solely on the parameterized knowledge within the model.

[0004] In order to solve the above problems, the existing recall task based on Retrieval-Augmented Generation (RAG) attempts to combine large models with external knowledge sources, but there are still many problems in practical applications. For example, in the problem processing link, there is a lack of effective analysis and transformation path for the initial question input by the user, and the various potential expressions and related sub-questions of the question are not fully explored, resulting in the subsequent retrieval may miss important information; in terms of the recall path, it fails to formulate refined and personalized recall plans for different types of questions, making the recalled document data not very targeted; the processing of the recall results is also relatively rough, lacking an effective screening, fusion and rearrangement mechanism, resulting in the answers finally presented to the user may have information redundancy, logical confusion and other problems, which cannot meet the user's needs for high-quality and efficient information acquisition.

[0005] Therefore, how to intelligently plan the RAG recall strategy to improve the accuracy and efficiency of the recall results in various scenarios has become an urgent problem to be solved. Summary of the invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is a large-model RAG recall strategy intelligent planning method, which includes the following steps:

[0007] S1, obtain the initial question input by the user, the preset large model, the preset database, the initial scene category and the preset recall path corresponding to each initial scene category, wherein the preset database stores the initial document data, and each preset recall path includes several sequential data recall links.

[0008] S2, inputting the initial question and the first preset prompt word into the preset large model, obtaining a number of target question data, wherein the first preset prompt word includes a question decomposition prompt word and a question expansion prompt word, and the target question data includes a number of target questions and a question identifier corresponding to each target question.

[0009] S3, extract key information and convert vectors for each target question to obtain a target key information list and target question vector corresponding to each target question.

[0010] S4, input the target key information list, target question vector, initial document data and second preset prompt word corresponding to each target question into the preset large model respectively, and obtain the reference recall data corresponding to each target question, wherein the reference recall data includes several documents and the recall priority corresponding to each document.

[0011] S5, input each initial scene category, the preset recall path corresponding to each initial scene category, each target question, the reference recall data corresponding to each target question and the third preset prompt word into the preset large model, and obtain the target recall path corresponding to the initial question, wherein the target recall path includes several sequential data recall links.

[0012] S6, executing all data recall links in the target recall path in sequence to obtain the target recall data corresponding to the initial question, wherein the target recall data includes several document fragments.

[0013] The present invention also provides a large-model RAG recall strategy intelligent planning device, the large-model RAG recall strategy intelligent planning device comprising:

[0014] The data acquisition module is used to obtain the initial question input by the user, the preset large model, the preset database, the initial scene category and the preset recall path corresponding to each initial scene category, wherein the preset database stores the initial document data, and each preset recall path includes several sequential data recall links.

[0015] The target question acquisition module is used to input the initial question and the first preset prompt word into the preset large model to obtain a number of target question data, wherein the first preset prompt word includes a question decomposition prompt word and a question expansion prompt word, and the target question data includes a number of target questions and a question identifier corresponding to each target question.

[0016] The target problem processing module is used to extract key information and convert vectors for each target problem, and obtain the target key information list and target problem vector corresponding to each target problem.

[0017] The first data recall module is used to input the target key information list, target question vector, initial document data and second preset prompt word corresponding to each target question into the preset large model respectively, and obtain the reference recall data corresponding to each target question, wherein the reference recall data includes a number of documents and the recall priority corresponding to each document.

[0018] The target recall path acquisition module is used to input each initial scene category, the preset recall path corresponding to each initial scene category, each target question, the reference recall data corresponding to each target question and the third preset prompt word into the preset large model, and obtain the target recall path corresponding to the initial question, wherein the target recall path includes several sequential data recall links.

[0019] The second data recall module is used to execute all data recall links in the target recall path in sequence to obtain the target recall data corresponding to the initial question, wherein the target recall data includes several document fragments.

[0020] The present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned large model RAG recall strategy intelligent planning method.

[0021] The present invention also provides an electronic device, comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0022] The present invention has at least the following beneficial effects: by introducing problem decomposition prompt words and problem expansion prompt words, the original initial problem is converted into a number of more targeted and detailed target problems, the comprehensiveness of the analysis of the initial problem by the large model is improved, the key information and vector of each target problem are multi-dimensionally processed, the target key information list, the target problem vector and the second preset prompt word corresponding to each target problem are respectively input into the preset large model, and the reference recall data corresponding to each target problem is obtained, each initial scene category, the preset recall path corresponding to each initial scene category, each target problem, the reference recall data corresponding to each target problem and the third preset prompt word are input into the preset large model, and the initial problem is obtained. The target recall path corresponding to the question is executed in sequence, all data recall links in the target recall path are executed in sequence, the target recall data corresponding to the initial question is obtained, and multi-dimensional reference information is provided to the large model based on the multi-dimensional input information, which gives full play to the natural language processing ability of the large model and the guiding role of different prompt words, thereby assisting the large model to obtain the target recall path composed of multiple data recall links arranged in sequence, making the data recall process more efficient, and the target recall path can be adjusted according to the specific target problem, so that the large model RAG recall strategy intelligent planning method can adapt to various complex and changeable application scenarios and problem types, thereby improving the accuracy and efficiency of the large model RAG recall strategy intelligent planning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of a large model RAG recall strategy intelligent planning method provided in the first embodiment of the present invention;

[0025] Figure 2 A schematic diagram of the structure of a large-model RAG recall strategy intelligent planning device provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the above-mentioned terms used to distinguish similar objects can be interchanged so that the present invention can also implement other embodiments other than the above-mentioned illustrated embodiments or described embodiments. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] This embodiment 1 provides a large model RAG recall strategy intelligent planning method, the large model RAG recall strategy intelligent planning method includes the following steps: Figure 1 As shown:

[0030] S1, obtain the initial question input by the user, the preset large model, the preset database, the initial scene category and the preset recall path corresponding to each initial scene category, wherein the preset database stores the initial document data, and each preset recall path includes several sequential data recall links.

[0031] Among them, the preset large model has powerful natural language processing capabilities and can understand the meaning of text, perform semantic analysis, text generation, etc. For example, the preset large model can be a specific large language model trained based on the Transformer architecture.

[0032] The initial document data stored in the preset database provides a source of knowledge for the entire recall process, which can cover various fields and types, such as academic papers, news reports, product manuals, encyclopedias, etc., including document fragments, basic document information, document paragraph information, document table information, document image information, document link information, document recommendation question and answer information and other types of document-related data as well as initial key information data and initial vector data, so as to increase the comprehensiveness and accuracy of answering questions through rich document data.

[0033] Among them, the document fragments can be document fragments obtained by segmenting the entire document according to the segmentation methods such as semantics, word count or paragraph. Basic document information can include information such as the word count, chapter count, summary, key information, vectorization method, entities contained in the title, etc. corresponding to each document. Document paragraph information can include information such as the word count, summary, summary and key information corresponding to each document paragraph. Document table information can include information such as the tables contained in each document and the page number of each table, the context corresponding to the table and the table description. Document image information can include information such as the pictures contained in each document and the page number of each picture, the context corresponding to the picture, the picture description and the picture source. Document link information can include information such as the links contained in each document and the URL corresponding to each link, title content, text content, publication date, publishing organization and publishing site. Document recommendation question and answer information can include recommended questions and recommended answers corresponding to each document.

[0034] This embodiment is applicable to problem analysis and document recall under various scenario categories. The initial scenario category can be set by the implementer according to the actual situation. Problems under different scenario categories differ in processing methods and expected recall results. Therefore, in order to improve the efficiency and accuracy of the recall, different recall paths are formulated for different initial scenario categories.

[0035] In a specific implementation, the initial scenario categories may include summary scenarios, recommended question and answer scenarios, original text recall scenarios, and summary recall scenarios; the preset data recall links include data recall methods and data processing methods; the data recall methods include at least one of text sorting recall, vector recall, summary recall, question and answer recall, chart recall, full text recall, summary recall, and external data recall; the data processing methods include at least one of no processing, data rearrangement, data compression, and data fusion.

[0036] Among them, the preset recall path corresponding to each initial scenario category can be obtained by testing and analyzing the recall path and recall results of a large number of questions under each initial scenario category. It can assist the large model to efficiently and accurately perform recall operations according to the established path, providing users with high-quality recall services and accurate recall results.

[0037] Among them, the implementation method of text sorting recall can be to sort the text data corresponding to the document based on a certain text relevance evaluation standard, and put the document fragments with high relevance to the query demand in the front for recall. The implementation method of vector recall can be to first convert text and other data into vector representation, and then measure the degree of correlation between the data by calculating the similarity between the question vector and the text vector, and recall the document fragments with high correlation. The implementation method of summary recall can be to select document fragments whose summary part matches the query content from many documents for recall. The implementation method of question and answer recall can be to recommend question and answer information for documents stored in the form of questions and answers, match and search in the existing document recommendation question and answer information according to the questions raised by the user, find the question and answer that is most similar to the user's question and can best answer the question, and recall it. The implementation method of chart recall can be to classify and annotate the pictures and tables in the document table information and document picture information according to the relevant needs of the user (such as finding pictures and tables related to a certain topic, pictures and tables of a specific type, etc.), and retrieve and match based on the text descriptions related to the pictures and tables, and recall the document fragments where the pictures and tables that meet the requirements are located. The implementation method of full-text recall can be a relatively broad search method such as simple keyword matching. As long as the document fragment contains content related to the query requirements, the full text of the document fragment will be recalled regardless of the relevance and the position of the document. The implementation method of summary recall can be to extract key information from a large number of document data, summarize and summarize them, form a relatively complete and concise summary answer, and recall document fragments related to the summary answer. The implementation method of external data recall can be to establish a connection with an external data source when the data in the database cannot meet the needs, and obtain relevant data from external databases, network resources, etc. according to certain interface protocols and retrieval rules for recall.

[0038] The implementation method of not processing can be to directly carry out the subsequent process or present the recalled document fragments intact without any additional operation. It is suitable for the situation that the recalled document fragments have met the requirements or the subsequent links have no special requirements for the data format. The implementation method of data rearrangement can be to rearrange the recalled document fragments according to certain rules, and can be rearranged based on indicators such as time sequence, relevance, and importance. The implementation method of data compression can be to streamline the recalled document fragments and reduce the amount of data, usually by removing redundant document fragments and extracting key document fragments. The implementation method of data fusion can be to integrate document fragments from different sources, different formats or with certain relevance to form a whole, so as to better meet user needs or facilitate subsequent analysis and processing.

[0039] Among them, the summary recall scenario is usually to query the summary of a specific document, such as "extract the summary content of the second section of Chapter 4 in document A". This type of problem requires accurate retrieval of relevant document fragments about the summary of the second section of Chapter 4 in document A from the database. The corresponding preset recall path can be summary recall and no processing.

[0040] The focus of the recommendation question and answer scenario is to filter out recommended items that meet the requirements from the database based on user needs. The database stores document recommendation question and answer information. It can perform similarity analysis on the questions entered by the user and the recommended questions corresponding to the older documents, and recall recommended questions and recommended answers based on the similarity. Therefore, the corresponding preset recall path can be question and answer recall.

[0041] The summary scenario requires identifying key information and core ideas from a large number of documents, such as extracting key information from numerous economic news, reports and other document data, and summarizing and summarizing them to form a relatively complete and concise summary answer. Taking "Summarize the main trends of global economic development this year" as an example, the corresponding recall path should be able to filter out relevant content about key trend information such as economic growth, inflation, and exchange rate changes from numerous economic documents. In order to recall more data to comprehensively analyze and answer the initial question, the corresponding preset recall path can be sequential text sorting recall, vector recall, chart recall, data fusion, data rearrangement and data compression. The input data volume of the large model is dynamically calculated based on the maximum tokens requirement corresponding to the preset large model and the output data volume corresponding to the initial question, and the recalled data volume is determined based on the input data volume of the large model and the input data volume corresponding to the initial question. At the same time, data screening, data fusion and data rearrangement operations are performed on the recalled data. Among them, when the total number of words in the recalled document data is less than the maximum tokens requirement corresponding to the large model, the original document can be directly recalled to improve the richness and completeness of the data. Tokens are the basic units of text processing by language models. Since the computing resources and memory of large models are limited, in order to ensure the performance and efficiency of the model, the number of tokens contained in the text is limited during the input or generation process, that is, the maximum tokens requirement is set.

[0042] Original text recall scenarios often require quick and accurate answers to original texts. Since documents are stored in the database, the questions entered by the user and the document content can be retrieved and analyzed to quickly locate the most directly relevant original text. The corresponding preset recall paths can be vector recall, data rearrangement, and data compression.

[0043] As described above, by obtaining the initial question input by the user, the preset large model, the initial scene category and the preset recall path corresponding to each initial scene category, a foundation is laid for the subsequent accurate and efficient processing of user questions and recall of appropriate document data.

[0044] S2, inputting the initial question and the first preset prompt word into the preset large model, obtaining a number of target question data, wherein the first preset prompt word includes a question decomposition prompt word and a question expansion prompt word, and the target question data includes a number of target questions and a question identifier corresponding to each target question.

[0045] In a specific implementation, the problem identifier includes a first problem identifier and a second problem identifier.

[0046] The preset large model is used to determine the preset question category corresponding to the initial question, wherein the preset question category includes related questions and non-related questions.

[0047] The preset large model is also used to obtain several decomposed questions and a first question identifier corresponding to each decomposed question based on the initial question and the problem decomposition prompt words when it is determined that the preset question category corresponding to the initial question is a related question, and to obtain several target questions corresponding to each decomposed question and a first question identifier corresponding to each target question based on each decomposed question, the problem identifier corresponding to each decomposed question and the problem expansion prompt words, wherein the first question identifier includes a main question and a sub-question.

[0048] The preset large model is also used to obtain several target questions corresponding to the initial question and the second question identifier corresponding to each target question based on the initial question and the question expansion prompt word when it is determined that the preset question category corresponding to the initial question is a non-related question, wherein the second question identifier includes a non-hierarchical question.

[0049] Among them, the preset question categories include related questions and non-related questions. Related questions mean that the initial question has a close internal connection with a specific subject, theme or context, and it is necessary to dig deep into various related elements for further analysis. Non-related questions are relatively more independent and general, and do not rely on deep related analysis of a specific thing.

[0050] When it is determined that the preset question category corresponding to the initial question is a related question, the preset large model decomposes the relatively complex, comprehensive, and broad initial related question into several easier to analyze and process, more specific decomposition questions based on the initial question and the problem decomposition prompt words, and obtains the first problem identifier corresponding to each decomposition question, that is, identifies whether each decomposition question is a main question or a sub-question, thereby clarifying the problem hierarchy structure, and then assisting the large model in planning an efficient problem-solving path. The large model then generates more detailed and diverse target questions and the first problem identifier corresponding to each target question based on each decomposition question, the problem identifier corresponding to each decomposition question, and the problem expansion prompt words, thereby gradually expanding from an initial related question to multiple more targeted target questions that can be explored in depth from different angles, which is helpful for the subsequent more comprehensive recall of document data related to the initial question.

[0051] When it is determined that the preset question category corresponding to the initial question is a non-related question, the large model refines and expands the general initial question based on the initial question and the question expansion prompt words, and obtains several target questions corresponding to the initial question and the second question identifier corresponding to each target question, which helps to more accurately recall document data related to the target question from the preset database in the future. For example, the initial question is "What are the common types of electronic product heat dissipation technology?", after adding the question expansion prompt words, the large model may generate "In the field of laptop computers, what are the common types of electronic product heat dissipation technology?", "From the perspective of heat dissipation principles, what are the common types of electronic product heat dissipation technology?", "What are the common types of heat dissipation technology for electronic products in different price ranges?" and other target questions, broadening the diversity and focus of the questions.

[0052] In a specific implementation, the related questions include parallel questions and context-dependent questions.

[0053] Among them, parallel questions involve information query on the same dimension of multiple subjects. When disassembling parallel questions, they can be disassembled according to the subject and the first question identifier corresponding to each disassembled question can be obtained, and then each disassembled question can be expanded to obtain the corresponding target question. For example, "What is the operating income of Company C and Company D in 2022?", where Company C and Company D are in a parallel relationship. In the actual document recall scenario, it is necessary to find the document part that can provide the relevant content of the operating income of Company C and Company D in 2022 from the preset database. The corresponding disassembled questions can be "What is the operating income of Company C in 2022?" and "What is the operating income of Company D in 2022?", and the corresponding first question identifiers are all main questions. The expanded target questions can be "What is the operating income of Company C in each quarter of 2022?", "How does Company C's operating income rank in the same industry in 2022?", "What is the operating income of Company D in each quarter of 2022?", "How does Company D's operating income rank in the same industry in 2022?", and the corresponding first question identifiers are all main questions.

[0054] The characteristic of the front-to-back dependency problem is that there is a logical sequence and dependency relationship between the front and back parts of the problem. When decomposing the front-to-back dependency problem, it is necessary to follow its inherent logical order to decompose it, such as decomposing it into basic information problems and related cause problems, or decomposing it into staged problems. For example, "What is the change in revenue of Company E from 2022 to 2021? What is the specific reason?", where the specific reason and revenue change belong to the front-to-back dependency relationship. In the actual document recall scenario, it is necessary to find a document that can provide data comparison and cause analysis related content at the same time. The corresponding decomposition questions can be "What is the change in revenue of Company E from 2022 to 2021?", and the corresponding first question is marked as the main question, and "What is the specific reason for the change in revenue of Company E from 2022 to 2021?", and the corresponding first question is marked as the sub-question. The expanded target question can be "What are the revenue changes of each business segment of Company E in 2022 and 2021 respectively?", and the corresponding first question is marked as the main question, "What are the revenue changes of Company E in different market regions in 2022 and 2021?", and the corresponding first question is marked as the main question, "From the perspective of market competition, what are the reasons for the change in Company E's revenue from 2022 to 2021?", and the corresponding first question is marked as a sub-question, "From the perspective of product path, what are the reasons for the change in Company E's revenue from 2022 to 2021?", and the corresponding first question is marked as a sub-question.

[0055] As mentioned above, by introducing question decomposition prompt words and question expansion prompt words, the original initial question is converted into a number of more targeted and detailed target questions and the corresponding question identifiers for each target question, which can improve the comprehensiveness of the large model's analysis of the initial question and provide a data basis for the subsequent more accurate recall of relevant document data from the preset database.

[0056] S3, extract key information and convert vectors for each target question to obtain a target key information list and target question vector corresponding to each target question.

[0057] Among them, key words or phrases are extracted from each target question to accurately reflect the core content of the target question, so as to quickly locate relevant content in the document data. The target question is converted into a vector form, so as to better measure the similarity between the target question and the document vector in the preset database in the vector space. In the high-dimensional vector space, the vector distance corresponding to texts with similar semantics will be relatively closer, so that when searching for relevant document data in the future, the document content that is closest to the semantics of the target question can be accurately recalled by calculating the similarity between vectors.

[0058] In a specific implementation, S3 includes the following steps:

[0059] S31, extract keywords for each target question and obtain a target keyword list corresponding to each target question.

[0060] S32, performing entity extraction on each target question to obtain a target entity list corresponding to each target question.

[0061] S33, obtaining a target key information list corresponding to each target question according to the target keyword list and the target entity list corresponding to each target question.

[0062] The specific keyword extraction method can be set by the implementer according to the actual situation. For example, the lexical analysis and part-of-speech tagging techniques in natural language processing can be used to identify key words such as nouns, verbs, and adjectives as keywords. It is also possible to combine some predefined domain vocabulary lists to further screen and determine important keywords to ensure that the extracted keywords can represent the key information of the target problem to the greatest extent.

[0063] Entity extraction can identify and extract entity information with specific meaning from the target question in text form, and organize the entities into a corresponding target entity list for further analysis, processing or use in other application scenarios. The specific entity extraction method can be set by the implementer according to the actual situation, for example, a rule-based method, a statistical machine learning method, or a deep learning method.

[0064] The specific vector conversion technology can be set by the implementer according to the actual situation. For example, using word embedding technology (such as Word2Vec, BERT and other pre-training models), first convert each word in the target problem into a corresponding vector representation, and then integrate the word vectors into the overall vector representation of the target problem by averaging, weighted averaging or a combination based on a neural network. Among them, those skilled in the art know that any keyword extraction method, any entity extraction method, and any vector conversion technology in the prior art fall within the protection scope of the present invention, and will not be repeated here.

[0065] As described above, multi-dimensional processing of key information and vectors for each target question lays the foundation for the next step of recalling document data related to the question from the database, thereby providing a data basis for subsequent more accurate and efficient reference to recall data from the preset database.

[0066] S4, inputting the target key information list, target question vector and second preset prompt word corresponding to each target question into the preset large model respectively, and obtaining the reference recall data corresponding to each target question, wherein the reference recall data includes several documents and the recall priority corresponding to each document.

[0067] Among them, the second preset prompt word plays the role of guiding and standardizing the large model to obtain reference recall data from the database. The specific content of the second preset prompt word can be set by the implementer according to the actual situation. For example, the third preset prompt word can be "Based on the above content, output the corresponding document from the database as reference recall data."

[0068] In a specific implementation, the reference recall data includes first reference recall data, second reference recall data and third reference recall data, and the initial document data includes a number of documents, initial key information data corresponding to the documents and initial vector data.

[0069] The preset large model is used to perform key information matching on the target key information list and the initial key information data, and to filter out first reference recall data from the initial document data according to the key information matching result, wherein the number of documents in the first reference recall data meets the first preset number.

[0070] The preset large model is also used to perform similarity matching on the target question vector and the initial vector data, and to filter out second reference recall data from the initial document data based on the similarity matching result, wherein the number of documents in the second reference recall data meets the second preset number.

[0071] The preset large model is also used to filter out third reference recall data from the data in the initial document data that has not been filtered as the first reference recall data and the second reference recall data, wherein the number of documents in the third reference recall data meets a third preset number.

[0072] The first preset number is greater than the second preset number, and the second preset number is greater than the third preset number.

[0073] Due to the huge number of documents in the initial document data, there may still be some potentially useful but unscreened document fragments under the two screening methods of key information matching and similarity matching. The third reference recall data is to screen again from the documents that have not been screened, aiming to dig out some documents with special angles or slightly weaker relevance but still of certain value, and further improve the comprehensiveness of the recalled documents. And as a data supplement for key information matching and similarity matching, the third preset number is set to be smaller than the first preset number and the second preset number, so as to form a hierarchical and focused document recall system, thereby improving the accuracy of the entire RAG recall method.

[0074] In a specific implementation, the initial key information data includes an initial key information set corresponding to each document, the key information matching result includes the number of key information repetitions between the initial key information set corresponding to each document and the target key information list, and the first reference recall data is screened out according to the key information matching result, including:

[0075] Documents that meet a first preset number are screened out in order of the number of key information repetitions from most to least, and first reference recall data is formed based on the screened documents that meet the first preset number.

[0076] Among them, when there is a lot of repeated key information between the target key information list and the initial key information set corresponding to the document, that is, there are more repeated keywords and repeated entities, the document is likely to be closely related to the target question in terms of content theme, and can provide more direct and clear information for answering the target question. Therefore, the first reference recall data is given the largest data volume quota to ensure that as many highly relevant documents as possible are recalled, providing sufficient material basis for subsequent analysis from the key information level of the text surface.

[0077] Specifically, documents meeting a first preset number are screened out in order of the number of key information repetitions from most to least, to ensure that the screened out first reference recall data has a strong correlation with the initial question, thereby improving the accuracy of the entire RAG recall method.

[0078] In a specific implementation, the initial vector data includes an initial vector set corresponding to each document, the similarity matching result includes the vector similarity between the initial vector set corresponding to each document and the target question vector, and the second reference recall data is screened out according to the similarity matching result, including:

[0079] Documents that meet the second preset number are screened out in descending order of similarity, and second reference recall data are formed based on the screened documents that meet the second preset number.

[0080] Among them, some documents may not have a large amount of repeated key information with the target question, but the overall semantic expression is highly consistent with the target question. Therefore, on the basis of key information matching, this embodiment also calculates the similarity between the target question vector and the initial vector set corresponding to the document, and recalls documents that are similar to the target question at the semantic level, so as to supplement the document information from a deep semantic understanding perspective, and provide richer and more semantically appropriate reference content for question answering. And as a data supplement for key information matching, the second preset number is set to be smaller than the first preset number to form a hierarchical and focused document recall system, thereby improving the accuracy of the entire RAG recall method.

[0081] Specifically, documents meeting a second preset number are screened out in descending order of similarity to ensure that the screened out second reference recall data is highly consistent with the initial question, thereby improving the accuracy and reliability of the entire RAG recall method.

[0082] As mentioned above, we first recall a large number of highly relevant documents based on key information matching, then select documents with high semantic fit through semantic vector similarity for supplementation, and finally use a comprehensive approach to fill in the gaps and select the third reference recall data, to ensure to the greatest extent possible that the recalled documents not only comprehensively cover the content directly related to the target question, but also take into account the deep semantic relevance and some potentially valuable information, providing a solid and rich material foundation for generating accurate final results that meet the user's initial question requirements, and improving the accuracy of the entire large-model RAG recall strategy intelligent planning method.

[0083] S5, input each initial scene category, the preset recall path corresponding to each initial scene category, each target question, the reference recall data corresponding to each target question and the third preset prompt word into the preset large model, and obtain the target recall path corresponding to the initial question, wherein the target recall path includes several sequential data recall links.

[0084] Among them, the third preset prompt word plays the role of guiding and regulating the model to generate the target recall path. The specific content of the third preset prompt word can be set by the implementer according to the actual situation. For example, the third preset prompt word can be "output the corresponding target recall path according to the above content."

[0085] The initial scenario category clarifies the big framework and background of the problem. Different initial scenario categories have different task priorities and data processing logic, corresponding to different preset recall paths. The preset recall path is a relatively standardized recall process obtained through a lot of practice, test analysis, etc. in the early stage. Each initial scenario category and the preset recall path corresponding to each initial scenario category provide a basic reference framework for the initial problem to be processed, so that the large model can be adjusted and optimized on the basis of the preset recall path according to the characteristics of the initial problem, ensuring the scientificity and effectiveness of the target recall path.

[0086] The target question clearly indicates the specific question or task to be solved, allowing the large model to accurately plan the recall path around the target question and ensure that the final recalled data is highly relevant to the initial question corresponding to the solution of the target question.

[0087] The reference recall data corresponding to each target question is a document that has a certain relevance to the target question, which helps the big model better understand the data range, characteristics, and association patterns involved in the target question, and helps the big model to construct the target recall path more reasonably.

[0088] In a specific implementation, S5 further includes the following steps:

[0089] S51, obtaining M historical similar questions corresponding to the initial question and the target recall path corresponding to each historical similar question.

[0090] S52, input each target question, each initial scene category, the preset recall path corresponding to each initial scene category, the reference recall data corresponding to each target question, M historical similar questions, the target recall path corresponding to each historical similar question and the third preset prompt word into the preset large model to obtain the target recall path corresponding to the initial question.

[0091] Among them, historical similar problems are problems that have been dealt with in the past and are similar to the current initial problem in terms of problem nature, theme, structure, etc. The target recall paths determined for similar problems in the past contain valuable experience and effective strategies. Therefore, historical similar problems and the corresponding target recall paths are input into the preset large model together to obtain the target recall path corresponding to the initial problem, so as to improve the efficiency and accuracy of generating the target recall path for the current initial problem.

[0092] In a specific implementation, S5 further includes the following steps:

[0093] S53, obtaining the description text corresponding to each preset data recall link.

[0094] S54, input each target question, each initial scene category, the preset recall path corresponding to each initial scene category, the reference recall data corresponding to each target question, the descriptive text corresponding to each preset data recall link and the third preset prompt word into the preset large model to obtain the target recall path corresponding to the initial question.

[0095] The description text elaborates on the specific working principles, operation methods, applicable scenarios, and related features of each data recall link. For example, the description text of the text sorting recall link can introduce how to score and sort based on factors such as keyword distribution and semantic relevance in the document. The description text of the vector recall link can introduce how to convert the text in the document into a vector representation, and how to perform data recall by calculating the similarity between vectors. By adding the description text of the data recall link to the input content of the large model, it helps the large model to grasp the details of each recall link more deeply, so as to improve the efficiency and accuracy of generating the recall path of the current initial problem target.

[0096] In a specific implementation, S5 further includes the following steps:

[0097] S55, obtaining M historical similar questions corresponding to the initial question, the target recall path corresponding to each historical similar question, and the description text corresponding to each preset data recall link.

[0098] S56, input each target question, each initial scene category, the preset recall path corresponding to each initial scene category, the reference recall data corresponding to each target question, M historically similar questions, the target recall path corresponding to each historically similar question, the descriptive text corresponding to each preset data recall link and the third preset prompt word into the preset large model to obtain the target recall path corresponding to the initial question.

[0099] As mentioned above, with the help of the preset big model, multi-dimensional reference information is provided to the big model based on a variety of input information, which assists the big model to obtain the target recall path composed of multiple data recall links arranged in sequence. The orderliness of the data recall links makes the data recall process more efficient, and the target recall path can be adjusted according to the specific target problem, so that the entire big model RAG recall strategy intelligent planning method can adapt to various complex and changeable application scenarios and problem types, and ensure the quality of the final target recall data.

[0100] S6, executing all data recall links in the target recall path in sequence to obtain the target recall data corresponding to the initial question, wherein the target recall data includes several document fragments.

[0101] Among them, the target recall data is presented in the form of several document fragments, which can be further sorted, analyzed, summarized, and other operations to provide strong support for related decision-making, research or other applications.

[0102] In the above, by introducing the problem decomposition prompt words and the problem expansion prompt words, the original initial question is converted into a number of more targeted and detailed target questions, which improves the comprehensiveness of the large model's analysis of the initial question, performs multi-dimensional processing of key information and vectors for each target question, and inputs the target key information list, target question vector and second preset prompt words corresponding to each target question into the preset large model respectively, obtains the reference recall data corresponding to each target question, and inputs each initial scene category, the preset recall path corresponding to each initial scene category, each target question, the reference recall data corresponding to each target question and the third preset prompt words into the preset large model, obtains the target recall data corresponding to the initial question. Path, execute all data recall links in the target recall path in sequence, obtain the target recall data corresponding to the initial question, and provide multi-dimensional reference information for the large model based on the multi-dimensional input information, giving full play to the natural language processing capabilities of the large model and the guiding role of different prompt words, thereby assisting the large model to obtain the target recall path composed of multiple data recall links arranged in sequence, making the data recall process more efficient, and the target recall path can be adjusted according to the specific target problem, so that the entire large model RAG recall strategy intelligent planning method can adapt to various complex and changeable application scenarios and problem types, improving the accuracy and efficiency of the entire large model RAG recall strategy intelligent planning method.

[0103] Embodiment 2

[0104] The second embodiment provides a large model RAG recall strategy intelligent planning device, the large model RAG recall strategy intelligent planning device includes: Figure 2 As shown:

[0105] The data acquisition module 21 is used to obtain the initial question input by the user, the preset large model, the preset database, the initial scene category and the preset recall path corresponding to each initial scene category, wherein the preset database stores the initial document data, and each preset recall path includes a number of sequential data recall links.

[0106] The target question acquisition module 22 is used to input the initial question and the first preset prompt word into the preset large model to obtain a number of target question data, wherein the first preset prompt word includes a question decomposition prompt word and a question expansion prompt word, and the target question data includes a number of target questions and a question identifier corresponding to each target question.

[0107] The target problem processing module 23 is used to extract key information and convert vectors for each target problem, and obtain a target key information list and a target problem vector corresponding to each target problem.

[0108] The first data recall module 24 is used to input the target key information list, target question vector, initial document data and second preset prompt word corresponding to each target question into the preset large model respectively, and obtain the reference recall data corresponding to each target question, wherein the reference recall data includes a number of documents and the recall priority corresponding to each document.

[0109] The target recall path acquisition module 25 is used to input each initial scene category, the preset recall path corresponding to each initial scene category, each target question, the reference recall data corresponding to each target question and the third preset prompt word into the preset large model, and obtain the target recall path corresponding to the initial question, wherein the target recall path includes several sequential data recall links.

[0110] The second data recall module 26 is used to sequentially execute all data recall links in the target recall path to obtain the target recall data corresponding to the initial question, wherein the target recall data includes a plurality of document fragments.

[0111] In a specific implementation, in the target problem acquisition module 22, the problem identifier includes a first problem identifier and a second problem identifier.

[0112] The preset large model is used to determine the preset question category corresponding to the initial question, wherein the preset question category includes related questions and non-related questions.

[0113] The preset large model is also used to obtain a number of decomposed questions and a problem identifier corresponding to each decomposed question based on the initial question and the problem decomposition prompt words when it is determined that the preset question category corresponding to the initial question is a related question, and to obtain a number of target questions corresponding to each decomposed question and a first problem identifier corresponding to each target question based on each decomposed question, the first problem identifier corresponding to each decomposed question and the problem expansion prompt words, wherein the first problem identifier includes a main question and a sub-question.

[0114] The preset large model is also used to obtain several target questions corresponding to the initial question and the second question identifier corresponding to each target question based on the initial question and the question expansion prompt word when it is determined that the preset question category corresponding to the initial question is a non-related question, wherein the second question identifier includes a non-hierarchical question.

[0115] In a specific implementation, the target problem processing module 23 includes:

[0116] The keyword extraction submodule is used to extract keywords for each target question and obtain a target keyword list corresponding to each target question.

[0117] The entity extraction submodule is used to extract entities for each target question and obtain a list of target entities corresponding to each target question.

[0118] The key information list acquisition submodule is used to obtain the target key information list corresponding to each target question based on the target keyword list and target entity list corresponding to each target question.

[0119] In a specific implementation, in the target recall path acquisition module 25, the reference recall data includes first reference recall data, second reference recall data and third reference recall data, and the initial document data includes several documents, initial key information data corresponding to the documents and initial vector data.

[0120] The preset large model is used to perform key information matching on the target key information list and the initial key information data, and to filter out first reference recall data from the initial document data according to the key information matching result, wherein the number of documents in the first reference recall data meets the first preset number.

[0121] The preset large model is also used to perform similarity matching on the target question vector and the initial vector data, and to filter out second reference recall data from the initial document data based on the similarity matching result, wherein the number of documents in the second reference recall data meets the second preset number.

[0122] The preset large model is also used to filter out third reference recall data from the data in the initial document data that has not been filtered as the first reference recall data and the second reference recall data, wherein the number of documents in the third reference recall data meets a third preset number.

[0123] In a specific implementation, the target recall path acquisition module 25 further includes:

[0124] The historical data acquisition submodule is used to obtain M historical similar questions corresponding to the initial question and the target recall path corresponding to each historical similar question.

[0125] The first target recall path acquisition submodule is used to input each target question, each initial scene category, the preset recall path corresponding to each initial scene category, the reference recall data corresponding to each target question, M historical similar questions, the target recall path corresponding to each historical similar question and the third preset prompt word into the preset large model to obtain the target recall path corresponding to the initial question.

[0126] In a specific implementation, the target recall path acquisition module 25 further includes:

[0127] The description text acquisition submodule is used to obtain the description text corresponding to each preset data recall link.

[0128] The second target recall path acquisition submodule is used to input each target question, each initial scene category, the preset recall path corresponding to each initial scene category, the reference recall data corresponding to each target question, the descriptive text corresponding to each preset data recall link and the third preset prompt word into the preset large model to obtain the target recall path corresponding to the initial question.

[0129] In a specific implementation, in the data acquisition module 21, the initial scenario categories include summary scenarios, recommendation question and answer scenarios, original text recall scenarios, and summary recall scenarios.

[0130] The preset data recall link includes a data recall method and a data processing method. The data recall method includes at least one of text sorting recall, vector recall, summary recall, question and answer recall, chart recall, full text recall, summary recall, and external data recall. The data processing method includes at least one of no processing, data rearrangement, data compression, and data fusion.

[0131] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0132] Embodiment 3

[0133] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the steps:

[0134] S1, obtain the initial question input by the user, the preset large model, the preset database, the initial scene category and the preset recall path corresponding to each initial scene category, wherein the preset database stores the initial document data, and each preset recall path includes several sequential data recall links.

[0135] S2, inputting the initial question and the first preset prompt word into the preset large model, obtaining a number of target question data, wherein the first preset prompt word includes a question decomposition prompt word and a question expansion prompt word, and the target question data includes a number of target questions and a question identifier corresponding to each target question.

[0136] S3, extract key information and convert vectors for each target question to obtain a target key information list and target question vector corresponding to each target question.

[0137] S4, input the target key information list, target question vector, initial document data and second preset prompt word corresponding to each target question into the preset large model respectively, and obtain the reference recall data corresponding to each target question, wherein the reference recall data includes several documents and the recall priority corresponding to each document.

[0138] S5, input each initial scene category, the preset recall path corresponding to each initial scene category, each target question, the reference recall data corresponding to each target question and the third preset prompt word into the preset large model, and obtain the target recall path corresponding to the initial question, wherein the target recall path includes several sequential data recall links.

[0139] S6, executing all data recall links in the target recall path in sequence to obtain the target recall data corresponding to the initial question, wherein the target recall data includes several document fragments.

[0140] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronization link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0141] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0142] Embodiment 4

[0143] A fourth embodiment of the present invention provides an electronic device, which includes a processor and the non-transitory computer-readable storage medium in the third embodiment of the present invention.

[0144] The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A large-model RAG recall strategy intelligent planning method, characterized in that: The large model RAG recall strategy intelligent planning method comprises the following steps: S1, obtaining an initial question input by a user, a preset large model, a preset database, an initial scene category, and a preset recall path corresponding to each initial scene category, wherein the preset database stores initial document data, and each preset recall path includes a plurality of sequential data recall links; S2, inputting the initial question and the first preset prompt word into the preset large model to obtain a plurality of target question data, wherein the first preset prompt word includes a question decomposition prompt word and a question expansion prompt word, and the target question data includes a plurality of target questions and a question identifier corresponding to each target question; S3, extract key information and convert vectors for each target question to obtain a target key information list and target question vector corresponding to each target question; S4, inputting the target key information list, the target question vector, the initial document data and the second preset prompt word corresponding to each target question into the preset large model respectively, and obtaining the reference recall data corresponding to each target question, wherein the reference recall data includes a number of documents and the recall priority corresponding to each document; S5, inputting each initial scene category, a preset recall path corresponding to each initial scene category, each target question, reference recall data corresponding to each target question, and a third preset prompt word into the preset large model, and obtaining a target recall path corresponding to the initial question, wherein the target recall path includes a plurality of sequential data recall links; S6, executing all data recall links in the target recall path in sequence to obtain the target recall data corresponding to the initial question, wherein the target recall data includes a plurality of document fragments.

2. The large model RAG recall strategy intelligent planning method according to claim 1 is characterized in that: The problem identifier includes a first problem identifier and a second problem identifier; The preset large model is used to determine the preset question category corresponding to the initial question, wherein the preset question category includes related questions and non-related questions; The preset large model is also used to obtain, when it is determined that the preset question category corresponding to the initial question is a related question, a number of disassembled questions and a question identifier corresponding to each disassembled question according to the initial question and the question disassembly prompt word, and to obtain a number of target questions corresponding to each disassembled question and a first question identifier corresponding to each target question according to each disassembled question, the first question identifier corresponding to each disassembled question and the question expansion prompt word, wherein the first question identifier includes a main question and a sub-question; The preset large model is also used to obtain several target questions corresponding to the initial question and a second question identifier corresponding to each target question based on the initial question and the question expansion prompt word when it is determined that the preset question category corresponding to the initial question is a non-related question, wherein the second question identifier includes a non-hierarchical question.

3. The large model RAG recall strategy intelligent planning method according to claim 1 is characterized in that: S3 includes the following steps: S31, extract keywords for each target question to obtain a target keyword list corresponding to each target question; S32, performing entity extraction on each target question to obtain a target entity list corresponding to each target question; S33, obtaining a target key information list corresponding to each target question according to the target keyword list and the target entity list corresponding to each target question.

4. The large model RAG recall strategy intelligent planning method according to claim 1 is characterized in that: The reference recall data includes first reference recall data, second reference recall data and third reference recall data, and the initial document data includes a plurality of documents, initial key information data corresponding to the documents and initial vector data; The preset large model is used to perform key information matching on the target key information list and the initial key information data, and to filter out first reference recall data from the initial document data according to the key information matching result, wherein the number of documents in the first reference recall data meets a first preset number; The preset large model is also used to perform similarity matching on the target question vector and the initial vector data, and to filter out second reference recall data from the initial document data according to the similarity matching result, wherein the number of documents in the second reference recall data meets the second preset number; The preset large model is also used to filter out third reference recall data from the data in the initial document data that has not been filtered as the first reference recall data and the second reference recall data, wherein the number of documents in the third reference recall data meets a third preset number.

5. The large model RAG recall strategy intelligent planning method according to claim 1 is characterized in that: S5 also includes the following steps: S51, obtaining M historical similar questions corresponding to the initial question and a target recall path corresponding to each historical similar question; S52, input each target question, each initial scene category, the preset recall path corresponding to each initial scene category, the reference recall data corresponding to each target question, M historically similar questions, the target recall path corresponding to each historically similar question and the third preset prompt word into the preset large model to obtain the target recall path corresponding to the initial question.

6. The large model RAG recall strategy intelligent planning method according to claim 1 is characterized in that: S5 also includes the following steps: S53, obtaining a description text corresponding to each preset data recall link; S54, input each target question, each initial scene category, the preset recall path corresponding to each initial scene category, the reference recall data corresponding to each target question, the descriptive text corresponding to each preset data recall link and the third preset prompt word into the preset large model to obtain the target recall path corresponding to the initial question.

7. The large model RAG recall strategy intelligent planning method according to claim 1 is characterized in that: The initial scenario categories include summary scenarios, recommendation question-answering scenarios, original text recall scenarios, and summary recall scenarios; The preset data recall link includes a data recall method and a data processing method. The data recall method includes at least one of text sorting recall, vector recall, summary recall, question and answer recall, chart recall, full text recall, summary recall, and external data recall. The data processing method includes at least one of no processing, data rearrangement, data compression, and data fusion.

8. A large-model RAG recall strategy intelligent planning device, characterized in that: The large model RAG recall strategy intelligent planning device includes: A data acquisition module, used to acquire an initial question input by a user, a preset large model, a preset database, an initial scene category, and a preset recall path corresponding to each initial scene category, wherein the preset database stores initial document data, and each preset recall path includes a number of sequential data recall links; A target question acquisition module, used to input the initial question and the first preset prompt word into the preset large model, and acquire a plurality of target question data, wherein the first preset prompt word includes a question decomposition prompt word and a question expansion prompt word, and the target question data includes a plurality of target questions and a question identifier corresponding to each target question; The target problem processing module is used to extract key information and convert vectors for each target problem, and obtain the target key information list and target problem vector corresponding to each target problem; A first data recall module is used to input the target key information list corresponding to each target question, the target question vector, the initial document data and the second preset prompt word into the preset large model respectively, and obtain reference recall data corresponding to each target question, wherein the reference recall data includes a number of documents and a recall priority corresponding to each document; A target recall path acquisition module is used to input each initial scene category, a preset recall path corresponding to each initial scene category, each target question, reference recall data corresponding to each target question, and a third preset prompt word into the preset large model, and acquire a target recall path corresponding to the initial question, wherein the target recall path includes a plurality of sequential data recall links; The second data recall module is used to execute all data recall links in the target recall path in sequence to obtain the target recall data corresponding to the initial question, wherein the target recall data includes a plurality of document fragments.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the large model RAG recall strategy intelligent planning method as described in any one of claims 1-7.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images