Retrieval enhancement generation method based on large language model and related equipment

Abstract

The invention provides a retrieval enhancement generation method based on a large language model and related equipment, and relates to the technical field of natural language processing. The method comprises the following steps: when an input question needs to be retrieved, determining a retrieval mode corresponding to the input question; the input question is retrieved according to the retrieval mode, and knowledge fragments are determined; and inputting the knowledge fragments and the input questions into the large language model, and determining an output result. According to the method, the corresponding retrieval method is determined according to the type of the input problem, the retrieval quality is improved, introduction of unnecessary information can be reduced, and the problem that unnecessary retrieval information affects large language model output is solved.

Description

Background Art

[0002] With the rapid development of artificial intelligence technology, word vector language models based on neural networks and large-scale pre-trained language models have gradually become the main methods for solving natural language processing problems. Although large language models perform well in many tasks, they cannot meet actual business needs due to their knowledge limitations.

[0003] In the prior art, retrieval-enhanced generation is used to overcome the limitations of large language model knowledge. However, since the retrieved information contains unnecessary information, directly using the retrieved information will affect the output of the large language model.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a retrieval enhancement generation method based on a large language model and related devices, which at least to a certain extent overcome the problem that unnecessary retrieval information in the related art affects the output of the large language model.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a retrieval enhancement generation method based on a large language model is provided, including: when an input question needs to be retrieved, determining a retrieval method corresponding to the input question; searching the input question according to the retrieval method to determine a knowledge fragment; inputting the knowledge fragment and the input question into a large language model to determine an output result.

[0008] In some embodiments, when the input question needs to be retrieved, before determining the retrieval method corresponding to the input question, the method further includes: obtaining the input question; and determining whether the input question needs to be retrieved.

[0009] In some embodiments, determining whether the input question needs to be retrieved includes: determining whether the input question needs to be retrieved from multiple preset dimensions, wherein the multiple preset dimensions include: specificity, professionalism, timeliness, privacy, and common sense.

[0010] In some embodiments, the method further includes: summing up the classification scores of the multiple preset dimensions to determine a total score; and determining whether the input question needs to be retrieved based on whether the total score reaches a threshold.

[0011] In some embodiments, the method further comprises: retrieving the input question when the uncertainty score of the self-awareness exceeds an empirical threshold.

[0012] In some embodiments, when the input question needs to be retrieved, determining the retrieval method corresponding to the input question includes: classifying the input question and determining the classification result; and determining the retrieval method corresponding to the input question based on the classification result.

[0013] In some embodiments, the classification results include: numerical questions, statistical questions, and open questions; searching the input questions according to the retrieval method to determine the knowledge fragments includes: when the classification result is a numerical question, searching the target table for the corresponding numerical value to determine the knowledge fragment; when the classification result is a statistical question, querying the target database to determine the knowledge fragment; when the classification result is an open question, performing a vector search in the target document to determine the knowledge fragment.

[0014] In some embodiments, inputting the knowledge fragment and the input question into a large language model to determine the output result includes: using a prompt word template to splice the knowledge fragment and the input question to determine splicing information; inputting the splicing information into a large language model to determine the output result.

[0015] According to another aspect of the present disclosure, a retrieval enhancement generation device based on a large language model is also provided, including: a retrieval method determination module, used to determine the retrieval method corresponding to the input question when the input question needs to be retrieved; a knowledge fragment determination module, used to search the input question according to the retrieval method and determine the knowledge fragment; an output result determination module, used to input the knowledge fragment and the input question into the large language model and determine the output result.

[0016] According to another aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned retrieval enhancement generation methods based on a large language model by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for generating retrieval enhancement based on a large language model described above is implemented.

[0018] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned retrieval enhancement generation methods based on a large language model.

[0019] The embodiment of the present disclosure provides a large language model-based retrieval enhancement generation. When an input question needs to be retrieved, a retrieval method corresponding to the input question is determined; the input question is retrieved according to the retrieval method to determine the knowledge fragment; the knowledge fragment and the input question are input into the large language model to determine the output result. The present disclosure determines the corresponding retrieval method according to the type of input question, improves the quality of the retrieval, can reduce the introduction of unnecessary information, and solves the problem that unnecessary retrieval information affects the output of the large language model.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 A schematic diagram showing a structure of a retrieval enhancement generation system based on a large language model in an embodiment of the present disclosure;

[0023] Figure 2 A flow chart of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown;

[0024] Figure 3 A flowchart showing a specific example of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure;

[0025] Figure 4 A flowchart of another specific example of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown;

[0026] Figure 5 A flowchart of another specific example of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown;

[0027] Figure 6 Another specific example flow chart of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown;

[0028] Figure 7 Another specific example flow chart of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown;

[0029] Figure 8A flowchart of another specific example of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown;

[0030] Fig. 9 A flowchart of a specific example of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown;

[0031] Fig.10 A schematic diagram of a retrieval enhancement generation device based on a large language model in an embodiment of the present disclosure is shown;

[0032] Fig.11 A structural block diagram of a computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0036] SQL:Structured Query Language server database, Structured Query Language;

[0037] BM25: Best Matching 25 is a probabilistic retrieval algorithm for information retrieval and text search. It is part of the BM series of algorithms and is designed to evaluate the relevance between documents and queries.

[0038] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0039] Figure 1FIG. 1 shows an exemplary application system architecture diagram to which the retrieval enhancement generation method based on a large language model in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0040] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0041] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0042] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0043] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0044] In one example of the present disclosure, a terminal device is used to perform the following steps: when an input question needs to be retrieved, determine a retrieval method corresponding to the input question; search the input question according to the retrieval method to determine a knowledge fragment; input the knowledge fragment and the input question into a large language model to determine an output result.

[0045] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0046] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0047] In one example of the present disclosure, the server is used to perform the following steps: when an input question needs to be retrieved, determine the retrieval method corresponding to the input question; search the input question according to the retrieval method to determine the knowledge fragment; input the knowledge fragment and the input question into the large language model to determine the output result.

[0048] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0049] Figure 2 A flowchart of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown. Figure 2 As shown, the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure includes the following steps:

[0050] S202: When a search is required for an input question, a search method corresponding to the input question is determined.

[0051] It should be noted that the above-mentioned input question can be a question that requires an answer or explanation, for example, where is Xiao Ming's father born? The above-mentioned search method can be the method used for the search. For example, if the input question requires providing specific numbers or calculation results, it may be a numerical question, and a numerical search method is used; if the question requires analyzing data or explaining data trends, or inquiring about probability, it may be a statistical question, and a statistical search method is used; if the question seeks opinions, explanations, suggestions or extensive discussions, rather than a single fact or value, it is an open question, and an open search method is used.

[0052] S204, searching the input question according to the search method to determine the knowledge fragment.

[0053] It should be noted that the above-mentioned knowledge fragments can be information obtained after retrieval. For example, if the classification result is a numerical question, the corresponding numerical value is found from the table, and the numerical value is the knowledge fragment. Among them, the table is a retrieved table related to the question. For example, if the total amount of gross domestic product is asked, the gross domestic product table is searched. If the classification result is a statistical question, the large language model is called to output the SQL statement required to solve the problem, and the result obtained by querying in the database is the knowledge fragment. If the classification result is an open question, the result obtained by using the BM25 algorithm to perform vector retrieval in the document is the knowledge fragment.

[0054] S206, input the knowledge fragment and the input question into the large language model to determine the output result.

[0055] It should be noted that the above-mentioned large language model can be a model for processing a variety of natural speech tasks.

[0056] The present disclosure determines a corresponding retrieval method according to the type of input question, improves the quality of retrieval, can reduce the introduction of unnecessary information, and solves the problem that unnecessary retrieval information affects the output of a large language model.

[0057] In a specific example, when an input question needs to be retrieved, before determining the retrieval method corresponding to the input question, the retrieval enhancement generation method based on the large language model also includes: obtaining the input question; and determining whether the input question needs to be retrieved.

[0058] In one embodiment of the present disclosure, Figure 3 As shown, the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure can determine whether an input question needs to be retrieved through the following steps, and can accurately select a corresponding retrieval method according to the type of question:

[0059] S302, judging whether the input question needs to be searched from multiple preset dimensions, where the multiple preset dimensions include: specificity, professionalism, timeliness, privacy, and common sense.

[0060] For example, a classifier is trained and used to classify questions from multiple dimensions. The classifier can determine whether the question needs to be retrieved from the following dimensions: the first dimension is whether the question seeks specific information, data or facts (equivalent to the above-mentioned specificity); the second dimension is whether the question involves complex concepts, theories or professional field knowledge (equivalent to the above-mentioned professionalism); the third dimension is whether the question is timely, such as whether it is about the latest events, news or data (equivalent to the above-mentioned timeliness); the fourth dimension is whether the question involves the user's personal information query (equivalent to the above-mentioned privacy); the fifth dimension is whether the question is a common sense, widely recognized question (equivalent to the above-mentioned common sense). If so, no retrieval is required. If any one dimension is not or any preset number of dimensions are not, a retrieval is performed.

[0061] In one embodiment of the present disclosure, Figure 4 As shown, the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure can determine whether an input question needs to be retrieved through the following steps, can determine whether a retrieval is needed based on the overall latitude, and can enhance robustness:

[0062] S402, summing up the classification scores of multiple preset dimensions to determine a total score;

[0063] S404: Determine whether the input question needs to be searched based on whether the total score reaches a threshold.

[0064] For example, the classification scores of multiple dimensions are summed up, and whether to perform retrieval is determined based on whether the total score reaches a threshold.

[0065] In one embodiment of the present disclosure, Figure 5 As shown, the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure can determine whether an input question needs to be retrieved through the following steps, can determine whether a retrieval is needed based on the uncertainty score of self-awareness, and can reduce invalid retrieval:

[0066] S502, when the uncertainty score of self-awareness exceeds the experience threshold, the input question is retrieved.

[0067] For example, based on the self-perceived uncertainty estimate, it is further determined whether a search is needed. If the uncertainty score of self-awareness exceeds the empirical threshold, the search is triggered. Specifically, after the large language model completes the output, all input and output (including the terminator) are extracted. <eos>) and average it as the vector representation of the entire context, which contains the information in all tokens. The same input is used multiple times to extract multiple sets of vector representations, and these vector representations are used to construct a covariance matrix, which can measure the correlation between each set of vectors. The covariance matrix is ​​then regularized and its determinant is calculated as the self-perceived uncertainty score. Self-perceived uncertainty is used to determine which situations require a wider range of information sources, thereby avoiding meaningless retrieval of questions that obviously do not require additional information. When the uncertainty score exceeds the threshold, it means that the system believes that the current problem or situation requires more in-depth exploration. At this time, triggering a retrieval can ensure that key information closely related to the problem is obtained.

[0068] In one embodiment of the present disclosure, Figure 6 As shown, in the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure, when an input question needs to be retrieved, determining the retrieval method corresponding to the input question includes the following steps:

[0069] S602, classifying the input question and determining the classification result;

[0070] S604: Determine a search method corresponding to the input question based on the classification result.

[0071] For example, train a classifier and use it to classify questions. For example, if a question requires specific numbers or calculation results, it may be a numerical question; if a question requires data analysis or interpretation of data trends, or asks about probability, it may be a statistical question; if a question seeks opinions, explanations, suggestions, or extensive discussions rather than single facts or values, it is an open question. Decide which retrieval method to use based on the classification results. Using a classifier to classify the types of questions and decide which retrieval method to use can improve the efficiency and accuracy of retrieval.

[0072] In one example of the present disclosure, the classification results include: numerical questions, statistical questions, and open questions; Figure 7 As shown, in the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure, searching the input question according to the retrieval method, and determining the knowledge fragment includes the following steps:

[0073] S702, when the classification result is a numerical question, searching the target table for a corresponding numerical value to determine it as a knowledge fragment;

[0074] S704, when the classification result is a statistical problem, query and determine the knowledge fragment in the target database;

[0075] S706: When the classification result is an open question, a vector search is performed in the target document to determine the knowledge fragment.

[0076] For example, if the classification result is a numerical problem, the corresponding value is found from the table. If the classification result is a statistical problem, the large language model is called to output the SQL statement required to solve the problem and query it in the database. If the classification result is an open question, the BM25 algorithm is used in the document for vector retrieval. For numerical problems, directly finding the corresponding value from the table is direct and efficient. It is suitable for processing problems with clear data tables and fields, reducing the intermediate links of data processing, and improving accuracy and response speed. For statistical problems. By calling a large language model (such as the GPT series) to generate the SQL statement required to solve the problem and querying it in the database, the advantages of natural language processing and database query are utilized, and it can flexibly respond to complex statistical needs while maintaining the accuracy and efficiency of the query. For open questions, the BM25 algorithm is used for vector retrieval, which is an effective information retrieval technology that can find the most relevant results in a large number of documents based on the content of the question. The BM25 algorithm takes into account the importance of vocabulary (such as TF-IDF) and position information, thereby improving the accuracy and relevance of retrieval.

[0077] In an example of the present disclosure, after searching using the corresponding search method, the method further includes: reordering the retrieved contents.

[0078] For example, the retrieved knowledge fragments are sequentially concatenated with the questions and input into the large language model, the corresponding self-perceived uncertainty scores are evaluated, and the knowledge fragment with the smallest uncertainty is selected as the retrieved information.

[0079] In one embodiment of the present disclosure, Figure 8 As shown, in the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure, a knowledge fragment and an input question are input into the large language model, and determining the output result includes the following steps:

[0080] S802, using the prompt word template to splice the knowledge fragment and the input question, and determining splicing information;

[0081] S804, input the concatenated information into the large language model to determine the output result.

[0082] For example, use a prompt word template to splice the retrieved information and the input question together, for example: "Please answer the question based on the following information: [question]. The relevant information is as follows: [information 1], [information 2]. ". Input the spliced ​​information into the large language model. [question]: Where was Zhang San's father born? [Information 1]: Zhang San's father is Zhang XX. [Information 2]: Zhang XX was born in a certain city in a certain province.

[0083] The present disclosure is used to perform knowledge enhancement on a large language model, using the retrieved knowledge to enhance the model performance only when necessary, and determining the retrieval method to be used according to the type of question, thereby improving the quality of retrieved information, reducing unnecessary retrieval operations, thereby reducing the time required for reasoning, and shortening the length of the input content. Specifically, the input content is first classified to determine whether to perform a retrieval, which avoids the introduction of unnecessary retrieval operations. Secondly, the type of question is combined to determine which retrieval method to use, thereby improving the quality of the retrieval. Since the optimal retrieval methods required for different questions are different, using an inappropriate retrieval method to retrieve information will reduce the retrieval effect.

[0084] Fig. 9 A flowchart of a specific example of a retrieval enhancement generation method based on a large language model in an embodiment of the present disclosure is shown. Fig. 9 As shown, the retrieval enhancement generation method based on a large language model provided in the embodiment of the present disclosure includes the following steps:

[0085] S901, obtaining input questions;

[0086] S902, check whether the input question needs to be searched, if yes, jump to S903, if no, jump to S908;

[0087] S903, detecting which search method is used, if it is a numerical question, jumping to S904, if it is a statistical question, jumping to S905, if it is an open question, jumping to S906;

[0088] S904, search for the corresponding value in the table and jump to S907;

[0089] S905, calling the large language model to output the SQL statement required to solve the input problem, querying in the database, and jumping to S907;

[0090] S906, perform vector search in the document, and jump to S907;

[0091] S907, select the search fragment with the smallest uncertainty, merge the searched information with the input content through the prompt word template, input it into the large language model, and jump to S909;

[0092] S908, inputting the input question into the large language model without introducing the search information;

[0093] S909, obtaining the model output result.

[0094] The present disclosure is used to perform knowledge enhancement on a large language model, and only uses the retrieved knowledge to enhance the model performance when necessary, and determines the retrieval method to be used according to the type of question, thereby improving the retrieval effect of the model, reducing the time required for reasoning, and shortening the length of the input content. The prior art introduces retrieval knowledge on all questions, and does not select a corresponding retrieval method for the type of question, which may lead to the introduction of unnecessary information and poor retrieval effect, resulting in a decrease in model performance and an increase in the time cost required for reasoning. The retrieval effect is effectively improved, while unnecessary retrieval operations are reduced when no retrieval is required, thereby reducing the time required for reasoning and shortening the length of the input content.

[0095] The present disclosure uses a classifier to classify input questions and combines it with self-perceived uncertainty assessment to decide whether to perform a retrieval operation, which avoids unnecessary retrieval operations. Existing methods do not fully consider the necessity of retrieval, resulting in unnecessary retrieval operations, wasting computing resources, and introducing potentially misleading information.

[0096] The present disclosure uses a classifier to classify the types of questions and decide which retrieval method to use, which improves the efficiency and accuracy of the retrieval. The existing method has a single retrieval method, which will lead to the use of an inappropriate retrieval method to retrieve information, affecting the quality of the retrieved information.

[0097] Based on the same inventive concept, the embodiment of the present disclosure also provides a retrieval enhancement generation device based on a large language model, as described in the following embodiment. Since the principle of solving the problem in the embodiment of the device is similar to that in the above method embodiment, the implementation of the embodiment of the device can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0098] Fig.10 A schematic diagram of a retrieval enhancement generation device based on a large language model in an embodiment of the present disclosure is shown. Fig.10 As shown, the device includes: a retrieval method determination module 1001, a knowledge fragment determination module 1002, an output result determination module 1003, and a judgment module 1004.

[0099] The search method determination module 1001 is used to determine the search method corresponding to the input question when the input question needs to be searched;

[0100] The knowledge fragment determination module 1002 is used to search the input question according to the search method and determine the knowledge fragment;

[0101] The output result determination module 1003 is used to input the knowledge fragment and the input question into the large language model to determine the output result.

[0102] In an example of the present disclosure, the above-mentioned retrieval enhancement generation device based on a large language model further includes a judgment module 1004, which is used to obtain an input question; and judge whether the input question needs to be retrieved.

[0103] In one example of the present disclosure, the above-mentioned judgment module 1004 is also used to judge whether the input question needs to be retrieved from multiple preset dimensions, where the multiple preset dimensions include: specificity, professionalism, timeliness, privacy, and common sense.

[0104] In an example of the present disclosure, the above-mentioned judgment module 1004 is also used to sum up the classification scores of multiple preset dimensions to determine the total score; and judge whether the input question needs to be retrieved based on whether the total score reaches a threshold.

[0105] In one example of the present disclosure, the judgment module 1004 is further configured to retrieve the input question when the uncertainty score of the self-awareness exceeds an empirical threshold.

[0106] In an example of the present disclosure, the above-mentioned search method determination module 1001 is also used to classify the input question and determine the classification result; and determine the search method corresponding to the input question according to the classification result.

[0107] In one example of the present disclosure, the classification results in the above-mentioned retrieval enhancement generation device based on a large language model include: numerical questions, statistical questions, and open questions; the above-mentioned knowledge fragment determination module 1002 is used to: when the classification result is a numerical question, search the corresponding numerical value in the target table to determine the knowledge fragment; when the classification result is a statistical question, query and determine the knowledge fragment in the target database; when the classification result is an open question, perform vector retrieval in the target document to determine the knowledge fragment.

[0108] In an example of the present disclosure, the output result determination module 1003 is further used to splice the knowledge fragment and the input question using the prompt word template to determine splicing information; and input the splicing information into the large language model to determine the output result.

[0109] It should be noted that the above-mentioned retrieval method determination module 1001, knowledge fragment determination module 1002, and output result determination module 1003 correspond to S202 to S204 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0110] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0111] Refer to the following Fig.11 1100 according to this embodiment of the present disclosure is described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0112] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110).

[0113] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0114] For example, the processing unit 1110 can execute the following steps of the above method embodiment: when an input question needs to be retrieved, determine the retrieval method corresponding to the input question; search the input question according to the retrieval method to determine the knowledge fragment; input the knowledge fragment and the input question into the large language model to determine the output result.

[0115] For example, the processing unit 1110 may execute the following steps of the above method embodiment: obtaining an input question; and determining whether the input question needs to be retrieved.

[0116] For example, the processing unit 1110 can execute the following steps of the above method embodiment: determine whether the input question needs to be retrieved from multiple preset dimensions, where the multiple preset dimensions include: specificity, professionalism, timeliness, privacy, and common sense.

[0117] For example, the processing unit 1110 may execute the following steps of the above method embodiment: summing the classification scores of multiple preset dimensions to determine the total score; and determining whether the input question needs to be retrieved based on whether the total score reaches a threshold.

[0118] For example, the processing unit 1110 may execute the following steps of the above method embodiment: when the uncertainty score of self-awareness exceeds an empirical threshold, the input question is retrieved.

[0119] For example, the processing unit 1110 may execute the following steps of the above method embodiment: classify the input question and determine the classification result; and determine the retrieval method corresponding to the input question according to the classification result.

[0120] For example, the processing unit 1110 can execute the following steps of the above method embodiment: when the classification result is a numerical question, searching the target table for the corresponding numerical value to determine the knowledge fragment; when the classification result is a statistical question, querying the target database to determine the knowledge fragment; when the classification result is an open question, performing vector retrieval in the target document to determine the knowledge fragment.

[0121] For example, the processing unit 1110 may execute the following steps of the above method embodiment: using the prompt word template to splice the knowledge fragment and the input question to determine splicing information; inputting the splicing information into the large language model to determine the output result.

[0122] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202 , and may further include a read-only storage unit (ROM) 11203 .

[0123] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0124] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0125] The electronic device 1100 may also communicate with one or more external devices 1140 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0126] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0127] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above-mentioned retrieval enhancement generation method based on a large language model when executed by a processor.

[0128] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of this specification.

[0129] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0131] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0132] In specific implementations, program codes for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0133] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0134] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0135] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0136] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.< / eos>

Claims

1. A retrieval enhancement generation method based on a large language model, characterized in that: include: When an input question needs to be searched, determining a search method corresponding to the input question; Search the input question according to the search method to determine the knowledge fragment; The knowledge fragment and the input question are input into a large language model to determine an output result.

2. The retrieval enhancement generation method based on a large language model according to claim 1, characterized in that: When the input question needs to be searched, before determining the search method corresponding to the input question, the method further includes: Get input question; Determine whether the input question needs to be searched.

3. The retrieval enhancement generation method based on a large language model according to claim 2, characterized in that: The determining whether the input question needs to be searched includes: Determine whether the input question needs to be searched based on multiple preset dimensions, wherein the multiple preset dimensions include: specificity, professionalism, timeliness, privacy, and common sense.

4. The retrieval enhancement generation method based on a large language model according to claim 3 is characterized in that: The method further comprises: Sum the classification scores of the multiple preset dimensions to determine a total score; Whether the input question needs to be searched is determined based on whether the total score reaches a threshold.

5. The retrieval enhancement generation method based on a large language model according to claim 4, characterized in that: The method further comprises: When the uncertainty score of the self-awareness exceeds an empirical threshold, the input question is retrieved.

6. The retrieval enhancement generation method based on a large language model according to any one of claims 1 to 5, characterized in that: When the input question needs to be searched, determining the search method corresponding to the input question includes: Classifying the input question and determining a classification result; According to the classification result, a retrieval method corresponding to the input question is determined.

7. The retrieval enhancement generation method based on a large language model according to claim 6, characterized in that: The classification results include: numerical questions, statistical questions, and open questions; searching the input questions according to the search method to determine the knowledge fragments includes: When the classification result is a numerical question, the corresponding numerical value is found in the target table to determine it as a knowledge fragment; When the classification result is a statistical problem, the target database is queried to determine the knowledge fragment; When the classification result is an open question, vector retrieval is performed in the target document to determine the knowledge fragment.

8. The retrieval enhancement generation method based on a large language model according to claim 1, characterized in that: The step of inputting the knowledge fragment and the input question into a large language model and determining an output result comprises: Using a prompt word template to splice the knowledge fragment and the input question, and determine splicing information; The concatenated information is input into a large language model to determine an output result.

9. A retrieval enhancement generation device based on a large language model, characterized in that: include: A search method determination module, used to determine the search method corresponding to the input question when the input question needs to be searched; A knowledge fragment determination module, used to search the input question according to the search method to determine the knowledge fragment; The output result determination module is used to input the knowledge fragment and the input question into the large language model to determine the output result.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the retrieval enhancement generation method based on a large language model as described in any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the retrieval enhancement generation method based on a large language model described in any one of claims 1 to 8 is implemented.

Images