Question answering system construction method based on large language model and question answering system
Through the construction method of Q&A system based on large language model, audit rules and modules are generated for data review, and a question and answer engine is built, which solves the problems of user input understanding and database data update in the port and shipping industry, and realizes intelligent auditing and efficient Q&A, reducing the number of questions and answers. labor costs and improved accuracy.
Patent Information
- Application Number
- CN202411455126.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-10-18
AI Technical Summary
The existing intelligent question-and-answer system is difficult to effectively understand the user input text and realize intelligent review and update of database data in the port and shipping industry, resulting in untimely and accurate knowledge updates.
Using a question-and-answer system construction method based on a large language model, by obtaining port and shipping data and industry knowledge, generating audit rules and audit modules, using multiple large language models for data review, and entering the database after the audit is passed, and building a question-and-answer engine to Improve user input comprehension ability.
It realizes intelligent audit of incoming data, improves audit accuracy and the accuracy of user Q&A, reduces manual audit costs, and solves the problems of lagging knowledge updates and low content accuracy in the existing technology.
Smart Images

Figure CN118964587A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent question answering, and in particular to a question answering system construction method and a question answering system based on a large language model. Background Art
[0002] With the rapid development of the port and shipping industry, the rapid update and timely acquisition of professional knowledge has become very important. And the intelligent question-answering system has gradually been applied in the port and shipping industry, improving the efficiency and accuracy of information acquisition through automation and intelligent means, so as to improve the efficiency of staff in obtaining port and shipping industry data.
[0003] In the existing technology, the data in the port and shipping industry involves a large number of professional terms and complex operating specifications. Traditional knowledge management systems often rely on manual maintenance, which leads to untimely knowledge updates, low review efficiency, and difficulty in ensuring content accuracy. In addition, most of the existing intelligent question-and-answer systems are rule-based question-and-answer systems and simple database query systems. The rule-based question-and-answer system conducts questions and answers through predefined rules and templates. Its rule updates are complex and inflexible, making it difficult to cope with complex and changeable professional knowledge, and its ability to understand user input text is poor. The database query system is based on a fixed database for query. During use, it is unable to intelligently review and store new data, making it difficult to ensure content accuracy and timeliness.
[0004] At present, no effective solution has been proposed to address the problems that existing question-and-answer systems have poor understanding of user input text and are unable to perform intelligent review and storage of incoming data. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a question-answering system construction method and a question-answering system based on a large language model, so as to at least solve the problem that the existing question-answering system has poor understanding ability of user input text and the database data cannot be intelligently reviewed and updated.
[0006] The embodiment of the present invention provides the following technical solutions: The embodiment of the present invention provides a method for constructing a question-answering system based on a large language model, comprising: Acquire port and shipping data, port and shipping industry knowledge and industry standards, and generate storage data after pre-processing the port and shipping data; Based on the port and shipping industry knowledge and the port and shipping industry standards, a large language model is used to generate audit rules, and the audit rules work in a way of search enhancement generation combined with prompt words; Based on the audit rules, the large language model is used to generate an audit module, the audit module includes an audit knowledge text and preset audit prompt words, and the audit module can perform retrieval enhancement generation on the incoming data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the audit knowledge text to obtain audit retrieval information corresponding to the incoming data; Based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the incoming data, and when multiple audit large language models have passed the audit, the incoming data is entered into the port and shipping database; A question-answering engine is constructed using the generated large language model, and user input is understood based on the question-answering engine and the port and shipping database, and corresponding user output is output.
[0007] Furthermore, based on the audit module and the audit retrieval information, the plurality of audit large language models are used to audit the incoming data. If at least one of the audit large language models fails the audit, the incoming data is transferred to manual audit.
[0008] Furthermore, the question-answering system construction method further includes: Based on the audit rules, an error recognition and completion module is generated using the generated large language model, wherein the error recognition and completion module includes an error recognition and completion text and a preset completion prompt word, and the error recognition and completion module can perform search enhancement generation on the incoming data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the error recognition and completion text to obtain completion search information corresponding to the incoming data; Based on the error identification and completion module and the completion retrieval information, the completion large language model is used to identify and correct erroneous information in the incoming data and / or complete the accurate information in the incoming data, and the incoming data after identification, correction and / or completion is entered into the port and shipping database.
[0009] Furthermore, a few sample prompts are used to add prompt word examples to the preset review prompt words and / or preset completion prompt words.
[0010] Furthermore, the use of the generated large language model to construct a question-answering engine, and understanding user input based on the question-answering engine and the port and shipping database, and outputting corresponding user output includes: Using the generated large language model to construct the question-answering engine, the question-answering engine includes question-answering knowledge text and question-answering prompt words; Based on the question-answering engine, the question-answering language model is used to obtain and understand the user input, and the question-answering language model generates and associates the port and shipping database through retrieval enhancement to obtain input retrieval information corresponding to the user input; Based on the user input and the input retrieval information corresponding to the user input, the question-answering large language model outputs a user output corresponding to the user input.
[0011] Furthermore, after using the generated large language model to build a question-answering engine, understanding user input based on the question-answering engine and the port and shipping database, and outputting corresponding user output, it also includes: Obtaining user negative feedback and interaction data corresponding to the user negative feedback; Manually review the user's negative feedback, the interaction data and the corresponding storage data, and re-enter the corresponding storage data into the port and shipping database after the review is passed.
[0012] Furthermore, after using the generated large language model to build a question-answering engine, understanding user input based on the question-answering engine and the port and shipping database, and outputting corresponding user output, it also includes: Detecting and identifying update data, where the update data is data that needs to be updated into the port and shipping database; Based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the updated data, and when multiple audit large language models have passed the audit, the updated data is entered into the port and shipping database; or Based on the error identification and completion module and the completion search information, the completion large language model is used to identify and correct erroneous information in the update data and / or to complete the accurate information in the update data, and the updated data after identification, correction and / or completion is entered into the port and shipping database.
[0013] Furthermore, after the updated data is entered into the port and shipping database, the method further includes: Record the time and content of the updated data entered into the port and shipping database.
[0014] Furthermore, the obtaining of port and shipping data and generating storage data after preprocessing the port and shipping data includes: Automatically collect the port and shipping data through multiple channels, wherein the multiple channels include one or more of port and shipping professional literature, databases, websites, and company internal documents; De-duplication, completion and formatting of the port and shipping data to ensure the integrity and consistency of the port and shipping data; According to the fields and themes of the port and shipping data, the port and shipping data are classified and sorted to generate the storage data.
[0015] The embodiment of the present invention further provides a question-answering system based on a large language model, comprising: A knowledge base construction module, which is used to obtain port and shipping data, port and shipping industry knowledge and industry standards, and to generate storage data after preprocessing the port and shipping data; A knowledge base audit module generates audit rules based on the port and shipping industry knowledge and the port and shipping industry standards using a generated large language model, wherein the audit rules work in a manner of search enhancement generation combined with prompt words; based on the audit rules, an audit module is generated using the generated large language model, wherein the audit module includes an audit knowledge text and preset audit prompt words, and the audit module can perform search enhancement generation on the incoming data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the audit knowledge text to obtain audit retrieval information corresponding to the incoming data; based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the incoming data, and when multiple audit large language models are audited and passed, the incoming data is entered into the port and shipping database; A question-answering engine construction module is used to construct a question-answering engine using the generated large language model, understand user input based on the question-answering engine and the port and shipping database, and output corresponding user output.
[0016] Compared with the prior art, the at least one technical solution adopted in the embodiment of the present invention can achieve the following beneficial effects: The invention discloses a method for constructing a question-answering system based on a large language model. The method generates audit rules based on port and shipping industry knowledge and port and shipping industry standards, and generates an audit module based on the audit rules using the generated large language model, so that the audit module can perform retrieval enhancement generation on the incoming data based on the port and shipping industry knowledge and port and shipping industry standards to obtain audit retrieval information, and when the incoming data and the audit retrieval information are input into the audit large language model, the audit large language model can understand the incoming data more accurately, and perform accurate audit on the incoming data under the guidance of preset audit prompt words, and finally enter the incoming data into the port and shipping database after multiple audit large language models have passed the audit, thereby realizing intelligent audit of the incoming data, solving the problem that the database data in the existing question-answering system cannot realize intelligent audit update and storage, and improving the accuracy of the audit of the incoming data; Furthermore, by using a large language model and question-answering prompt words to build a question-answering engine, and using retrieval enhancement to generate an associated port and shipping database, the question-answering engine's ability to understand user input can be improved. Since the question-answering engine generates an external port and shipping database based on retrieval enhancement, the correctness of the output answer is also improved, which solves the problem in the prior art that question-answering through predefined rules and templates makes it difficult to cope with complex and changeable professional knowledge and has poor understanding of user input text. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 A method for constructing a question-answering system based on a large language model according to an embodiment of the present invention; Figure 2 A specific process for constructing a question-answering system based on a large language model according to an embodiment of the present invention; Figure 3 The structure diagram of a question-answering system based on a large language model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0020] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.
[0021] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.
[0022] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0023] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.
[0024] With the rapid development of the port and shipping industry, the rapid updating and timely acquisition of professional knowledge has become particularly important. Intelligent question and answer systems have gradually been used in the port and shipping industry to improve the efficiency and accuracy of information acquisition through automation and intelligent means.
[0025] However, the existing technology still has many problems in the construction, review and update of the knowledge base, as well as the intelligence level of the question-answering system. For example, the knowledge base cannot be updated automatically and requires manual maintenance, review, and update of the knowledge base data. The question-answering system has poor understanding of user questions and finds it difficult to fully understand user questions, or cannot flexibly match relevant answers to users after understanding user questions. Therefore, the existing question-answering system has poor performance and usually requires manual problem solving.
[0026] Because the port and shipping industry involves a large number of professional terms and complex operating specifications, traditional knowledge management systems often rely on manual maintenance, and manual review of stored data leads to untimely knowledge updates, low review efficiency, and difficulty in ensuring content accuracy (if it is not reviewed by experts in the port and shipping industry, there will be problems such as content matching errors or content update errors). Since the knowledge in the port and shipping industry is complex, the knowledge is updated quickly, and it is difficult to understand, the intelligent question-answering system is also required to have efficient knowledge update capabilities, understanding capabilities, and review capabilities to meet the timely and accurate acquisition of professional knowledge in the port and shipping industry.
[0027] Since the knowledge in the port and shipping field is not well known to the general public, and the knowledge is complex, professional and diverse, and the current question-answering systems in the port and shipping field are mostly built based on predefined rules and limited databases, their ability to understand user questions is poor. Because the data in their databases is limited, the existing question-answering systems are also difficult to provide users with correct answers.
[0028] Specifically, existing question-answering systems have the following problems: (1) High manual review costs: The review and update of existing databases in the port and shipping fields require a lot of manpower, which is costly and inefficient. (2) Low content accuracy: Due to the subjectivity and tediousness of manual review, the data stored is prone to errors or omissions, which results in the subsequent question-and-answer system being unable to correctly answer user questions; (3) Update lag: Database updates rely on manual operations, making it difficult to reflect the latest industry trends and knowledge in a timely manner; (4) Lack of automated supplementation: It is difficult to automatically supplement and update the existing database, resulting in an incomplete knowledge base.
[0029] Therefore, the above problems limit the application effect of intelligent question-answering system in the port and shipping industry to a certain extent, and it is difficult to meet the needs of users for efficient and accurate information acquisition.
[0030] The following is an intelligent question-and-answer system in the prior art. The existing intelligent question-and-answer system generally includes a rule-based question-and-answer system and a simple database query system. The rule-based question-and-answer system generally performs questions and answers based on predefined rules and templates. Its rule updates are complex and inflexible, and it is difficult to cope with complex and changing professional knowledge; the database query system performs queries based on a fixed database, which cannot achieve intelligent knowledge update and review, and the accuracy and timeliness of the content are difficult to guarantee.
[0031] Based on this, this specification embodiment proposes a processing solution: Figure 1 As shown, the present invention provides a method for constructing a question-answering system based on a large language model, which automatically obtains port and shipping data, generates incoming data after preprocessing, and then generates audit rules applicable to the port and shipping field based on port and shipping industry knowledge and port and shipping industry standards, so that the subsequently generated audit module can perform retrieval enhancement generation on the incoming data to obtain audit retrieval information, so that when the audit module is used to audit the large language model, the audit large language model can obtain the incoming data and the audit retrieval information corresponding to the incoming data to accurately understand the incoming data, and finally the audit large language model accurately audits the incoming data under the guidance of preset audit prompt words and enters the data into the database after the audit is passed, which not only improves the accuracy of the audit of the incoming data, but also solves the problem that the database data in the existing question and answer system cannot be intelligently audited and updated into the warehouse.
[0032] Furthermore, the present invention constructs a question-and-answer engine based on the generation of a large language model, and generates an associated port and shipping database based on retrieval enhancement. It can also enable the question-and-answer large language model to accurately understand user input based on retrieval enhancement generation, and accurately output corresponding outputs, thereby improving the accuracy of user questions and answers and reducing the cost of manual review.
[0033] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.
[0034] Example 1 The embodiment of the present invention provides a method for constructing a question-answering system based on a large language model. Figure 1 As shown, the method includes: Step S102: Acquire port and shipping data, port and shipping industry knowledge and port and shipping industry standards, and generate storage data after pre-processing the port and shipping data.
[0035] When acquiring port and shipping data, a data acquisition unit may be used to automatically acquire the port and shipping data, such as using a Python crawler to automatically crawl the port and shipping data.
[0036] Among them, crawling tools can also be used to automatically obtain port and shipping industry knowledge and port and shipping industry standards.
[0037] Specifically, step S102 includes: Step S102a, automatically collecting port and shipping data through multiple channels, where the multiple channels include one or more of port and shipping professional literature, databases, websites, and company internal documents; Step S102b, de-duplication, completion and formatting of the port and shipping data to ensure the integrity and consistency of the port and shipping data; Step S102c: classify and sort the port and shipping data according to the fields and themes of the port and shipping data to generate storage data.
[0038] Among them, port and shipping data can be automatically collected through multiple channels through the data collection unit.
[0039] Among them, the internal company documents are knowledge documents published by internal employees of the company, which can be documents reviewed by experts or unreviewed documents.
[0040] By preprocessing the collected data through steps S102b and S102c, port and shipping data with relatively high integrity and consistency can be obtained. Finally, the port and shipping data are classified and sorted according to the fields and themes of the port and shipping data to generate storage data, thereby facilitating subsequent review of the storage data.
[0041] Step S102 realizes the automatic collection and crawling of the port and shipping data in the port and shipping field, and after a series of processing on the port and shipping data, the data is formed into the content format that can be entered into the port and shipping database, and it is also convenient for the subsequent intelligent review and storage of the data.
[0042] Step S104: Based on the knowledge and standards of the port and shipping industry, a large language model is used to generate audit rules, and the audit rules work in a manner of search enhancement generation combined with prompt words.
[0043] Among them, the audit rules are derived from the industry knowledge characteristics manually reviewed by experts (such as the International Maritime Organization standards, the TOS system information of a certain terminal, technical specifications, operating procedures, etc.) and the port and shipping industry standards to ensure the professionalism and accuracy of the rules, so that when the search enhancement generation searches the input data, it can also ensure that the retrieved audit search information is strongly related to the port and shipping industry and complies with the port and shipping industry standards and characteristics.
[0044] Since the amount of data on existing professional knowledge in the port and shipping field is insufficient to reach the level of model fine-tuning, the audit rules are generated by retrieval enhancement combined with audit prompt words, so that the subsequently generated audit module can still complete the audit task well and maintain a certain accuracy and relevance even when the data in the port and shipping field is limited.
[0045] Specifically, retrieval enhancement generation mainly includes three processes: retrieval, enhancement, and generation: retrieval is used to obtain relevant information from an external knowledge base based on the user's query content. Specifically, the user's query is converted into a vector through an embedding model so that it can be compared with the relevant knowledge stored in the vector database. Through similarity search, the top K data that best matches the query are found. Enhancement is to embed the user's query content and the retrieved relevant knowledge into a preset prompt word template; generation is to input the prompt word content that has been enhanced by retrieval into a large language model to generate the required output. Therefore, after obtaining the user input, retrieval enhancement generation can search in its associated database, and input the retrieval and the retrieved content into the large language model, so that the information obtained by the large language model can understand the user input more accurately.
[0046] Among them, prompt words are used to guide the generation of audit rules for the large language model to help the large language model to clarify the direction of generating content according to the port and shipping industry knowledge and port and shipping industry standards, so as to ensure that the generated audit rules can effectively cover the industry standards and port and shipping industry knowledge in the port and shipping field, so as to adjust and optimize the generation process.
[0047] Step S104 works by combining search enhancement generation with prompt words, so that when the amount of data in the port and shipping field is not enough to reach the level of model fine-tuning, it is still possible to generate audit rules well and maintain a certain accuracy and relevance, thereby overcoming the problem of insufficient data that may be caused by relying solely on the intrinsic knowledge of the model, and further improving the quality and practicality of audit rule generation. In addition, by searching within the port and shipping industry knowledge and port and shipping industry standards through search enhancement generation, the retrieved information can also be made more in line with the port and shipping field.
[0048] Step S106: Based on the audit rules, a large language model is used to generate an audit module, wherein the audit module includes an audit knowledge text and preset audit prompt words. The audit module can perform retrieval enhancement generation on the incoming data based at least on the port and shipping industry knowledge, port and shipping industry standards and the audit knowledge text to obtain audit retrieval information corresponding to the incoming data.
[0049] Among them, the audit module is used to audit the incoming data when it is applied to the audit large language model.
[0050] Among them, the audit knowledge text is a knowledge document related to the port and shipping field that has been manually reviewed by experts. It can also be a knowledge document of a specific sub-industry within the port and shipping field. The audit knowledge text can also be used to provide part of the data source for the audit large language model when the audit module is applied to the audit large language model, or to provide the audit module with the contextual information required for the actual audit.
[0051] Among them, the preset review prompt words are used to guide the review large language model to review the incoming data when the review module is applied to the review large language model; the preset review prompt words are manually set, and they can be flexibly set according to needs, as long as their purpose is to guide the review large language model to review the incoming data.
[0052] The following is an example of an audit module generated by generating a large language model based on preset audit prompt words and combined with audit rules: “ <context> [context] < / context> You are now an expert assistant in the field of ports and shipping. You need to complete the audit task with the help of the given context information.
[0053] Your task is to review the correctness of the content entered by the user.
[0054] If you are not sure whether what you entered is correct, just reply "I'm not sure".
[0055] If you are sure that what you entered is correct, you only need to reply "OK".
[0056] If you are sure that there is wrong information in the input, you only need to reply "There is an error".
[0057] ” Among them, the text part in the tree example is the preset review prompt word, and the context information is the review knowledge text, which is a knowledge document reviewed by experts.
[0058] Specifically, after the audit module obtains the incoming data, since the audit rules are based on the port and shipping industry knowledge and the port and shipping industry standards and work in the form of retrieval enhancement generation combined with prompt words, the audit module can perform retrieval enhancement on the incoming data within the port and shipping industry knowledge and the port and shipping industry standards through retrieval enhancement generation, and generate audit retrieval information. Then the audit module sends the incoming data and the audit retrieval information to the corresponding audit large language model, so that the audit large language model can accurately understand the incoming data in the port and shipping field, and can generate outputs corresponding to the incoming data according to the associated port and shipping industry knowledge and port and shipping industry standards based on the retrieval enhancement generation, or generate explanations corresponding to the incoming data, or accurately audit the incoming data under the action of preset audit prompt words.
[0059] Among them, retrieval enhancement generation includes input enhancement, retrieval enhancement, generator enhancement, result enhancement and RAG process enhancement, etc. Therefore, after the retrieval enhancement generation works in conjunction with prompt words and associates with databases such as port and shipping industry knowledge, port and shipping industry standards, and audit knowledge texts, the audit module can obtain audit retrieval information that is associated with the input data and conforms to the port and shipping industry knowledge characteristics and port and shipping industry standards based on retrieval enhancement generation, so as to provide more accurate and reliable port and shipping field knowledge for the audit large language model, reduce the possibility of generating hallucinations, and by accessing the latest external knowledge base, it can also maintain the timeliness and accuracy of the output.
[0060] Step S106 generates an audit module using a large language model through audit rules. The audit module can perform retrieval enhancement on the incoming data to obtain audit retrieval information, and send the incoming data and the audit retrieval information after retrieval enhancement to the audit large language model, so that the audit large language model can understand the incoming data more accurately, avoiding the problem of hallucinations caused by the audit large language model after obtaining the incoming data.
[0061] Step S108: Based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the incoming data. When multiple audit large language models have passed the audit, the incoming data is entered into the port and shipping database.
[0062] Among them, when the audit module is integrated and connected to the audit big language model, the audit module sends the incoming data and the audit retrieval information corresponding to the incoming data to the audit big language model. The audit big language can understand the incoming data based on the audit retrieval information on the premise of complying with the port and shipping industry knowledge and port and shipping industry standards, and can, under the guidance of the preset audit prompt words in the audit module, combine the audit knowledge text, audit retrieval information and its own retrieval data to audit the incoming data.
[0063] Among them, after the audit large language model obtains the incoming data and audit retrieval information sent by the audit module, the audit large language model can understand the incoming data more accurately, so that the audit large language model can perform accurate queries in its own associated database, and then audit the incoming data under the guidance of the preset audit prompt words of the audit module.
[0064] Among them, the multiple audit large language models can be different large language models, and the large language model can be the same as any audit large language model.
[0065] For example, when multiple audit large language models include Qwen, LLama, gemma, Yi and other models, the generated large language model can be any one of the modules such as Qwen, LLama, gemma, Yi and so on.
[0066] Among them, the audit module can be integrated and designed into multiple audit large language models at the same time, and the incoming data can be audited simultaneously through multiple audit large language models. Then, after the multiple audit large language models have passed the audit at the same time, the incoming data can be entered into the database, thereby effectively solving the hallucination problem of the large model and improving the audit accuracy of the audit large language model.
[0067] Therefore, in step S108, the audit module uses a voting mechanism with multiple audit language models, and only the incoming data that passes unanimously can directly enter the port and shipping database.
[0068] Specifically, the audit module is used through multiple audit large language models to simultaneously audit the incoming data, and after the multiple audit large language models have passed the audit, the incoming data is directly entered into the database, thereby realizing the intelligent entry of incoming data. Moreover, because the audit module and the audit large language model both have good comprehension and logical capabilities, and the port and shipping industry knowledge and port and shipping industry standards are associated through retrieval enhancement, the correctness of the incoming data can be guaranteed, which solves the problem of manual review of incoming data in the prior art and improves the efficiency of incoming data entry.
[0069] Furthermore, based on the audit module and the audit retrieval information, multiple audit large language models are used to perform data audit on the incoming data. If at least one audit large language model fails the audit, the incoming data will be transferred to manual audit, and after the manual audit passes, the incoming data will be added to the port and shipping database.
[0070] Among them, when the manual review fails, the stored data will be discarded.
[0071] By submitting the incoming data that has failed to pass the review of multiple large language models to manual or relevant experts for review, the accuracy of knowledge can be ensured, the hallucination problem of the large language model can be avoided, and the correctness of the answers output by the subsequent question-and-answer engine can be improved.
[0072] In step S108, the incoming data is audited simultaneously by using multiple audit large language models and combined with manual audit of the incoming data that differs in the multiple audit large language models, so as to ensure the correctness of the incoming data in the port and shipping database. In addition, by using multiple audit large language models to automatically audit the incoming data, intelligent updating and auditing of the data in the port and shipping database is also achieved.
[0073] In some embodiments, the question-answering system construction method further includes: Step S110: Based on the audit rules, an error recognition and completion module is generated using a large language model. The error recognition and completion module includes an error recognition and completion text and a preset completion prompt word. The error recognition and completion module can perform search enhancement generation on the incoming data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the error recognition and completion text to obtain the completion search information corresponding to the incoming data; Step S112: Based on the error recognition and completion module and the completion search information, the completion large language model is used to identify and correct the erroneous information in the incoming data and / or to complete the correct information in the incoming data, and the identified, corrected and / or completed incoming data is entered into the port and shipping database. Among them, the error recognition and completion module is used to perform retrieval enhancement and generation of incoming data based on port and shipping industry knowledge, port and shipping industry standards and error recognition and completion text to obtain completion and retrieval information, and then send the incoming data and the corresponding completion and retrieval information to the completion large language model. The completion large language model uses error recognition and completion text and completion and retrieval information to accurately understand the incoming data, and performs error recognition or completion on the incoming data under the guidance of preset completion prompt words.
[0074] The following is an error recognition and completion module generated by generating a large language model based on preset completion prompt words and combined with review rules: “ <context> [context] < / context> You are now an expert assistant in the field of port and shipping. You need to complete the task with the help of the given context information.
[0075] First you need to identify and correct errors in user input. Secondly, you need to complete the actual information in the input content.
[0076] If the user input content is modified, you need to output the modified content; If the user input content has not been modified, you only need to output the user content.
[0077] And answer in Chinese.
[0078] ” The text part is the preset completion prompt word, and the context information is the error recognition completion text.
[0079] In step S112 , completing the large language model may be the same as or different from generating the large language model or reviewing the large language model.
[0080] Among them, since the error recognition and completion module is generated based on the audit rules, and the audit rules work based on the search enhancement generation and prompt words, the completed search information obtained by the error recognition and completion module after the search enhancement of the incoming data is strongly correlated with the incoming data and conforms to the port and shipping industry knowledge and port and shipping industry standards. Therefore, after the completion of the large language model obtains the completed search information and the incoming data, it can perform error recognition and correction on the incoming data or complete the actual information in the incoming data, and enter the incoming data after error recognition or completion into the port and shipping database.
[0081] In some of the embodiments, a few sample prompts can be used to add prompt word examples to the preset review prompt words and / or the preset completion prompt words, so that the completion large language model or the review large language model can greatly improve its understanding of its own role positioning, so as to effectively reduce the hallucination problem of the large language model and improve the accuracy of review, error recognition correction, completion and question and answer.
[0082] Through steps S110 to S112, intelligent error recognition and completion of the incoming data are achieved, further improving the accuracy of the incoming data entered into the port and shipping database.
[0083] Step S114: Use the generated large language model to build a question-answering engine, understand the user input based on the question-answering engine and the port and shipping database, and output the corresponding user output.
[0084] Among them, after building the port and shipping database, the generated large language model can be used to build a question-answering engine to conduct questions and answers with users.
[0085] Specifically, after the question-answering engine obtains and understands the user input, it can match the corresponding user output for the user input based on the port and shipping database. And because the question-answering engine is built based on a large language model, it has good text understanding and generation capabilities, and can thus provide users with user outputs with a higher accuracy rate.
[0086] In some of the embodiments, when the question-and-answer engine cannot fully understand the user input, the question-and-answer engine can continue to question the user, so that the user continues to input the user question until the question-and-answer engine fully understands the user question.
[0087] Specifically, step S114 includes: Step S114a: Use the generated large language model to build a question-answering engine, where the question-answering engine includes question-answering knowledge text and question-answering prompt words.
[0088] The question-answering knowledge text includes at least all the data stored in the port and shipping database; the question-answering prompt words are used to guide the question-answering large language model to answer the user input when the question-answering engine is used for the question-answering large language model.
[0089] The following is an example of a question-answering engine generated by a large language model: “ <context> [context] < / context> You are now an expert assistant in the field of ports and shipping. You need to answer the user's input with the help of the given context information.
[0090] If you don't know the answer, just say you don't know.
[0091] If you are unsure of the answer, ask the user for more details.
[0092] And answer according to the language of the user's question.
[0093] ” The text part is the question and answer prompt words, and the context information is the question and answer knowledge text, which at least includes all the knowledge documents in the knowledge base.
[0094] Step S114b: Based on the question-answering engine, the question-answering language model is used to obtain and understand the user input, and the question-answering language model generates a related port and shipping database through retrieval enhancement to obtain input retrieval information corresponding to the user input; Among them, the question-answering large language model generates an associated port and shipping database through retrieval enhancement. Its retrieval enhancement generation can search the user input based on the port and shipping database to obtain input retrieval information, and then the retrieval enhancement generation sends the input retrieval information to the question-answering large language model, so that the question-answering large language model combines the input retrieval information and user input to accurately understand the user input.
[0095] Step S114c: Based on the user input and the input retrieval information corresponding to the user input, the question-answering language model outputs the user output corresponding to the user input.
[0096] After the question-answering large language model obtains the user input and input retrieval information to accurately understand the user input, the question-answering large language model can obtain the user output based on the question-answering knowledge text and its own retrieval, and output the user output under the action of the question-answering prompt words.
[0097] Steps S114a~S114c first use retrieval enhancement generation to enhance the user input to obtain input retrieval information, so that the question-answering large language model can accurately understand the user input based on the input retrieval information, thereby improving the question-answering large language model's ability to understand user input and also improving the question-answering engine's answer accuracy.
[0098] In summary, the question-answering engine built based on the large language model in step S110 supports questions and answers in natural language, and with the help of the powerful semantic understanding ability of the large language model, it can accurately understand the user's input content. At the same time, it uses retrieval enhancement generation technology to associate with the private port and shipping database, and can also provide users with professional and accurate answers.
[0099] Furthermore, the question-answering system construction method also includes: Step S116: Obtain user negative feedback and interaction data corresponding to the user negative feedback; Step S118: manually review the user's negative feedback, interaction data and corresponding storage data, and re-enter the corresponding storage data into the port and shipping database after the review is passed.
[0100] Among them, user negative feedback refers to the user's negative evaluation of the answer output by the question and answer engine, and the interactive data refers to the question and answer data of the question and answer engine, which includes user question data and question and answer engine answer data.
[0101] Among them, users can give positive or negative feedback on the answers output by the question-and-answer engine. Questions that users frequently feedback indicate that they are high-frequency triggering areas of hallucinations in the question-and-answer engine. After review and confirmation by human experts, they can be put into the database for correction to improve the accuracy of subsequent questions and answers.
[0102] Furthermore, the question-answering system construction method further includes: Step S202, detecting and identifying update data, where the update data is data that needs to update the port entry database; Step S204: Based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the updated data. When multiple audit large language models are audited and passed, the updated data is entered into the port and shipping database; or Step S206: Based on the error recognition and completion module and the completed search information, the completed large language model is used to identify and correct the erroneous information in the updated data generated after the search enhancement and / or to complete the correct information in the updated data, and the updated data after identification, correction and / or completion is entered into the port and shipping database.
[0103] Among them, after building the port and shipping database and question-answering engine, it is also necessary to detect the latest developments, information and internal company knowledge documents in the port and shipping field to identify the knowledge content that needs to be updated.
[0104] In step S202, a data acquisition unit may be used to detect and identify update data that needs to be updated in the port and shipping field, such as a port and shipping database.
[0105] Step S204 and step S206 are used to review, identify errors or complete the updated data to ensure the correctness of the updated data.
[0106] Furthermore, after the updated data is entered into the port and shipping database, the method further includes: Step S208, recording the time and content of the updated data being entered into the port and shipping database.
[0107] Step S208 records the time and content of the updated data entered into the port and shipping database, thereby facilitating the management of different versions of the port and shipping database and recording the content and time of each update for easy tracing and management.
[0108] like Figure 3 As shown, an embodiment of the present invention further provides a question-answering system based on a large language model, including a knowledge base construction module 10, a knowledge base review module 20 and a question-answering engine construction module 30. Among them, the knowledge base construction module 10 is used to obtain port and shipping data, port and shipping industry knowledge and industry standards, and generate storage data after pre-processing the port and shipping data; the knowledge base audit module 20 is used to generate audit rules based on the port and shipping industry knowledge and port and shipping industry standards using the generated large language model, and the audit rules work in the form of retrieval enhancement generation combined with prompt words; based on the audit rules, the generated large language model is used to generate an audit module, the audit module includes an audit knowledge text and preset audit prompt words, and the audit module can at least perform retrieval enhancement generation on the storage data based on the port and shipping industry knowledge, port and shipping industry standards and the audit knowledge text to obtain audit retrieval information corresponding to the storage data; and based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the storage data. When multiple audit large language models are audited and passed, the storage data is entered into the port and shipping database; the question and answer engine construction module 30 is used to use the generated large language model to build a question and answer engine, and understand user input based on the question and answer engine and the port and shipping database, and output corresponding user output.
[0109] The knowledge base construction module 10 further includes a data collection unit, a data cleaning unit and a knowledge classification unit. The data collection unit is used to automatically collect port and shipping professional knowledge data from relevant literature, websites and internal knowledge platforms in the field of port and shipping; the data cleaning unit is used to remove duplicates and format the collected data to ensure the availability and consistency of the data; the knowledge classification unit is used to classify and organize the data according to the fields and themes of port and shipping professional knowledge for subsequent query and management.
[0110] The knowledge audit module is used to intelligently audit and update the content in the knowledge base to ensure the accuracy and timeliness of the knowledge, and includes an audit rule unit and an automatic audit unit. Among them, the audit rule generation unit is used to generate audit rules based on the port and shipping industry knowledge and port and shipping industry standards using a generated large language model, and the audit rules work in the form of retrieval enhancement generation combined with prompt words; the automatic audit unit is used to generate an audit module based on the audit rules using the generated large language model, and the audit module includes an audit knowledge text and preset audit prompt words, and the audit module can perform retrieval enhancement generation on the incoming data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the audit knowledge text to obtain audit retrieval information corresponding to the incoming data; based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the incoming data, and when multiple audit large language models are audited and passed, the incoming data is entered into the port and shipping database.
[0111] The automatic review unit is also used to use a large language model to automatically review the incoming data, identify and correct erroneous information, update outdated content, and supplement missing contextual information in the incoming data.
[0112] Among them, the automatic review unit is also used to adopt multiple language models for review and use the final voting mechanism to decide whether manual review is required to solve the hallucination problem caused by a single large language model.
[0113] The question-answering system building module is used to build an intelligent question-answering engine to achieve efficient interaction with users. The question-answering system building module includes a question-answering engine unit, a knowledge matching unit, and a learning optimization unit. Among them, the question-answering engine unit is used for the question-answering engine built based on LLM, which supports natural language questions and answers; the knowledge matching unit is used to match the user's questions with the content in the knowledge base and provide the most relevant and accurate answers; the learning optimization unit continuously optimizes the performance of the question-answering engine and the accuracy of answers through user feedback and interaction data.
[0114] The question-answering system construction device also includes a knowledge update module, which is used to realize the timed update and automatic supplement of the knowledge base content, and includes a timed monitoring unit, a content generation unit, and a version control unit. Among them, the timed detection unit is used to monitor the latest developments and information in the field of ports and shipping, and identify the knowledge content that needs to be updated; the content generation unit is used to generate new knowledge items using a large language model and automatically supplement them to the knowledge base; the version control unit is used to manage different versions of the knowledge base, record the content and time of each update, so as to facilitate traceability and management.
[0115] Through the collaborative work of the above-mentioned units, the embodiment of the present invention constructs an efficient, accurate and intelligent port and shipping professional knowledge question and answer system, which greatly improves the human-computer interaction experience, solves many problems in the existing technology, and meets the port and shipping industry's needs for efficient acquisition and updating of professional knowledge.
[0116] Compared with the prior art, the method and device for constructing a question-answering system based on a large language model in the embodiment of the present invention have the following beneficial effects: 1. Reduce manual review costs By introducing the intelligent audit module, the present invention realizes the automatic audit and update of the knowledge base content, greatly reducing the workload and cost of manual audit. By utilizing the natural language processing capabilities of the large language model, it is possible to efficiently identify and correct erroneous information in the knowledge base, reducing the reliance on manual audit.
[0117] 2. Improve audit efficiency The automatic audit module of the present invention can quickly and efficiently audit the knowledge base content to ensure the timeliness and accuracy of the knowledge. Compared with the traditional method that relies on manual auditing one by one, the audit module can complete the audit of a large amount of data in a short time, improving the overall audit efficiency.
[0118] 3. Enhance content accuracy By combining automatic audit and manual review, the knowledge audit module of the present invention ensures the high accuracy of the knowledge base content. The automatic audit module first uses multiple models to conduct a preliminary audit, locates complex or difficult problems based on the voting mechanism, and then is processed by the manual review unit to ensure the correctness and reliability of the content.
[0119] 4. Automatic replenishment and update The knowledge update module of the present invention can automatically monitor the latest developments in the field of ports and shipping, and use LLM to generate new knowledge items to achieve real-time update and automatic supplement of knowledge base content. This solves the problem of failure to automatically supplement knowledge when updating in the prior art, and ensures that the knowledge base always reflects the latest professional information.
[0120] 5. Improve user interaction experience By building an intelligent question-and-answer system (question-and-answer engine), the present invention significantly improves the user's interactive experience. The question-and-answer engine based on a large language model can understand the natural language questions raised by users and provide accurate and professional answers. The knowledge matching unit ensures the relevance and accuracy of the answers, and the learning optimization unit continuously improves system performance through user feedback, further enhancing the user experience.
[0121] 6. Reduce maintenance costs Through automated knowledge review and update, the present invention reduces the frequency and cost of manual maintenance. The system can automatically monitor and update the knowledge base content, reducing manual intervention and improving the maintenance efficiency and effect of the system.
[0122] In summary, the present invention provides an intelligent, automated, efficient and accurate port and shipping professional knowledge question and answer system, which significantly improves the performance of the existing technology, solves the problems of high manual review cost, low content accuracy, lagging knowledge update and lack of automatic supplement, and has broad application prospects and significant technical advantages in the intelligent question and answer application of the port and shipping industry.
[0123] Example 2 This embodiment is a specific implementation of embodiment 1. Figure 2 As shown, the specific steps are as follows: Step S401, collecting professional knowledge; Step S402: review / supplement knowledge content of multiple agents (large language model); Step S403: Manually review the content that fails the review; Step S404, building the professional knowledge base is completed; Step S405: construct an intelligent question-answering system based on RAG technology; Step S406, review the prompt words to build domain experts; Step S407: system deployment.
[0124] In this specification, the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the product embodiment described later, since it corresponds to the method, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.
[0125] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for constructing a question-answering system based on a large language model, characterized in that: include: Acquire port and shipping data, port and shipping industry knowledge and port and shipping industry standards, and generate storage data after pre-processing the port and shipping data; Based on the port and shipping industry knowledge and the port and shipping industry standards, a large language model is used to generate audit rules, and the audit rules work in a way of search enhancement generation combined with prompt words; Based on the audit rules, the large language model is used to generate an audit module, the audit module includes an audit knowledge text and preset audit prompt words, and the audit module can perform retrieval enhancement generation on the incoming data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the audit knowledge text to obtain audit retrieval information corresponding to the incoming data; Based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the incoming data, and when multiple audit large language models have passed the audit, the incoming data is entered into the port and shipping database; A question-answering engine is constructed using the generated large language model, and user input is understood based on the question-answering engine and the port and shipping database, and corresponding user output is output.
2. The question-answering system construction method according to claim 1, characterized in that: Based on the audit module and the audit retrieval information, the input data is audited using the multiple audit large language models. If at least one of the audit large language models fails the audit, the input data is transferred to manual audit.
3. The question-answering system construction method according to any one of claims 1 or 2, characterized in that: Also includes: Based on the audit rules, an error recognition and completion module is generated using the generated large language model, wherein the error recognition and completion module includes an error recognition and completion text and a preset completion prompt word, and the error recognition and completion module can perform search enhancement generation on the incoming data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the error recognition and completion text to obtain completion search information corresponding to the incoming data; Based on the error identification and completion module and the completion search information, a completion large language model is used to identify and correct erroneous information in the incoming data and / or to complete the accurate information in the incoming data, and the incoming data after identification, correction and / or completion is entered into the port and shipping database.
4. The question-answering system construction method according to claim 3, characterized in that: Use a few sample prompts to add prompt word examples to the preset review prompt words and / or preset completion prompt words.
5. The question-answering system construction method according to claim 3, characterized in that: The use of the generated large language model to construct a question-answering engine, and understanding user input based on the question-answering engine and the port and shipping database, and outputting corresponding user output includes: Using the generated large language model to construct the question-answering engine, the question-answering engine includes question-answering knowledge text and question-answering prompt words; Based on the question-answering engine, a question-answering language model is used to obtain and understand user input, and the question-answering language model generates and associates the port and shipping database through retrieval enhancement to obtain input retrieval information corresponding to the user input; Based on the user input and the input retrieval information, the question-answering large language model outputs a user output corresponding to the user input.
6. The question-answering system construction method according to claim 5, characterized in that: After using the generated large language model to build a question-answering engine, understanding user input based on the question-answering engine and the port and shipping database, and outputting corresponding user output, the method further includes: Obtaining user negative feedback and interaction data corresponding to the user negative feedback; Manually review the user's negative feedback, the interaction data and the corresponding storage data, and re-enter the corresponding storage data into the port and shipping database after the review is passed.
7. The method for constructing a question-answering system according to claim 5, characterized in that: After using the generated large language model to build a question-answering engine, understanding user input based on the question-answering engine and the port and shipping database, and outputting corresponding user output, the method further includes: Detecting and identifying update data, where the update data is data that needs to be updated into the port and shipping database; Based on the audit module and the audit retrieval information, the updated data is audited using a plurality of the audit large language models, and when the plurality of the audit large language models are audited and passed, the updated data is entered into the port and shipping database; or Based on the error identification and completion module and the completion retrieval information, the completion large language model is used to identify and correct erroneous information in the update data and / or complete the accurate information in the update data, and the updated data after identification, correction and / or completion is entered into the port and shipping database.
8. The question-answering system construction method according to claim 7, characterized in that: After entering the updated data into the port and shipping database, the method further includes: Record the time and content of the updated data entered into the port and shipping database.
9. The question-answering system construction method according to claim 1, characterized in that: The obtaining of port and shipping data and generating storage data after preprocessing the port and shipping data includes: Automatically collect the port and shipping data through multiple channels, wherein the multiple channels include one or more of port and shipping professional literature, databases, websites, and company internal documents; De-duplication, completion and formatting of the port and shipping data to ensure the integrity and consistency of the port and shipping data; According to the fields and themes of the port and shipping data, the port and shipping data are classified and sorted to generate the storage data.
10. A question-answering system based on a large language model, characterized in that: include: A knowledge base construction module, which is used to obtain port and shipping data, port and shipping industry knowledge and industry standards, and to generate storage data after preprocessing the port and shipping data; A knowledge base audit module generates audit rules based on the port and shipping industry knowledge and the port and shipping industry standards using a generated large language model, wherein the audit rules work in a manner of search enhancement generation combined with prompt words; based on the audit rules, an audit module is generated using the generated large language model, wherein the audit module includes an audit knowledge text and preset audit prompt words, and the audit module can perform search enhancement generation on the input data based on at least the port and shipping industry knowledge, the port and shipping industry standards and the audit knowledge text to obtain audit retrieval information corresponding to the input data; Based on the audit module and the audit retrieval information, multiple audit large language models are used to audit the incoming data, and when multiple audit large language models have passed the audit, the incoming data is entered into the port and shipping database; A question-answering engine construction module is used to construct a question-answering engine using the generated large language model, understand user input based on the question-answering engine and the port and shipping database, and output corresponding user output.
Citation Information
Patent Citations
A name list auditing system and an auditing method thereof
CN109685463A
Real estate data exchange system based on ownership survey data
CN116266170A
Knowledge question-answering method and system based on large language model
CN117708282A
Insurance claim settlement material auditing method and device based on large language model
CN118350773A
Collection strategies that facilitate arranging portions of documents into content collections
US20180143975A1
Cited By
Automatic component representation method and device based on fine tuning and knowledge graph
CN119443045A
Output result generation method and system, electronic equipment and storage medium
CN120277131A
Design basis text intelligent auditing system and use method thereof
CN120374041A
Operation ticket auditing rule self-learning method, operation ticket auditing method, operation ticket auditing system and related equipment
CN120494469A
Artificial intelligence vertical large model training method focusing on petroleum coke industry
CN120744027A
