Model training method, problem data processing method and related device

Abstract

The invention provides a model training method, a problem data processing device and a related device. The embodiment of the invention can be applied to the field of artificial intelligence. The model training method comprises the steps of obtaining an original question and answer pair, wherein the original question and answer pair comprises an original question and an original answer; retrieving a first result of the original question based on the industry knowledge base; retrieving a second result of the original question based on the search engine; combining the first result, the second result and the target text of the original answer to obtain a reference answer; inputting the original question and the reference answer into a large language model to obtain a predicted answer output by the large language model; a big language model is trained according to the predicted answer and the original answer serving as the label, the trained big language model is used for obtaining a target answer of the target question according to the target reference answer, and the target reference answer is obtained through retrieval in an industry knowledge base and a search engine according to the target question. The trained large language model can improve the accuracy of answering the professional questions in the industry.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, a problem data processing method, and related devices. Background Art

[0002] A Large Language Model (LLM) is a deep learning model trained using large amounts of text data, designed to understand and generate human language. By training on large amounts of text data, LLM learns the ability to serve human language understanding and generation, and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models can handle a variety of natural language tasks, such as text classification, question-answering, and conversation. General large language models are generally pre-trained based on a wide range of public literature and online information. By learning language knowledge from large-scale corpora, they can handle text tasks in most scenarios. However, for industry problems with high technical expertise, if the capabilities of general large models are directly used to answer them, they often "seriously" generate incorrect or irrelevant responses (hallucination phenomenon), which leads users to make wrong judgments. Summary of the Invention

[0003] The embodiments of the present application provide a model training method, a question data processing method, and related devices for improving the accuracy of large language models in answering industry-specific questions.

[0004] The first aspect of the present application provides a model training method, comprising:

[0005] Obtaining an original question-answer pair, where the original question-answer pair includes an original question and an original answer corresponding to the original question;

[0006] Retrieving at least one first result of the original question based on the industry knowledge base;

[0007] Retrieving at least one second result for the original question based on the search engine;

[0008] Combine the first result, the second result, and the target text of the original answer to obtain a reference answer;

[0009] Input the original question and reference answer into the pre-trained large language model to obtain the predicted answer to the original question output by the large language model;

[0010] A large language model is trained based on the predicted answers and the original answers as labels. The trained large language model is used to obtain the target answer to the target question based on the target reference answer. The target reference answer is retrieved from the industry knowledge base and search engine based on the target question.

[0011] In a possible implementation method, retrieving at least one first result of the original question based on the industry knowledge base includes:

[0012] Input the industry knowledge base and the original question into the large language model to obtain the first result output by the large language model.

[0013] In a possible implementation method, obtaining the original question-answer pair includes:

[0014] Split the text in the industry knowledge base into multiple text segments based on preset lengths;

[0015] The text segment is input into the large language model to obtain the original question-answer pair corresponding to the text segment output by the large language model.

[0016] In a possible implementation method, retrieving at least one first result of the original question based on the industry knowledge base includes:

[0017] Based on the relevance of the original question and the text fragments, at least one first result is obtained, the first result belongs to multiple text fragments, and the relevance of the first result to the original question is higher than a first preset value.

[0018] In a possible implementation method, the original answer includes multiple answer segments, and the multiple answer segments are extracted from the multiple target segments in the order of the multiple answer segments; the target text is obtained by disrupting the order of the multiple target segments.

[0019] In a possible implementation method, after obtaining the original question-answer pair, the following steps are further included:

[0020] Expanding the original question to obtain a corresponding similar question, wherein the semantic matching degree between the similar question and the original question is higher than a second preset value;

[0021] Similar questions and reference answers are input into the large language model to obtain the predicted answer to the original question output by the large language model.

[0022] In a possible implementation method, retrieving at least one second result of the original question based on a search engine includes:

[0023] Enter the original question into the search engine and get multiple search results;

[0024] Duplicate results in the multiple search results are filtered to obtain at least one second result.

[0025] In one possible implementation method, if the number of second results is greater than K, where K is an integer greater than 1, after retrieving at least one second result for the original question based on a search engine, the method further includes:

[0026] The relevance between the original question and the second results is determined; the reference answer includes the K second results with the highest relevance.

[0027] A second aspect of the present application provides a method for processing problem data, comprising:

[0028] Get the target question;

[0029] Retrieving at least one first target result for the target question based on the industry knowledge base;

[0030] Retrieving at least one second target result for the target question based on the search engine;

[0031] Combine the first target result and the second target result to obtain a target reference answer;

[0032] The target question and the target reference answer are input into the pre-trained large language model to obtain the target answer to the target question output by the large language model, which is trained according to the model training method of the first aspect mentioned above.

[0033] In a possible implementation method, before retrieving at least one first target result for the target question based on the industry knowledge base, the method further includes:

[0034] Split the text in the industry knowledge base into multiple text segments based on preset lengths;

[0035] Retrieving at least one first target result for the target question based on the industry knowledge base, including:

[0036] Based on the relevance between the target question and the text fragments, at least one first target result is obtained, the first target result belongs to multiple text fragments, and the relevance between the first target result and the target question is higher than a first preset value.

[0037] In a possible implementation method, retrieving at least one second target result for the target question based on a search engine includes:

[0038] Enter the target question into the search engine and get multiple search results;

[0039] Duplicate results in the multiple search results are filtered to obtain at least one second target result.

[0040] In a possible implementation method, after retrieving at least one second target result for the target question based on a search engine, the method further includes:

[0041] If the number of second target results is greater than K, where K is an integer greater than 1, the correlation between the target question and the second target results is determined; the target reference answer includes the K second target results with the highest correlation.

[0042] A third aspect of the present application provides a model training device, comprising:

[0043] A first acquisition module is used to obtain an original question-answer pair, where the original question-answer pair includes an original question and an original answer corresponding to the original question;

[0044] A first retrieval module is configured to retrieve at least one first result of the original question based on the industry knowledge base; and is also configured to retrieve at least one second result of the original question based on a search engine;

[0045] A first combining module is used to combine the first result, the second result and the target text of the original answer to obtain a reference answer;

[0046] The first input module is used to input the original question and the reference answer into the pre-trained large language model to obtain the predicted answer to the original question output by the large language model;

[0047] The training module is used to train a large language model based on the predicted answers and the original answers as labels. The trained large language model is used to obtain the target answer to the target question based on the target reference answer. The target reference answer is retrieved from the industry knowledge base and search engine based on the target question.

[0048] In a possible implementation method, the first retrieval module is specifically configured to input the industry knowledge base and the original question into the large language model to obtain a first result output by the large language model.

[0049] In one possible implementation method, the first acquisition module is specifically used to segment the text in the industry knowledge base based on a preset length to obtain multiple text segments; input the text segments into the large language model to obtain the original question-answer pairs corresponding to the text segments output by the large language model.

[0050] In a possible case of a possible implementation method, the first retrieval module is specifically used to obtain at least one first result based on the relevance of the original question and the text fragment, the first result belongs to multiple text fragments, and the relevance of the first result to the original question is higher than a first preset value.

[0051] In a possible implementation method, the original answer includes multiple answer segments, and the multiple answer segments are extracted from the multiple target segments in the order of the multiple answer segments; the target text is obtained by disrupting the order of the multiple target segments.

[0052] In a possible implementation method, the first retrieval module is specifically configured to input an original question into a search engine to obtain multiple search results; and filter duplicate results in the multiple search results to obtain at least one second result.

[0053] In one possible implementation method, if the number of second results is greater than K, where K is an integer greater than 1, the first retrieval module is also used to determine the correlation between the original question and the second results; the reference answer includes the K second results with the highest correlation.

[0054] In a possible implementation method, the method further includes: a similar question expansion module, configured to expand the original question to obtain corresponding similar questions, wherein the semantic matching degree between the similar question and the original question is higher than a second preset value;

[0055] The training module is also used to input similar questions and reference answers into the large language model to obtain the predicted answer to the original question output by the large language model.

[0056] A fourth aspect of the present application provides a problem data processing device, comprising:

[0057] The second acquisition module is used to obtain the target question;

[0058] A second retrieval module is configured to retrieve at least one first target result for the target question based on the industry knowledge base; and is further configured to retrieve at least one second target result for the target question based on a search engine;

[0059] A second combining module is used to combine the first target result and the second target result to obtain a reference answer;

[0060] The second input module is used to input the target question and the target reference answer into the pre-trained large language model to obtain the target answer to the target question output by the large language model, and the large language model is trained according to the model training method of the first aspect mentioned above.

[0061] In one possible implementation method, the second retrieval module is specifically used to divide the text in the industry knowledge base based on a preset length to obtain multiple text fragments; based on the relevance of the target question and the text fragments, at least one first target result is obtained, the first target result belongs to multiple text fragments, and the relevance of the first target result to the target question is higher than a first preset value.

[0062] In one possible implementation, the second search module is specifically configured to input a target question into a search engine to obtain multiple search results; filter duplicate results from the multiple search results to obtain at least one second target result. In another possible implementation, if the number of second target results is greater than K, where K is an integer greater than 1, the second search module is further configured to determine the relevance between the target question and the second target results; in this case, the target reference answer includes the K second target results with the highest relevance.

[0063] A fifth aspect of the present application provides a computer device, including:

[0064] memories, transceivers, processors, and bus systems;

[0065] Wherein, the memory is used to store programs;

[0066] The processor is used to execute the program in the memory, including executing the above-mentioned methods;

[0067] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0068] In a sixth aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the methods in the above aspects.

[0069] In a seventh aspect, the present application provides a computer program product or computer program, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0070] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0071] The present application provides a model training method, a question data processing method and related devices, which first obtain an original question-answer pair, which includes an original question and a corresponding original answer, and the original answer is a standard answer to the original question; then, based on the industry knowledge base of a specified industry, a first result of the original question is retrieved, and it can be understood that the first result comes from the industry knowledge base; then the original question is input into a search engine to obtain a second result returned by the search engine; the first result, the second result and the target text of the original answer are combined to obtain a reference answer, and it can be understood that the reference answer includes both the standard original answer, the answer in the industry knowledge base, and the answer of the search engine; the reference answer and the original question are input into a large language model to obtain a predicted answer to the original question; finally, the large language model is trained based on the predicted answer and the original answer as a label. Since the training data of the large language model includes not only the standard answers to the original questions, but also supplementary answers based on industry knowledge bases and search engine retrieval, so that the large language model can answer more professional questions within the industry, this method can be trained on the existing large language model, thereby reducing the iterative cost of training professional industry large language models and customizing more accurate industry large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1This is an application environment diagram of the model training method and the problem data processing method in the embodiments of the present application;

[0073] Figure 2 A flow chart of the model training method provided in the embodiment of the present application;

[0074] Figure 3 A flow chart of the model training method provided in the embodiment of the present application;

[0075] Figure 4 A flowchart of a method for processing problem data in an embodiment of the present application;

[0076] Figure 5 A schematic diagram of an embodiment of a model training device in an embodiment of the present application;

[0077] Figure 6 This is a schematic diagram of an embodiment of a problem data processing device in an embodiment of the present application;

[0078] Figure 7 A framework diagram of the large language model provided in this embodiment;

[0079] Figure 8 This is a schematic diagram of a server structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0081] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0082] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0083] Natural language processing (NLP) is a key area of research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0084] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0085] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0086] The solution provided in the embodiments of this application involves natural language processing and machine learning technologies. By fine-tuning and training existing large language models based on specific industry content, it can more accurately answer user questions within the specified industry. The specific examples below will illustrate this.

[0087] LLM: Large Language Model is a deep learning model trained using a large amount of text data. It understands and generates human language by processing large amounts of text data. It includes the LLaMA2 model with an autoregressive Transformer architecture (model parameters range from 7B to 65B), the Chinese-English bilingual pre-trained model ChatGLM2-6B, the ChatGPT model, and the GPT4 model, among which ChatGPT and GPT4 are chatbot programs.

[0088] Large language models (LLMs) are trained on large amounts of text data to develop capabilities that support human language understanding and generation. They can perform a wide range of tasks, including text summarization, translation, and sentiment analysis. Large language models can handle a variety of natural language tasks, such as text classification, question-answering, and conversation. General-purpose large language models are typically pre-trained based on a wide range of public literature and online information. By leveraging language knowledge learned from large corpora, they can handle text tasks in most scenarios.

[0089] However, when answering highly technical industry questions, directly leveraging the capabilities of a general-purpose large model often results in incorrect or irrelevant responses (hallucinations), leading users to misjudge answers. While fine-tuning a large language model directly based on specialized domain knowledge can help the model better answer specialized questions, this requires accumulating user data for fine-tuning, resulting in high model iteration costs.

[0090] In order to alleviate this problem, this application proposes a model training method, a question data processing method and related devices, which use the professional knowledge base and search engine of any industry to search for common industry questions and obtain corresponding answers, and then train the existing large language model based on the industry questions and answers, so that the trained large language model can enhance its ability to answer these professional questions, and can provide users with more accurate services.

[0091] For easier understanding, see Figure 1 , Figure 1 This is an application environment diagram of the model training method and the problem data processing method in the embodiment of the present application, such as Figure 1 As shown, the model training method and the problem data processing method in the embodiment of the present application are applied to the problem data processing system. The problem data processing system includes: a server and a terminal device; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and the embodiment of the present application is not limited here.

[0092] First, the model training method in the embodiment of the present application is introduced from the perspective of the server. The server first obtains the original question-answer pair, which includes the original question and the corresponding original answer, and the original answer is the standard answer to the original question; then, based on the industry knowledge base of the specified industry, the first result of the original question is retrieved. It can be understood that the first result comes from the industry knowledge base; then the server inputs the original question into the search engine to obtain the second result returned by the search engine; the server combines the first result, the second result and the target text of the original answer to obtain a reference answer. It can be understood that the reference answer includes both the standard original answer, the answer in the industry knowledge base, and the answer of the search engine; the reference answer and the original question are input into the large language model to obtain the predicted answer to the original question; finally, the large language model is trained based on the predicted answer and the original answer as a label. Since the training data of the large language model includes not only the standard answer to the original question, but also the supplementary answer obtained based on the industry knowledge base and the search engine retrieval, the trained large language model can answer more professional technical questions in the industry.

[0093] The following describes the model training method in this application from the perspective of the server. Figure 2 , Figure 2 This is a flow chart of the model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application includes:

[0094] 201. Obtain an original question-answer pair, where the original question-answer pair includes an original question and an original answer corresponding to the original question.

[0095] Understandably, to enable large language models to answer more specialized technical questions, existing conventional large language models need to be trained based on training data. This first requires obtaining training data. Large language models can achieve better results using supervised learning. The corresponding training data consists of question-answer pairs, which consist of questions and answers. Questions serve as the model's input, and answers serve as the model's desired output.

[0096] In this step, it is necessary to obtain multiple original question-answer pairs in advance. Each original question-answer pair includes an original question and an original answer corresponding to the original question. There are many ways to obtain the original question. For example, historical questions raised by users during the use of conventional large language models can be summarized and screened to specifically screen out questions that are highly relevant to the specified industry; questionnaires or topic discussions can be posted on social media platforms to collect user feedback and opinions to summarize various issues in the specified industry; various products or services in the current industry can be analyzed to collect problems encountered by users in the process of using various products and services; industry experts or professionals in the field can be consulted to collect problems they encounter in actual work; and questions and data related to the specified industry can be obtained from public data sources, such as academic papers, industry reports, government public data, etc.

[0097] It is understood that the collected original questions can be further screened and sorted to remove duplicate, invalid or vague questions to ensure the accuracy and clarity of the questions. In addition, the questions can be classified and summarized for subsequent processing and analysis.

[0098] The original answer is the standard answer corresponding to the original question. The original answer can be obtained based on relevant materials in the field, such as textbooks, academic papers, authoritative guides or professional websites, and then provided by professionals, researchers, scholars or people with relevant knowledge based on their professional knowledge and experience to ensure accurate and reliable answers.

[0099] 202, retrieving at least one first result of the original question based on the industry knowledge base;

[0100] After obtaining multiple original questions, the answers corresponding to each original question can be obtained based on the industry knowledge base, and the answers are the first results. The first result can be obtained through the following feasible methods: by searching for text related to the original question in the industry knowledge base to find the most matching answer. This method can use text matching algorithms or natural language processing technologies, such as cosine similarity, Jaccard similarity, etc.; according to preset rules, find answers that match the original question in the industry knowledge base. The rules can be based on different aspects such as semantics, grammar, context, etc., and can be manually defined; use machine learning algorithms to train the industry knowledge base to learn how to answer the original questions, such as through a large amount of training data and computing resources, select appropriate machine learning models and algorithms. Common machine learning methods include deep learning, support vector machines, naive Bayes, etc.; build a knowledge graph to represent the knowledge in the industry knowledge base in a structured manner, and then use a graph query language to query the graph to find answers related to the original question.

[0101] 203. Obtain at least one second result of the original question based on the search engine.

[0102] It can be understood that in order to enable the large language model to better answer general or real-time questions raised by users, the method provided in this application introduces a search engine to crawl responses to the original question on web pages from various sources, that is, using a search engine to find information related to the original question, and selecting one or more second results from the search results as answers or solutions to the original question.

[0103] Some feasible methods for this step include: entering keywords or phrases that exactly match the original question to search, such as copying and pasting the original question directly into the search engine to find related web pages or resources; optimizing keywords for the original question, selecting the most relevant keywords to combine to improve the accuracy and relevance of the search results, for example, you can use the search engine's keyword tool or analysis tool to optimize keyword selection; using the search engine's advanced search functions, such as limiting the search scope, filtering search results, excluding specific keywords, etc., to improve search efficiency and accuracy; filtering out the most relevant, authoritative, and reliable websites or resources in the search results, excluding invalid content such as advertisements and low-quality websites, and ensuring Ensure the accuracy and reliability of the target results; if an original question has multiple target results, the multiple results can be compared and comprehensively analyzed to obtain more comprehensive and accurate answers or solutions; in addition to traditional web searches, you can also use industry-related knowledge question and answer platforms to enter the original question and obtain answers or experience sharing provided by users. These platforms usually gather the wisdom and experience of a large number of professionals and ordinary users, and can provide answers and solutions from multiple angles; if the original question is a question in a certain academic field, you can use academic search engines to enter the question keywords to obtain relevant academic papers, research reports and other resources. These resources usually come from authoritative academic institutions or experts and have high reference value.

[0104] 204 , combining the first result, the second result, and the target text of the original answer to obtain a reference answer.

[0105] It is understandable that the first result is an answer generated based on the industry knowledge base, and the second result is a result of searching the original question based on a search engine.

[0106] Among them, the target text for obtaining the original answer refers to the text material of the standard answer obtained based on the original question, which can be obtained from an industry knowledge base, a search engine, or the answer or experience sharing provided by experts or technicians in this field based on the original question.

[0107] The reference answer obtained by combining the above three types of answers not only has a high accuracy rate in professional fields, but also improves the versatility and effectiveness of the answer to a certain extent. For example, if the designated industry is the robotics industry, this industry is currently a developing industry. Not only is the technology updated rapidly, but the series of policies issued for the industry are also continuously updated. If the questions raised by users in this industry are timely, such as "Please provide a province or city's corporate incentive policy for a certain segment of the robotics industry", then in order to answer the questions raised by users more accurately, it is necessary not only to refer to the basic industry knowledge base, but also to consider some continuously updated policy information in the industry. This type of policy information can be obtained based on search engines.

[0108] 205 , the original question and the reference question and answer are input into the pre-trained large language model to obtain the predicted answer to the original question output by the large language model.

[0109] It is understandable that the pre-trained large language model has semantic understanding and natural language processing capabilities. Therefore, after the original question and reference answer are input into the large language model, the predicted answer to the original question can be sorted out based on the reference answer according to the understanding ability of the large language model.

[0110] For example, a preset query statement (prompt) is used to guide the model to generate a specific output: prompt = "Based on the following known information, answer the following specified question concisely and professionally. If you cannot get an answer from it, ignore the question. The known content includes: {***}, and the question includes: {###}". Among them, "***" indicates the above reference answer, and "###" indicates the above original question.

[0111] 206. A large language model is trained based on the predicted answer and the original answer as a label. The trained large language model is used to obtain a target answer to a target question based on a target reference answer. The target reference answer is retrieved from an industry knowledge base and a search engine based on the target question.

[0112] The original question and original answer in a raw question-answer pair are equivalent to the training data and labels used in model training. By training a large language model with this training data, a large language model capable of answering industry questions can be obtained. This training data is the raw question-answer pair, which consists of the original question and the corresponding original answer.

[0113] Training a large language model based on the predicted answers and the labeled original answers is equivalent to supervised training of the large language model. The parameters in the large language model are adjusted using a loss function, where the loss function indicates the similarity between the predicted answer and the expected answer, which is the original answer in the original question-answer pair. It can be understood that based on the above, since the industry knowledge base and search engine results were introduced into the large language model during its training, the large language model has the ability to respond to professional questions. Therefore, it is also possible to obtain the target answer corresponding to the target question based on the large language model, and the target question is a professional question within the industry.

[0114] The model training method provided in the embodiment of the present application first obtains the original question-answer pair, in which the original answer is the standard answer to the original question; then, the first result and the second result of the original question are obtained based on the industry knowledge base and the search engine respectively, the first result comes from the industry knowledge base, so it has a strong professionalism, and the second result is crawled after the original question is input into the search engine, so the second result has a high versatility and timeliness; the first result, the second result and the original answer are combined to obtain a reference answer, which ensures the diversity of the answer content; the original question and the reference answer are input into the pre-trained large language model to obtain the predicted answer to the original question output by the large language model; the large language model is supervisedly trained based on the predicted answer and the original answer as a label, so that the trained large language model can effectively improve the accuracy of the industry question, and finally the target question of the industry profession is answered by the large language model to obtain the corresponding target answer. Since the embodiment of the present application is equivalent to training an existing large language model, it can reduce the iterative cost of model training and customize a more accurate industry large language model.

[0115] In this application Figure 2 In an optional embodiment of the model training method provided in the corresponding embodiment, please refer to Figure 3 , Figure 3 The method flow chart of the model training method provided in the embodiment of the present application includes:

[0116] 301, segmenting the text in the industry knowledge base based on a preset length to obtain multiple text segments;

[0117] 302 : Input the text segment into the large language model to obtain the original question-answer pair corresponding to the text segment output by the large language model.

[0118] It is understandable that step 301 and step 302 are Figure 2 This corresponds to a possible implementation of step 201 in the embodiment.

[0119] In this embodiment, the industry knowledge base is sliced in advance, for example, all the industry knowledge base texts are sliced according to a preset word count (for example, 500 words) using a text slicing algorithm to obtain multiple text segments.

[0120] It is understandable that when slicing the industry knowledge base, a sliding window algorithm can be introduced to complete the end content of each segment. Furthermore, the industry knowledge base may include formats such as PDF, Word, and Markdown. Before slicing the professional field text materials in the industry knowledge base, a unified format operation can be performed, such as converting them into TXT text to facilitate slicing.

[0121] The resulting text segments are then fed into a large language model. Based on the model's natural language processing algorithms and models, the large language model processes the text segments into original answers that conform to natural language rules. A corresponding question, known as the original question, is generated for each original answer. The original question and original answer corresponding to each text segment constitute an original question-answer pair.

[0122] 303 : Input the industry knowledge base and the original question into the large language model to obtain the first result output by the large language model.

[0123] In this embodiment, step 303 is Figure 2 This corresponds to a possible implementation of step 202 in the embodiment.

[0124] It is understandable that since the existing large language models are trained on a large amount of corpus data and have a certain reading comprehension ability, the method of obtaining the first result corresponding to the original question based on the industry knowledge base of a specified industry can also be implemented through the existing large language models. For example, prompt = "Based on the following known information, answer the following specified questions concisely and professionally. If you cannot get the answer from it, ignore the question. The known content includes: {***}, and the questions include: {1, ***; 2, ***; 3, ***; ....}". The known content is the industry knowledge base, and the questions include multiple original questions.

[0125] Furthermore, considering that some smaller large language models (such as 7B or 13B models) may have problems understanding long texts, the industry knowledge base can be pre-sliced to alleviate this problem:

[0126] In one possible implementation, Figure 2 The corresponding step 202 may also include:

[0127] Based on the relevance between the original question and the text fragments, at least one first result is obtained, where the first result belongs to multiple text fragments, and the relevance between the first result and the original question is higher than a first preset value.

[0128] In this implementation, if the industry knowledge base is pre-sliced, the relevance between each segmented text segment and the original question can be calculated, and the text segment with a relevance above a preset threshold is determined as the first result. Specifically, after obtaining each text segment, a text retrieval model is used to extract features based on the semantics of each text segment to facilitate the calculation of relevance.

[0129] Since the first result is a text fragment, after obtaining the first result, the first result can be processed into text content that conforms to natural language rules through algorithms and models based on natural language processing.

[0130] 304, input the original question into the search engine and obtain multiple search results;

[0131] 305 : Filter duplicate results in the multiple search results to obtain at least one second result.

[0132] In this embodiment, steps 304 and 305 are Figure 2 This corresponds to a possible implementation of step 203 in the embodiment.

[0133] It is understandable that, considering that the content of web pages from different sources searched based on a search engine may be the same, that is, the same text content is published on different web pages or platforms, after the original question is input into the search engine, the embodiment of the present application can compare the similarities between the multiple search results obtained and remove duplicate content.

[0134] For example, you can determine whether two web pages are similar or duplicate by comparing their text content, URL addresses, keyword density, etc. If the similarity is high, you can exclude one of the results and keep the other non-duplicate content as the second result.

[0135] Furthermore, due to the diversity of the crawled web pages, the number of words on some web pages may be as long as tens of thousands. Therefore, for each web page content, a text slicing algorithm can be used to divide the web page according to a preset number of words, and a sliding window algorithm can be introduced to fill in the beginning and end content of each segment. Then, a text retrieval model is used to extract features based on the semantics of each text segment. Then, based on the extracted features, one or more segments with the highest relevance score to the original question are selected as the first result for each web page.

[0136] 306. If the number of second results is greater than K, where K is an integer greater than 1, determine the correlation between the original question and the second results; and include the K second results with the highest correlation in the reference answer.

[0137] It is understandable that since there may be a large number of web pages crawled, the number of second results is also large. Due to the limitation of the reasoning ability of the large language model, it is difficult to include all the second results in the reference answer. For this problem, the embodiment of the present application can use a correlation algorithm to calculate the correlation between the original question and each second result, sort the multiple second results based on the correlation score, and then combine the K second results with the highest correlation into the reference answer. Specifically, the minimum edit distance algorithm can be introduced to calculate the character similarity score between the original question and each second result, re-sort the multiple second results from high to low based on the character similarity score, and then combine the first K second results from different sources with the first result and the original answer.

[0138] In one possible implementation, if the text content in both the industry knowledge base and the webpage is sliced, then the first and second results are also composed of text slices. In this case, features can be extracted from the text slices of the first and second results, and their correlation with the original question can be calculated. Then, based on the correlation results, all text slices are reordered, and the top M text segments (including the text segments in the first result and the second result) are selected and combined with the original answer to generate the final reference answer.

[0139] 307 , combining the first result, the second result, and the text segment of the original answer to obtain a reference answer.

[0140] It is understandable that step 307 and Figure 2 This corresponds to step 205 in the embodiment.

[0141] In a possible implementation method, the original answer includes multiple answer segments, and the multiple answer segments are extracted from the multiple target segments in the order of the multiple answer segments; the target text is obtained by disrupting the order of the multiple target segments.

[0142] In this embodiment, the original answer includes multiple answer segments, and there is a sequence between the multiple answer segments. For example, the original answer is a text segment with a sequence mark, and the sequence mark refers to various serial numbers and labels including node keywords such as "Chapter X", "Section X", and "Step X". Each sequence mark corresponds to an answer segment, and each answer segment corresponds to a target segment. The original answer is: multiple answer segments are arranged in order, and the text material is obtained after being processed by a large language model. The target text obtained by obtaining the original answer includes the above-mentioned multiple target segments, but the order of the multiple target segments in the target text is very messed up.

[0143] Since the answer fragments in the original answer are arranged in the correct order, while the order of the target fragments in the target text is disrupted, training the model through the original answer is equivalent to adding training features including text order. It is expected that the trained large language model can output answers corresponding to the order in the original answer, so that the text order in the expected output predicted answer is consistent with the logic in the original answer.

[0144] For example, the target segments of the original answer include target segment X, target segment Y, and target segment Z. The original answers obtained by inputting into the large language model include: "First step: AAA", "Second step: BBB", and "Third step: CCC", where "AAA" corresponds to target segment X, "BBB" corresponds to target segment Y, and "CCC" corresponds to target segment Z.

[0145] Then the target text also includes target segment X, target segment Y and target segment Z, but in the target text, the arrangement of the above segments can be: target segment X, target segment Z and target segment Y; or target segment Y, target segment X and target segment Z; or target segment Y, target segment Z and target segment X; or target segment Z, target segment X and target segment Y; or target segment Z, target segment Y and target segment X.

[0146] 308, inputting the original question and the reference answer into the pre-trained large language model to obtain a predicted answer to the original question output by the large language model;

[0147] 309, a large language model is trained based on the predicted answer and the original answer as a label. The trained large language model is used to obtain a target answer to a target question based on a target reference answer. The target reference answer is retrieved from an industry knowledge base and a search engine based on the target question.

[0148] In this embodiment, step 308 and step 309 are respectively Figure 2 Step 205 corresponds to step 206 in the corresponding embodiment. For related descriptions, please refer to the above and will not be repeated here.

[0149] In a possible implementation method, before step 307, the method further includes:

[0150] Expanding the original question to obtain a corresponding similar question, wherein the semantic matching degree between the similar question and the original question is higher than a second preset value;

[0151] Similar questions and reference answers are input into the large language model to obtain the predicted answer to the original question output by the large language model.

[0152] In this embodiment, considering that existing large language models may not be able to respond effectively to similar questions or similar descriptions of the same question, similar questions can generate different responses. Therefore, the original question can be expanded to obtain a set of questions with similar semantics. These similar questions have a high degree of semantic match with the original question, that is, they are close or related to the original question in content, topic, or intent. By asking similar questions, knowledge in related fields can be more comprehensively explored and richer and more diverse answers can be obtained.

[0153] To expand the original question, you can mine similar terms based on the industry knowledge base and replace the existing question. You can also use a preset prompt to guide the existing large language model to output similar questions. For example, prompt = "Please output five similar questions based on the following question content. Question: ****".

[0154] The model training method provided in the embodiment of the present application introduces a professional industry knowledge base to limit the large language model to answer questions within the retrieved text fragment content, thereby reducing the large language model's own generation of erroneous or false technical content and regulations, wherein the text fragment is a fragment obtained by segmenting the industry knowledge base content; at the same time, a search engine is referenced, through which the latest policies or regulations and common non-professional questions can be retrieved in real time, thereby reducing the work of maintaining the professional knowledge base and greatly improving the response performance of the large language model; the search engine's search results are also deduplicated, screened, and sorted, further improving the accuracy of the answers; this embodiment also expands similar questions based on the original question to enhance the ability to understand similar questions.

[0155] The following is a detailed description of the problem data processing method in this application. Figure 4 , Figure 4 This is a flowchart of a method for processing problem data in an embodiment of the present application, including:

[0156] 401, get target question;

[0157] It is understandable that the target question refers to the question that needs to be answered by the large language model. The question can be a professional question in a specified industry. For the professional question in the specified industry, the target question can be answered by the large language model according to the above Figure 2 or Figure 3 The large language model trained by the model training method in the corresponding embodiment is used to answer. For the specified industry, the server stores a corresponding industry knowledge base.

[0158] 402. Retrieve at least one first target result for the target question based on the industry knowledge base;

[0159] After obtaining the target question, the answer corresponding to the target question can be obtained based on the industry knowledge base, and the answer is the first target result. The first target result can be obtained in the same way as above. Figure 2 or Figure 3 The method for obtaining the first result in the corresponding embodiment is similar and will not be described again here.

[0160] In a possible implementation method, before step 402, the method further includes:

[0161] The text in the industry knowledge base is segmented based on a preset length to obtain multiple text segments.

[0162] At this time, step 402 specifically includes:

[0163] Based on the relevance between the target question and the text fragments, at least one first target result is obtained, the first target result belongs to multiple text fragments, and the relevance between the first target result and the target question is higher than a first preset value.

[0164] It is understandable that, considering the large amount of text content in the industry knowledge base, it takes a lot of time and computing power for a large language model to understand and process long texts, and the processing results are difficult to meet expectations. In order to solve this problem, the industry knowledge base can be sliced in advance and stored in the server. In this case, for each text segment after segmentation, the correlation between each text segment and the target question can be calculated, and the text segment with a correlation higher than a preset threshold is determined as the first target result. Specifically, after obtaining each text segment, the text retrieval model is used to extract features based on the semantics of each text segment to facilitate the calculation of correlation.

[0165] Furthermore, since the first target result is a text fragment, after obtaining the first target result, the first target result can be processed into text content that conforms to natural language rules through algorithms and models based on natural language processing.

[0166] 403. Retrieve at least one second target result for the target question based on the search engine.

[0167] After obtaining the target question, the method further includes obtaining an answer corresponding to the target question based on the search engine, and the answer is the second target result. The method for obtaining the second target result can be the same as the above method. Figure 2 or Figure 3 The method for obtaining the second result in the corresponding embodiment is similar and will not be described again here.

[0168] In a possible implementation method, step 403 specifically includes:

[0169] 4031, input the target question into the search engine and obtain multiple search results;

[0170] 4032. Filter duplicate results in the multiple search results to obtain at least one second target result.

[0171] It is understandable that, considering that the content of web pages from different sources searched based on a search engine may be the same, that is, the same text content is published on different web pages or platforms, after the target question is input into the search engine, the embodiment of the present application can compare the similarities between the multiple search results obtained and remove duplicate content.

[0172] 404, combining the first target result and the second target result to obtain a target reference answer;

[0173] It can be understood that the first target result is the answer generated based on the industry knowledge base, and the second target result is the result of searching the target question based on a search engine. The target reference answer obtained by combining the two has a high accuracy rate in the professional field and also improves the versatility and effectiveness of the answer to a certain extent.

[0174] In a possible implementation method, when the industry knowledge base is a text material with sequential tags, and when there are multiple first target results, the multiple first target results in the target reference answer are sorted according to the sequential tags.

[0175] It is understandable that the answer corresponding to the target question retrieved from the industry knowledge base may exist in multiple text fragments. In the process of determining the relevance between the original question and each text fragment, the order of the text fragments may be disrupted, and the output of multiple first target results may have causal relationship transformation and logical confusion. To address this problem, after obtaining multiple first target results, this embodiment will take into account the order between related knowledge points. That is, for the sentences with the above-mentioned sequence mark, the multiple first target results will be sorted according to the sequence mark when combining them into the target reference answer.

[0176] In a possible implementation method, after step 403, the following steps are further included:

[0177] If the number of second target results is greater than K, where K is an integer greater than 1, the correlation between the target question and the second target results is determined; at this time, the target reference answer includes the K second target results with the highest correlation.

[0178] It is understandable that since there may be a large number of crawled web pages, the number of second target results is also large. Due to the limitation of the reasoning ability of the large language model, it is difficult to include all second target results in the target reference answer. To address this issue, the embodiment of the present application can use a correlation algorithm to calculate the correlation between the target question and each second target result, sort the multiple second target results based on the correlation score, and then combine the K second target results with the highest correlation into the target reference answer.

[0179] 405, the target question and the target reference answer are input into the pre-trained large language model to obtain the target answer of the target question output by the large language model. The large language model is based on the above Figure 2 and Figure 3 The model training method of the corresponding embodiment is trained.

[0180] It is understandable that after the above Figure 2 and Figure 3 The large language model trained by the model training method in the corresponding embodiment is not only imported with the results of the industry knowledge base and search engine during the training process, but also includes the standard answers (corresponding to the original answers) to professional questions (corresponding to the original questions). Therefore, the large language model has the ability to answer professional questions. Then, based on the large language model, the target answer corresponding to the target question can be obtained. The target question corresponds to a professional question in the industry. The target answer also integrates the results of the target question based on the industry knowledge base and search engine, and has stronger professionalism, versatility and timeliness.

[0181] The following is an introduction to the model training device in this application. Figure 5 , Figure 5 FIG. 5 is a schematic diagram of an embodiment of a model training device 500 in an embodiment of the present application. The model training device 500 includes:

[0182] A first acquisition module 501 is configured to acquire an original question-answer pair, where the original question-answer pair includes an original question and an original answer corresponding to the original question;

[0183] A first retrieval module 502 is configured to retrieve at least one first result of the original question based on the industry knowledge base; and further configured to retrieve at least one second result of the original question based on a search engine;

[0184] A first combining module 503 is configured to combine the first result, the second result, and the target text of the original answer to obtain a reference answer;

[0185] A first input module 504 is configured to input the original question and the reference answer into the pre-trained large language model to obtain a predicted answer to the original question output by the large language model;

[0186] The training module 505 is used to train a large language model based on the predicted answer and the original answer as a label. The trained large language model is used to obtain a target answer to the target question based on the target reference answer. The target reference answer is retrieved from the industry knowledge base and search engine based on the target question.

[0187] In a possible implementation method, the first retrieval module 502 is specifically configured to input the industry knowledge base and the original question into the large language model to obtain a first result output by the large language model.

[0188] In one possible implementation method, the first acquisition module 501 is specifically used to segment the text in the industry knowledge base based on a preset length to obtain multiple text segments; input the text segments into the large language model to obtain the original question-answer pairs corresponding to the text segments output by the large language model.

[0189] In one possible case of a possible implementation method, the first retrieval module 502 is specifically used to obtain at least one first result based on the relevance between the original question and the text fragment, the first result belongs to multiple text fragments, and the relevance of the first result to the original question is higher than a first preset value.

[0190] In a possible implementation method, the original answer includes multiple answer segments, and the multiple answer segments are extracted from the multiple target segments in the order of the multiple answer segments; the target text is obtained by disrupting the order of the multiple target segments.

[0191] In a possible implementation method, the first retrieval module 502 is specifically configured to input the original question into a search engine to obtain multiple search results; and filter duplicate results in the multiple search results to obtain at least one second result.

[0192] In one possible implementation method, if the number of second results is greater than K, where K is an integer greater than 1, the first retrieval module 502 is also used to determine the correlation between the original question and the second results; the reference answer includes the K second results with the highest correlation.

[0193] In a possible implementation method, the method further includes: a similar question expansion module, configured to expand the original question to obtain corresponding similar questions, wherein the semantic matching degree between the similar question and the original question is higher than a second preset value;

[0194] The training module 505 is further configured to input similar questions and reference answers into the large language model to obtain a predicted answer to the original question output by the large language model.

[0195] It is understandable that the model training device provided in the embodiment of the present application is used to perform the above Figure 2 or Figure 3 For the detailed description of the steps in the model training method in the corresponding embodiment, please refer to the above and will not be repeated here.

[0196] The following is a detailed description of the problem data processing device in this application. Figure 6 . Figure 6 This is a schematic diagram of an embodiment of a problem data processing device 600 in an embodiment of the present application. The problem data processing device 600 includes:

[0197] The second acquisition module 601 is used to acquire the target question;

[0198] A second retrieval module 602 is configured to retrieve at least one first target result for the target question based on an industry knowledge base; and further configured to retrieve at least one second target result for the target question based on a search engine;

[0199] A second combining module 603 is configured to combine the first target result and the second target result to obtain a reference answer;

[0200] The second input module 604 is used to input the target question and the target reference answer into the pre-trained large language model to obtain the target answer of the target question output by the large language model. The large language model is based on the above Figure 2 or Figure 3 The model is trained using the model training method in the corresponding embodiment.

[0201] In one possible implementation method, the second retrieval module 602 is specifically used to divide the text in the industry knowledge base based on a preset length to obtain multiple text fragments; based on the relevance between the target question and the text fragments, obtain at least one first target result, the first target result belongs to multiple text fragments, and the relevance of the first target result to the target question is higher than a first preset value.

[0202] In a possible implementation method, when the industry knowledge base is a text material with sequential tags, and when there are multiple first target results, the multiple first target results in the target reference answer are sorted according to the sequential tags.

[0203] In one possible implementation, the second search module 602 is specifically configured to input a target question into a search engine to obtain multiple search results; filter duplicate results from the multiple search results to obtain at least one second target result. In one possible implementation, if the number of second target results is greater than K, where K is an integer greater than 1, the second search module 602 is further configured to determine the relevance between the target question and the second target results; in this case, the target reference answer includes the K second target results with the highest relevance.

[0204] It is understandable that the problem data processing device provided in the embodiment of the present application is used to perform the above Figure 4 For the detailed description of the steps in the problem data processing method in the corresponding embodiment, please refer to the above text and will not be repeated here.

[0205] See also Figure 7 , Figure 7 The framework diagram of the large language model provided in this embodiment. For the input target question, retrieval is performed through two parts, namely professional knowledge base retrieval and Internet search engine retrieval (equivalent to the second retrieval module 602). Among them, the professional knowledge base mainly involves knowledge related to the user's professional field. By organizing and slicing the materials provided by the user into a database, maintaining a knowledge base closely related to the user's questions and answers facilitates retrieval of fragments related to the questions; then an Internet search engine is introduced to retrieve replies to related questions. Since the crawled web page content may be large, the content of different web page sources can also be sliced to extract the fragments most relevant to the questions and answers. The professional knowledge base and the text content of the Internet search are combined (second combination module 603) and used as supplementary answers to the corresponding questions to train the large language model (second input module 604). Among them, for the combined results of the professional knowledge base and the text content of the Internet search, the minimum edit distance can be further used to calculate the relevance of the fragments from different sources and the user text, and sorted in descending order, and then the K text fragments with the highest relevance are selected as supplementary answers to the corresponding questions, and the reading comprehension prompt is constructed and input into the large language model for question and answer generation.

[0206] Figure 7The generation process of large language models for two industries is provided in the article. One of the questions belongs to the finance and taxation industry: "How are small-scale taxpayers identified as general taxpayers?", and the other question belongs to the tourism industry: "How is the travel situation during the National Day holiday this year?". For the above two industries, professional knowledge bases are pre-built (the industry knowledge base for the finance and taxation industry and the industry knowledge base for the tourism industry), and then the two questions are searched based on the professional knowledge base and the Internet (search engine). The search results are combined and input into the large language model to realize the training of the large language model. The trained large language model can output more professional and timely answers based on the industry knowledge base and Internet search results. The specific answer content is as follows: Figure 7 As shown, no further description is given here.

[0207] In order to prove that the large language model generated by the model training method provided in the application embodiment has a high accuracy rate when answering professional questions, the following is a test based on the tax professional field.

[0208] First, a test dataset of 100 professional tax questions was constructed for testing in the tax field. The experimental results were evaluated using the accuracy of manually evaluated questions. To more intuitively demonstrate the importance of large language models' reading comprehension capabilities in professional domain knowledge, GPT4 was first introduced to evaluate the performance of general large models in question-answering within this domain. As shown in Table 1, it can be seen that infusing professional domain knowledge by using the prompt "Based on the following known information, answer the user's question concisely and professionally. If the answer cannot be obtained, ignore the text and answer the user's question in Chinese.\nKnown content: {}\nQuestion: {}" can significantly address large language models' hallucinations or lack of understanding of questions. However, large models like GPT4, with hundreds of billions of parameters, are difficult to deploy privately. Therefore, smaller large language models, such as 7 or 13 Bytes, can be fine-tuned to achieve even stronger reading comprehension capabilities than GPT4.

[0209] Table 1 Test results of different models based on knowledge injection indicators

[0210]

[0211] In actual application scenarios, it is difficult to directly give the text fragment corresponding to the real question raised by the user, that is, it is impossible to accurately obtain the known content of the question. In the face of this situation, based on the problem data processing method provided in the embodiment of the present application, the relevant text fragments involved in the user's question are retrieved by introducing a professional knowledge base and an Internet search engine. As shown in Table 2, by combining the professional knowledge base + Internet search engine, more diverse knowledge is injected into the large language model, which can effectively enhance the reading comprehension ability of the large language model and improve the accuracy of the large language model's answers to professional questions. However, we analyzed the wrong questions in the test data set and found that the current large language model still lacks the ability to handle some longer text fragments, similar questions or similar descriptions and cross-document summaries (logical confusion of the context), resulting in the model's answer quality to such questions being not high.

[0212] Table 2 Indicators of different knowledge injection methods for large language models

[0213]

[0214] To further improve the reading comprehension capabilities of large language models, we constructed training data for supervised fine-tuning (SFT) to address issues such as long text comprehension, similar question comprehension, and cross-document summarization. The experimental results are shown in Table 3. By constructing targeted training data for fine-tuning, we can effectively improve the reading comprehension capabilities of large language models, even surpassing the performance of general-purpose large models such as GPT4.

[0215] Table 3. Indicators of large language models with supervised fine-tuning

[0216]

[0217] In summary, the method provided in the embodiments of the present application can effectively reduce the problem of hallucinations in professional fields of large language models by injecting professional knowledge through the introduction of a multi-source (industry knowledge base + search engine) retrieval method. Moreover, this method can be trained on a small-scale large language model, greatly reducing the cost of large language model training iterations, thereby customizing more accurate industry large language models.

[0218] Figure 8: This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 322 (for example, one or more processors) and memories 332, and one or more storage media 330 (for example, one or more massive storage devices) for storing application programs 342 or data 344. Among them, the memories 332 and the storage media 330 may be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0219] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0220] The steps performed by the server in the above embodiment can be based on the Figure 8 The server structure shown.

[0221] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0222] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0223] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0224] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0225] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0226] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0227] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: include: Obtaining an original question-answer pair, the original question-answer pair including an original question and an original answer corresponding to the original question; Retrieving at least one first result of the original question based on the industry knowledge base; Retrieving at least one second result for the original question based on a search engine; Combining the first result, the second result, and the target text of the original answer to obtain a reference answer; Inputting the original question and the reference answer into a pre-trained large language model to obtain a predicted answer to the original question output by the large language model; The large language model is trained based on the predicted answer and the original answer as a label. The trained large language model is used to obtain a target answer to a target question based on a target reference answer. The target reference answer is retrieved from an industry knowledge base and a search engine based on the target question.

2. The method according to claim 1, characterized in that The retrieving at least one first result of the original question based on the industry knowledge base includes: The industry knowledge base and the original question are input into the large language model to obtain the first result output by the large language model.

3. The method according to claim 1, characterized in that The obtaining of the original question-answer pair includes: Segmenting the text in the industry knowledge base based on a preset length to obtain multiple text segments; The text segment is input into the large language model to obtain the original question-answer pair corresponding to the text segment output by the large language model.

4. The method according to claim 3, characterized in that The retrieving at least one first result of the original question based on the industry knowledge base includes: Based on the relevance between the original question and the text fragments, at least one first result is obtained, the first result belongs to multiple text fragments, and the relevance between the first result and the original question is higher than a first preset value.

5. The method according to claim 4, characterized in that The original answer includes multiple answer segments, and the multiple answer segments are extracted from multiple target segments in the order of the multiple answer segments; the target text is obtained by disrupting the order of the multiple target segments.

6. The method according to claim 1, characterized in that After obtaining the original question-answer pair, the following steps are also included: Expanding the original question to obtain a corresponding similar question, wherein the semantic matching degree between the similar question and the original question is higher than a second preset value; The similar question and the reference answer are input into the large language model to obtain the predicted answer to the original question output by the large language model.

7. The method according to claim 1, characterized in that If the number of the second results is greater than K, where K is an integer greater than 1, after retrieving at least one second result of the original question based on the search engine, the method further includes: Determine the correlation between the original question and the second results; the reference answer includes K second results with the highest correlation.

8. A method for processing problem data, characterized in that: include: Get the target question; Retrieving at least one first target result for the target question based on an industry knowledge base; Retrieving at least one second target result for the target question based on a search engine; Combining the first target result and the second target result to obtain a target reference answer; The target question and the target reference answer are input into a pre-trained large language model to obtain a target answer to the target question output by the large language model, wherein the large language model is trained according to the model training method according to any one of claims 1 to 7.

9. A model training device, characterized in that: include: A first acquisition module is configured to acquire an original question-answer pair, wherein the original question-answer pair includes an original question and an original answer corresponding to the original question; A first retrieval module is configured to retrieve at least one first result of the original question based on an industry knowledge base; and is further configured to retrieve at least one second result of the original question based on a search engine; A first combining module is configured to combine the first result, the second result, and the target text of the original answer to obtain a reference answer; A first input module is configured to input the original question and the reference answer into a pre-trained large language model to obtain a predicted answer to the original question output by the large language model; A training module is used to train the large language model based on the predicted answer and the original answer as a label, and the trained large language model is used to obtain a target answer to a target question based on a target reference answer, and the target reference answer is retrieved from an industry knowledge base and a search engine based on the target question.

10. A problem data processing device, characterized in that: include: The second acquisition module is used to obtain the target question; A second retrieval module is configured to retrieve at least one first target result for the target question based on the industry knowledge base; Also used for retrieving at least one second target result of the target question based on a search engine; A second combining module is used to combine the first target result and the second target result to obtain a reference answer; A second input module is used to input the target question and the target reference answer into a pre-trained large language model to obtain the target answer to the target question output by the large language model, wherein the large language model is trained according to the model training method according to any one of claims 1 to 7.

11. A computer device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, including executing the model training method according to any one of claims 1 to 7, or executing the problem data processing method according to claim 8; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

12. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to execute the model training method as described in any one of claims 1 to 7, or the problem data processing method as described in claim 8.

13. A computer program product comprising a computer program, characterized in that The computer program is used by a processor to execute the model training method according to any one of claims 1 to 7, or to execute the problem data processing method according to claim 8.

Images