Large language model training method and device, electronic equipment and storage medium

Abstract

The invention discloses a large language model training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and reinforcement learning. The method comprises the steps of obtaining an initial prediction answer statement corresponding to a question statement; semantic expansion retrieval is carried out on the initial prediction answer statement, and a retrieval answer statement corresponding to the initial prediction answer statement is determined; scoring the retrieval answer statement, and determining a target reward value of the initial prediction answer statement based on a scoring result of the retrieval answer statement; and if the target reward value is smaller than a preset reward threshold value, adjusting the pre-trained large language model based on the target reward value, and inputting the question statement into the adjusted model again until the finally obtained target reward value is greater than or equal to the preset reward threshold value, thereby completing training of the pre-trained large language model. The optimization training efficiency of the large language model and the accuracy of model output can be improved, and the stability of large language model output is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and reinforcement learning technology, and in particular to a large language model training method, device, electronic device and storage medium. Background Art

[0002] At present, in today's artificial intelligence field, the application of large language models is becoming more and more extensive. Reinforcement learning, as an important technical learning method, learns the best behavior strategy by interacting with the environment, and is of great significance in the application of large language models.

[0003] Reinforcement learning can be used to train models, generating more fluent and reasonable language expressions, improving both dialogue generation and text generation capabilities. This helps models continuously refine their language generation capabilities, enabling them to better meet user needs and enhance overall model performance. Through interactive learning with the environment, large language models can automatically learn and improve their expressive capabilities, generating more natural and fluent text. This is of great significance for applications such as dialogue systems, intelligent customer service, and intelligent writing.

[0004] Therefore, an effective training method for reinforcement learning of large language models is needed to optimize the training and reinforcement learning of large language models. Summary of the Invention

[0005] The embodiments of the present application provide a large language model training method, device, electronic device and storage medium. The embodiments provided by the present application solve the technical problems in the prior art of low optimization efficiency, poor output accuracy and poor robustness of large language model training. The embodiments provided by the present application can improve the optimization training efficiency of the large language model and the accuracy of the model output, and improve the stability of the large language model output.

[0006] In a first aspect of the embodiments of the present application, the embodiments of the present application provide a large language model training method, the method comprising:

[0007] Input the question sentence sent by the user into the pre-trained large language model, and output the initial predicted answer sentence corresponding to the question sentence;

[0008] Performing semantic expansion retrieval on the initial predicted answer sentence to determine a retrieval answer sentence corresponding to the initial predicted answer sentence;

[0009] Scoring the retrieved answer sentence, and determining a target reward value for the initial predicted answer sentence based on the scoring result of the retrieved answer sentence;

[0010] If the target reward value is less than the preset reward threshold, the pre-trained large language model is adjusted based on the target reward value, and the question statement is re-input into the adjusted model until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model.

[0011] In a feasible implementation, performing semantic expansion search on the initial predicted answer sentence to determine a search answer sentence corresponding to the initial predicted answer sentence includes:

[0012] Inputting the initial predicted answer sentence into a preset retrieval enhancement generation model, performing semantic expansion of the initial predicted answer sentence, and determining a semantic expansion sentence corresponding to the initial predicted answer sentence;

[0013] A semantic search is performed on the semantically expanded sentence to determine a search answer sentence corresponding to the initial predicted answer sentence.

[0014] In a feasible implementation, inputting the initial predicted answer sentence into a preset retrieval enhancement generation model, performing semantic expansion of the initial predicted answer sentence, and determining a semantic expansion sentence corresponding to the initial predicted answer sentence includes:

[0015] Inputting the initial predicted answer sentence into a preset retrieval enhancement generation model for data expansion, and determining a sample sentence database associated with the initial predicted answer sentence;

[0016] The sample sentence database is subjected to information fusion with the initial predicted answer sentence to determine a semantically expanded sentence corresponding to the initial predicted answer sentence.

[0017] In a feasible implementation manner, after performing semantic expansion search on the initial predicted answer sentence and determining a search answer sentence corresponding to the initial predicted answer sentence, the method further includes:

[0018] The reward value of the initial predicted answer sentence is calculated according to a preset format reward algorithm to determine the format reward value corresponding to the initial predicted answer sentence.

[0019] In a feasible implementation, scoring the retrieved answer statement and determining the target reward value of the initial predicted answer statement based on the scoring result of the retrieved answer statement includes:

[0020] Scoring the search answer statement to determine a scoring result of the search answer statement;

[0021] Based on the scoring result and the format reward value, a target reward value of the initial predicted answer sentence is determined.

[0022] In a feasible implementation manner, scoring the search answer statement and determining the scoring result of the search answer statement includes:

[0023] The retrieval answer statement is input into a preset teacher model for scoring, a retrieval reward value corresponding to the retrieval answer statement is determined, and the retrieval reward value is determined as a scoring result of the retrieval answer statement.

[0024] In a feasible implementation manner, determining the target reward value of the initial predicted answer sentence based on the scoring result and the format reward value includes:

[0025] The retrieval reward value and the format reward value are weighted and summed according to a preset weight coefficient ratio to determine the target reward value of the initial predicted answer sentence.

[0026] In a second aspect of the embodiments of the present application, the embodiments of the present application provide a large language model training device, an output module for inputting a question sentence issued by a user into a pre-trained large language model, and outputting an initial predicted answer sentence corresponding to the question sentence;

[0027] A first determining module is configured to perform semantic expansion retrieval on the initial predicted answer sentence to determine a retrieved answer sentence corresponding to the initial predicted answer sentence;

[0028] a second determination module, configured to score the retrieved answer statement and determine a target reward value for the initial predicted answer statement based on the score result of the retrieved answer statement;

[0029] An adjustment module is used to adjust the pre-trained large language model based on the target reward value if the target reward value is less than a preset reward threshold, and re-input the question statement into the adjusted model until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model.

[0030] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus, and the machine-readable instructions are executed by the processor as the steps of the large language model training method described above.

[0031] In a fourth aspect of the embodiments of the present application, the embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the large language model training method as described above are executed.

[0032] The large language model training method, device, electronic device and storage medium provided in the embodiments of the present application, compared with the prior art, the embodiments provided by the present application input the question sentence issued by the user into the pre-trained large language model, output the initial predicted answer sentence corresponding to the question sentence, and perform semantic expansion retrieval on the initial predicted answer sentence to determine the retrieval answer sentence corresponding to the initial predicted answer sentence; score the retrieval answer sentence, and then determine the target reward value of the initial predicted answer sentence based on the scoring result of the retrieval answer sentence; when the target reward value is less than a preset reward threshold, adjust the pre-trained large language model based on the target reward value, and input the question sentence into the adjusted model again until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model. The present application can improve the optimization training efficiency of the large language model and the accuracy of the model output, and improve the stability of the large language model output. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a large language model training method provided in an embodiment of the present application is shown;

[0034] Figure 2 A flowchart of determining a target reward value in a large language model training method provided in an embodiment of the present application is shown;

[0035] Figure 3 A structural block diagram of a large language model training device provided in an embodiment of the present application is shown;

[0036] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0037] Figure 3 and Figure 4 The corresponding relationship between the reference numerals and the names of the drawings is as follows:

[0038] 300 large language model training device; 310 output module; 320 first determination module; 330 third determination module; 340 second determination module; 350 adjustment module; 400 electronic device; 410 processor; 420 memory; 440 bus. DETAILED DESCRIPTION

[0039] In order to better understand the technical solutions provided by the embodiments of this specification, the technical solutions of the embodiments of this specification are described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0040] In this article, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprising a ..." do not exclude the presence of other identical elements in the process, method, article or equipment comprising the elements. The term "two or more" includes two or more than two cases.

[0041] First, the application scenarios to which this application is applicable are introduced. The embodiments provided in this application are applicable to the fields of artificial intelligence and reinforcement learning technology, and in particular, relate to a large language model training method, device, electronic device and storage medium.

[0042] Currently, there is a need for an effective training method for reinforcement learning of large language models to optimize the training and reinforcement learning of large language models.

[0043] Based on this, the embodiments of the present application provide a large language model training method, device, electronic device and storage medium. The embodiments provided by the present application solve the technical problems in the prior art of low optimization efficiency, poor output accuracy and poor robustness of large language model training. The embodiments provided by the present application can improve the optimization training efficiency of the large language model and the accuracy of the model output, and improve the stability of the large language model output.

[0044] Figure 1 This is a flowchart of a large language model training method provided in an embodiment of the present application. Figure 1 As shown, the large language model training method includes the following steps:

[0045] S101: Input the question sentence sent by the user into the pre-trained large language model, and output the initial predicted answer sentence corresponding to the question sentence.

[0046] In this step, the embodiment provided in this application requires using the question sentence issued by the user to perform reinforcement learning training on the pre-trained large language model, and outputs the initial predicted answer sentence corresponding to the question sentence during the first round of training of the large language model.

[0047] It can be understood that the type of pre-trained large language model in the embodiments provided in this application can be customized according to different application scenarios. The pre-trained large language model in the embodiments provided in this application can be a language training model based on the Transformer architecture.

[0048] For example: a series of models of pre-trained language models (Generative Pre-trained Transformer, GPT).

[0049] Among them, Large Language Models (LLM) are language processing models with a large number of parameters. These models can understand and generate natural language text and are used in scenarios such as interactive question and answer.

[0050] S102: Perform semantic expansion search on the initial predicted answer sentence to determine the search answer sentence corresponding to the initial predicted answer sentence.

[0051] In this step, after determining the initial predicted answer sentence output by the first round of iteration, the embodiment provided in this application starts to perform semantic expansion retrieval on the initial predicted answer sentence using the preset retrieval enhancement generation model in the pre-trained large language model, and then generates a new answer after the retrieval, and determines the newly generated answer as the retrieval answer sentence corresponding to the initial predicted answer sentence.

[0052] It can be understood that the embodiment provided in this application uses the semantic expansion retrieval function of the preset retrieval enhancement generation model to replace the traditional similarity matching model. Compared with the traditional similarity matching, the semantic expansion of the preset retrieval enhancement generation model can more accurately determine the answer sentence that has a high match with the initial predicted answer sentence and is semantically similar, and better reflect the language quality, logical consistency and matching degree of the generated content.

[0053] Exemplarily, the initial predicted answer sentence is input into a preset retrieval enhancement generation model, semantic expansion of the initial predicted answer sentence is performed, and the semantic expansion sentence corresponding to the initial predicted answer sentence is determined; semantic retrieval is performed on the semantic expansion sentence to determine the retrieval answer sentence corresponding to the initial predicted answer sentence.

[0054] In the above, the embodiment provided by the present application uses a preset retrieval enhancement generation model to perform data enhancement and data expansion on the initial predicted answer statement, by supplementing the sample with scarce or low-frequency information related to the initial predicted answer statement in certain application scenarios, generating a semantic expansion statement, and then performing semantic retrieval of the corresponding question for the above semantic expansion statement to determine the retrieval answer statement corresponding to the initial predicted answer statement.

[0055] The embodiments provided in this application improve the diversity of initial predicted answer sentences by supplementing sample data expansion, thereby reducing the negative impact of data sparsity on the pre-trained large language model and improving the utilization rate of semantically expanded sentences.

[0056] The preset retrieval-augmented generation model may be a Retrieval-Augmented Generation (RAG) model.

[0057] Here, the preset retrieval enhancement generation model can dynamically correct its own parameters based on feedback information such as user feedback collected during the actual training process, thereby more accurately reflecting the quality of the output of the pre-trained large language model.

[0058] Exemplarily, the semantic expansion sentence corresponding to the initial predicted answer sentence is determined in the following manner:

[0059] The initial predicted answer sentence is input into the preset retrieval enhancement generation model for data expansion, and a sample sentence database associated with the initial predicted answer sentence is determined; the sample sentence database is information-fused with the initial predicted answer sentence to determine the semantic expansion sentence corresponding to the initial predicted answer sentence.

[0060] In the above, the embodiment provided by the present application introduces a database of different external data sources, and determines a sample sentence database associated with the initial predicted answer sentence from the external database as the data expanded by the preset retrieval enhancement generation model, and then fuses the above-mentioned expanded sample sentence database with the initial predicted answer sentence to generate a semantically expanded sentence that matches the initial predicted answer sentence, which is used to enrich the training data, so that the pre-trained large language model can use more comprehensive contextual information when making decisions, thereby improving the large language model's understanding and decision-making capabilities for complex tasks.

[0061] It should be noted that the process of information fusion between the sample sentence database and the initial predicted answer sentence can be expressed by the following formula:

[0062] s′=s⊕k;

[0063] Among them, s′ is used to represent the semantic expansion sentence; s is used to represent the initial predicted answer sentence; k is used to represent the sample sentence data of the sample sentence database; ⊕ is used to represent the information fusion operation.

[0064] The embodiment provided in this application performs semantic expansion retrieval on the initial predicted answer sentence through the RAG model.

[0065] S103: Score the retrieved answer sentence, and determine the target reward value of the initial predicted answer sentence based on the score result of the retrieved answer sentence.

[0066] In this step, after the retrieval answer statement is determined, the embodiment provided in this application will score the above-mentioned retrieval answer statement, determine the scoring result of the retrieval answer statement, and then determine the overall target reward value of the initial predicted answer statement based on the result.

[0067] It can be understood that the embodiments provided in this application perform reinforcement learning training on the pre-trained large model through adaptive reward values, and the setting of the target reward value in the embodiments provided in this application can determine the correctness of the grammar of the retrieval answer statement and judge whether the logic of the retrieval answer statement is smooth, so as to provide more detailed gradient feedback for the optimization and training of the pre-trained large language model.

[0068] For example, after determining the search answer statement, the embodiment provided by this application may further:

[0069] The reward value of the initial predicted answer sentence is calculated according to the preset format reward algorithm to determine the format reward value corresponding to the initial predicted answer sentence.

[0070] In this step, the embodiment provided by the present application can perform semantic expansion retrieval on the initial predicted answer statement and calculate the target reward value. At the same time, it can also calculate the reward value of the above-mentioned initial predicted answer statement according to a preset format reward algorithm, and use regular expressions to determine whether the format is correct. After determining that the format is correct, it determines the format reward value corresponding to the initial predicted answer statement.

[0071] It can be understood that the embodiment provided by the present application calculates a format reward value of 1 point for the initial predicted answer statement when it is determined that the format of the initial predicted answer statement is correct; when it is determined that the format of the initial predicted answer statement is incorrect, the reward value of the initial predicted answer statement is not calculated (or the reward value of the initial predicted answer statement is determined to be 0 this time).

[0072] Among them, the embodiment provided in this application can calculate the format reward value by adding a format tag representing the format type judgment in the initial predicted answer sentence.

[0073] For example, Figure 2 FIG. 1 shows a flow chart of determining a target reward value in a large language model training method provided in an embodiment of the present application. Figure 2 As shown, the steps to determine the target reward value are as follows:

[0074] S1031. Score the search answer statement to determine the score result of the search answer statement.

[0075] In this step, after determining the semantic expansion statement corresponding to the initial predicted answer statement according to the preset retrieval enhancement generation model, the embodiment provided by the present application will input the semantic expansion statement into the scoring module in the above-mentioned preset retrieval enhancement generation model to score the above-mentioned retrieval answer statement and determine the scoring result corresponding to the retrieval answer statement.

[0076] It should be noted that the embodiment provided in this application is: the retrieval answer statement is input into a preset teacher model (i.e., a scoring module) for scoring, the retrieval reward value corresponding to the retrieval answer statement is determined, and the retrieval reward value is determined as the scoring result of the retrieval answer statement.

[0077] It can be understood that the embodiment provided in the present application optimizes the scoring of the retrieval answer statement through the preset teacher model built in the preset retrieval enhancement generation model, determines the retrieval reward value corresponding to the retrieval answer statement, so as to ensure the quality of the retrieval answer statement and the consistency of the retrieval answer statement with the initial predicted answer statement.

[0078] S1032. Determine the target reward value of the initial predicted answer sentence based on the scoring result and the format reward value.

[0079] In this step, the embodiment provided in the present application performs weighted summation of the retrieval reward value and the format reward value according to a preset weight coefficient ratio to determine the target reward value of the initial predicted answer statement.

[0080] It should be noted that the embodiment provided in this application ensures the optimal target reward value obtained in different task scenarios through weighted summation.

[0081] Among them, the preset weight coefficient ratios in the embodiments provided in this application can be customized and used according to different application scenarios and task scenarios.

[0082] S104. If the target reward value is less than the preset reward threshold, the pre-trained large language model is adjusted based on the target reward value, and the question statement is input into the adjusted model again until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model.

[0083] In this step, after determining the target reward value of the initial predicted answer sentence, the embodiment provided by the present application needs to judge the training degree of the pre-trained large language model that has been iteratively trained once. The embodiment provided by the present application is to judge by comparing the target reward value with the preset reward threshold. When it is determined that the target reward value is less than the preset reward threshold, it is necessary to perform iterative training and deep learning adjustments on the pre-trained large language model. After multiple iterations or training, when it is determined that the final target reward value is greater than or equal to the preset reward threshold (or the final target reward value begins to converge), the pre-trained large language model is stopped, the training is completed, and the trained large language model is determined.

[0084] It can be understood that in the process of continuous training and iteration, the embodiments provided in the present application will continuously compare and separate the intermediate output predicted answer sentences and the original question sentences, and dynamically adjust the parameters of the above-mentioned pre-trained large language model through the target reward value, so that the target reward value is always close to the real question sentences issued by the user, thereby promoting the continuous improvement of the pre-trained large language model.

[0085] The large language model training method provided in the embodiments of the present application, compared with the prior art, the embodiments provided in the present application input the question sentence issued by the user into the pre-trained large language model, output the initial predicted answer sentence corresponding to the question sentence, and perform semantic expansion retrieval on the initial predicted answer sentence to determine the retrieval answer sentence corresponding to the initial predicted answer sentence; score the retrieval answer sentence, and then determine the target reward value of the initial predicted answer sentence based on the scoring result of the retrieval answer sentence, and when the target reward value is less than a preset reward threshold, adjust the pre-trained large language model based on the target reward value, and input the question sentence into the adjusted model again until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model. The present application can improve the optimization training efficiency of the large language model and the accuracy of the model output, and improve the stability of the large language model output and the convergence speed of the pre-trained large language model.

[0086] The large language model training method provided in the embodiment of the present application utilizes data enhancement, expansion, and retrieval technology driven by a preset retrieval enhancement generation model, dynamically introduces a database of external knowledge, and generates multi-dimensional reward value data through an adaptive reward value parameter adjustment method guided by a pre-trained large language model, thereby realizing the training and updating of the pre-trained large language model, optimizing the accuracy and efficiency of the training, and thus reducing the problems of data sparsity and inaccurate reward values ​​in traditional methods.

[0087] Figure 3 This is a structural block diagram of a large language model training device provided in an embodiment of the present application. Figure 3As shown, the large language model training device 300 includes:

[0088] The output module 310 is used to input the question sentence issued by the user into the pre-trained large language model and output the initial predicted answer sentence corresponding to the question sentence.

[0089] The first determination module 320 is configured to perform semantic expansion retrieval on the initial predicted answer sentence and determine a retrieved answer sentence corresponding to the initial predicted answer sentence.

[0090] The third determination module 330 is used to calculate the reward value of the initial predicted answer sentence according to a preset format reward algorithm to determine the format reward value corresponding to the initial predicted answer sentence.

[0091] The second determination module 340 is configured to score the retrieved answer sentence and determine a target reward value for the initial predicted answer sentence based on the score result of the retrieved answer sentence.

[0092] Adjustment module 350 is used to adjust the pre-trained large language model based on the target reward value if the target reward value is less than the preset reward threshold, and re-input the question statement into the adjusted model until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model.

[0093] Exemplarily, the first determining module 320 is specifically configured to:

[0094] The initial predicted answer sentence is input into the preset retrieval enhancement generation model, the semantic expansion of the initial predicted answer sentence is performed, and the semantic expansion sentence corresponding to the initial predicted answer sentence is determined.

[0095] Semantic retrieval is performed on the semantically expanded sentence to determine the retrieval answer sentence corresponding to the initial predicted answer sentence.

[0096] Exemplarily, the initial predicted answer sentence is input into a preset search enhancement generation model, semantic expansion of the initial predicted answer sentence is performed, and a semantic expansion sentence corresponding to the initial predicted answer sentence is determined, including:

[0097] The initial predicted answer sentence is input into the preset retrieval enhancement generation model for data expansion, and a sample sentence database associated with the initial predicted answer sentence is determined.

[0098] The sample sentence database is fused with the initial predicted answer sentence to determine the semantically expanded sentence corresponding to the initial predicted answer sentence.

[0099] Exemplarily, the second determining module 340 is specifically configured to:

[0100] Scoring the retrieval answer sentence to determine the scoring result of the retrieval answer sentence.

[0101] Based on the scoring results and the format reward value, the target reward value of the initial predicted answer sentence is determined.

[0102] Exemplarily, scoring the search answer statement and determining the scoring result of the search answer statement includes:

[0103] The retrieval answer statement is input into the preset teacher model for scoring, the retrieval reward value corresponding to the retrieval answer statement is determined, and the retrieval reward value is determined as the scoring result of the retrieval answer statement.

[0104] Exemplarily, determining a target reward value for the initial predicted answer sentence based on the scoring result and the format reward value includes:

[0105] The retrieval reward value and the format reward value are weighted and summed according to the preset weight coefficient ratio to determine the target reward value of the initial predicted answer sentence.

[0106] The large language model training device 300 provided in an embodiment of the present application, compared with the prior art, inputs a question sentence issued by a user into a pre-trained large language model, outputs an initial predicted answer sentence corresponding to the question sentence, and performs semantic expansion retrieval on the initial predicted answer sentence to determine the retrieval answer sentence corresponding to the initial predicted answer sentence; scores the retrieval answer sentence, and then determines the target reward value of the initial predicted answer sentence based on the scoring result of the retrieval answer sentence; when the target reward value is less than a preset reward threshold, the pre-trained large language model is adjusted based on the target reward value, and the question sentence is input into the adjusted model again until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model. The present application can improve the optimization training efficiency of the large language model and the accuracy of the model output, and improve the stability of the large language model output.

[0107] The large language model training device 300 provided in the embodiment of the present application utilizes data enhancement, expansion, and retrieval technology driven by a preset retrieval enhancement generation model to dynamically introduce a database of external knowledge, and generates multi-dimensional reward value data through an adaptive reward value parameter adjustment method guided by a pre-trained large language model, thereby realizing the training and updating of the pre-trained large language model, optimizing the accuracy and efficiency of the training, and thus reducing the problems of data sparsity and inaccurate reward values ​​in traditional methods.

[0108] Figure 4 1 is a schematic diagram showing the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, electronic device 400 includes a processor 410 , a memory 420 , and a bus 440 .

[0109] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 440. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the large language model training method in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0110] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the large language model training method in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0111] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0112] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0113] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-readable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] An embodiment of the present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of the large language model training method.

[0118] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0119] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0121] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0125] Although the preferred embodiments of this specification have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this specification.

[0126] Obviously, those skilled in the art may make various changes and modifications to this specification without departing from the spirit and scope of this specification. Thus, if such changes and modifications fall within the scope of the claims of this specification and their equivalents, this specification is intended to include such changes and modifications.

Claims

1. A large language model training method, characterized in that: The method comprises: Input the question sentence sent by the user into the pre-trained large language model, and output the initial predicted answer sentence corresponding to the question sentence; Performing semantic expansion retrieval on the initial predicted answer sentence to determine a retrieval answer sentence corresponding to the initial predicted answer sentence; Scoring the retrieved answer sentence, and determining a target reward value for the initial predicted answer sentence based on the scoring result of the retrieved answer sentence; If the target reward value is less than the preset reward threshold, the pre-trained large language model is adjusted based on the target reward value, and the question statement is re-input into the adjusted model until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model.

2. The large language model training method according to claim 1, characterized in that The performing semantic expansion retrieval on the initial predicted answer sentence to determine a retrieved answer sentence corresponding to the initial predicted answer sentence includes: Inputting the initial predicted answer sentence into a preset retrieval enhancement generation model, performing semantic expansion of the initial predicted answer sentence, and determining a semantic expansion sentence corresponding to the initial predicted answer sentence; A semantic search is performed on the semantically expanded sentence to determine a search answer sentence corresponding to the initial predicted answer sentence.

3. The large language model training method according to claim 2, characterized in that The step of inputting the initial predicted answer sentence into a preset search enhancement generation model, performing semantic expansion on the initial predicted answer sentence, and determining a semantic expansion sentence corresponding to the initial predicted answer sentence includes: Inputting the initial predicted answer sentence into a preset retrieval enhancement generation model for data expansion, and determining a sample sentence database associated with the initial predicted answer sentence; The sample sentence database is subjected to information fusion with the initial predicted answer sentence to determine a semantically expanded sentence corresponding to the initial predicted answer sentence.

4. The large language model training method according to claim 1, characterized in that After performing semantic expansion retrieval on the initial predicted answer sentence and determining a retrieved answer sentence corresponding to the initial predicted answer sentence, the method further includes: The reward value of the initial predicted answer sentence is calculated according to a preset format reward algorithm to determine the format reward value corresponding to the initial predicted answer sentence.

5. The large language model training method according to claim 4, characterized in that Scoring the retrieved answer sentence and determining a target reward value for the initial predicted answer sentence based on the scoring result of the retrieved answer sentence includes: Scoring the search answer statement to determine a scoring result of the search answer statement; Based on the scoring result and the format reward value, a target reward value of the initial predicted answer sentence is determined.

6. The large language model training method according to claim 5, characterized in that: Scoring the search answer statement and determining a scoring result of the search answer statement includes: The retrieval answer statement is input into a preset teacher model for scoring, a retrieval reward value corresponding to the retrieval answer statement is determined, and the retrieval reward value is determined as a scoring result of the retrieval answer statement.

7. The large language model training method according to claim 6, characterized in that: The determining, based on the scoring result and the format reward value, a target reward value for the initial predicted answer sentence includes: The retrieval reward value and the format reward value are weighted and summed according to a preset weight coefficient ratio to determine the target reward value of the initial predicted answer sentence.

8. A large language model training device, characterized in that: The large language model training device includes: An output module, configured to input a question sentence sent by a user into a pre-trained large language model and output an initial predicted answer sentence corresponding to the question sentence; A first determining module is configured to perform semantic expansion retrieval on the initial predicted answer sentence to determine a retrieved answer sentence corresponding to the initial predicted answer sentence; a second determination module, configured to score the retrieved answer statement and determine a target reward value for the initial predicted answer statement based on the score result of the retrieved answer statement; An adjustment module is used to adjust the pre-trained large language model based on the target reward value if the target reward value is less than a preset reward threshold, and re-input the question statement into the adjusted model until the final target reward value is greater than or equal to the preset reward threshold, thereby completing the training of the pre-trained large language model.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to perform the steps of the large language model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the large language model training method according to any one of claims 1 to 7.

Images