Model training method and device, equipment, storage medium and program product
By acquiring and strengthening the CoT block of the thinking chain of the large language model, the problem of insufficient thinking and reasoning ability on specific tasks is solved, and the accuracy of the reasoning results and logical reasoning ability are improved.
Patent Information
- Application Number
- CN202411270502.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2025-01-03
AI Technical Summary
The existing large language models lack thinking and reasoning ability on specific tasks, resulting in low accuracy of reasoning results.
By obtaining multiple thinking chain CoT blocks of a large language model, a collection of CoT instructions is obtained based on the CoT block of each task, including CoT tags used to indicate the inference process, the large language model is trained to strengthen its logic on a specific task Reasoning ability.
By discovering and strengthening CoT blocks related to tasks in large language models, their inference accuracy on specific tasks is improved and logical reasoning capabilities are enhanced.
Smart Images
Figure CN119250153A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing technology, and in particular to a model training method, device, equipment, storage medium and program product. Background Art
[0002] At present, the construction of a large language model mainly goes through three stages: Stage 1: the unsupervised pre-training stage, which mainly establishes the large language model's ability to remember, understand, and reason about implicit scenario tasks in the corpus; Stage 2: the supervised instruction fine-tuning stage, which mainly strengthens the large language model's problem-solving ideas and logic for specific tasks; Stage 3: the supervised value alignment stage, which mainly solves the problem of aligning the large language model's output to human preferences.
[0003] Currently, due to the insufficient thinking and reasoning ability of large language models in specific tasks, the accuracy of the results obtained by reasoning using large language models is not high. Summary of the invention
[0004] The purpose of the present invention is to provide a model training method, device, equipment, storage medium and program product to solve the problem of low accuracy of the results obtained by reasoning using the existing large language model.
[0005] In order to achieve the above object, in a first aspect, the present invention provides a model training method, comprising:
[0006] Obtain multiple thought chain CoT blocks of the large language model. Different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task;
[0007] Based on the CoT block corresponding to each task, a CoT instruction set of each task is obtained, where the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process;
[0008] Based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model.
[0009] In some embodiments, the step of obtaining a plurality of thought chain CoT blocks of a large language model includes:
[0010] For each of the multiple tasks, obtaining a basic instruction set of the task;
[0011] For each instruction in the basic instruction set, input the instruction into the large language model, obtain the word unit token output by each layer in the large language model, and visualize and display it;
[0012] Based on the visualized tokens, obtain the blocks in the large language model that are strongly related to the task;
[0013] According to the obtained multiple blocks strongly related to the task in the large language model, a CoT block corresponding to the task in the large language model is determined.
[0014] In some embodiments, obtaining a basic instruction set for the task includes:
[0015] Acquire a pre-built first seed instruction, wherein the first seed instruction includes a first preset text related to the task, a first preset instruction for the first preset text, and a format for outputting the first preset instruction;
[0016] The first seed instructions are learned to generate a basic instruction set for the task.
[0017] In some embodiments, acquiring the CoT instruction set of each task based on the CoT block corresponding to each task includes:
[0018] For each of the multiple tasks, obtain a pre-constructed second seed instruction, where the second seed instruction includes a second preset text related to the task, a second preset instruction for the second preset text, a format for an output of the second preset instruction, and a CoT tag for indicating a reasoning process, where the reasoning process is determined based on a token output by a CoT block corresponding to the task;
[0019] The second seed instruction is learned to generate a CoT instruction set of the task.
[0020] In some embodiments, the CoT instruction set based on all tasks, training the large language model to obtain the trained large language model, includes:
[0021] The following steps are adopted to traverse each instruction in the CoT instruction set of all tasks, train the large language model, and obtain the trained large language model:
[0022] Input the instruction into the large language model, obtain the token output by each layer in the large language model, and calculate the loss value corresponding to the token output by each layer;
[0023] Based on the loss value corresponding to the token output by each layer, calculate the loss value of all layers in the large language model;
[0024] Based on the loss values of all the layers, the network parameters of the large language model are adjusted to obtain an adjusted large language model, and the large language model is updated to the adjusted large language model.
[0025] In some embodiments, adjusting the network parameters of the large language model based on the loss values of all layers to obtain the adjusted large language model includes:
[0026] For the first instruction, when the specified layer in the large language model does not output a token that conforms to the reasoning process, the parameter weight of the specified layer is adjusted based on the loss values of all the layers to obtain the adjusted large language model; wherein the specified layer is at least one layer in the CoT block corresponding to the task to which the first instruction belongs, and the first instruction is the current instruction.
[0027] In a second aspect, the present invention further provides a model training device, comprising:
[0028] The first acquisition module is used to obtain multiple thought chain CoT blocks of the large language model. Different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task;
[0029] A second acquisition module, configured to acquire a CoT instruction set for each task based on the CoT block corresponding to each task, wherein the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process;
[0030] The model training module is used to train the large language model based on the CoT instruction set of all tasks to obtain the trained large language model.
[0031] In a third aspect, the present invention further provides a model training device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, and the processor is used to perform the following operations:
[0032] Obtain multiple thought chain CoT blocks of the large language model. Different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task;
[0033] Based on the CoT block corresponding to each task, a CoT instruction set of each task is obtained, where the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process;
[0034] Based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model.
[0035] In a fourth aspect, the present invention also provides a model training device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, the model training method as described in the first aspect above is implemented.
[0036] In a fifth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the model training method described in the first aspect above.
[0037] In a sixth aspect, the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the model training method described in the first aspect above.
[0038] The above technical solution of the present invention has at least the following beneficial effects:
[0039] In an embodiment of the present invention, by obtaining multiple thinking chain CoT blocks of a large language model, different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task; then, based on the CoT block corresponding to each task, a CoT instruction set for each task is obtained, and the instructions in the CoT instruction set include a CoT label for indicating the reasoning process; finally, based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model. In this way, by discovering different CoT blocks corresponding to different tasks in the large language model, the CoT block is strongly related to the task, reflecting the logical reasoning ability of the large language model on the task, and its reasoning process has a certain logic from abstract to concrete; then, the CoT instruction set of the task is constructed based on the CoT block, and the CoT block in the large language model is strengthened. The CoT ability of the trained large language model is strengthened. In subsequent applications, the large language model trained by the present invention is used to reason on specific tasks, which can improve the accuracy of the obtained results. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 A schematic diagram showing a flow chart of a model training method provided by an embodiment of the present invention;
[0041] Figure 2 A schematic diagram showing a module of a model training device provided by an embodiment of the present invention;
[0042] Figure 3 A schematic diagram showing the hardware structure of a model training device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0044] In order to make the large language model have stronger thinking and reasoning ability on specific tasks, it is necessary to strengthen the Chain-of-Thought (CoT) ability of the large language model. At present, the main means of strengthening is to construct an instruction set for a specific task, and the label of the instruction set contains the CoT process of the task, and let the large language model learn the reasoning logic of the task end to end through the supervised learning mode. This method learns through the instruction task with CoT, and the learning process is a black box operation. Whether it can be learned and how much it can be learned are all information in the black box. That is, the task CoT learning process of the large language model is uncontrollable and unexplainable, so there will be insufficient thinking and reasoning ability of the large language model on specific tasks, which will lead to the problem that the accuracy of the results obtained by reasoning using the large language model is not high.
[0045] In order to solve the above technical problems, a model training method, device, equipment, storage medium and program product are provided. The method and the device are based on the same application concept. Since the principles of solving the problem by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.
[0046] like Figure 1 FIG. 1 is a flow chart of a model training method provided by an embodiment of the present invention, and the method may include:
[0047] Step 101, obtain multiple thought chain CoT blocks of the large language model, different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task.
[0048] In an optional embodiment, step 101 includes:
[0049] Step 1011: for each of the multiple tasks, obtain a basic instruction set of the task.
[0050] Here, the task involved in the present invention is a specific task, and its specific meaning is to group together knowledge with the same human cognition, such as named entity recognition (NER), relation extraction (RE), event extraction (EE), etc.
[0051] Specifically, the basic instruction set of the task is obtained, including:
[0052] 1) Obtaining a pre-built first seed instruction, wherein the first seed instruction includes a first preset text related to the task, a first preset instruction for the first preset text, and an output format for the first preset instruction.
[0053] It should be noted that the first sub-instruction may be an artificially constructed instruction to ensure the rigor of the task. Generally, there are multiple first sub-instructions.
[0054] In one example, the task is a NER task, and the first preset text is: Guangzhou's weather is really hot. The first preset instruction for the first preset text is: Please extract entities in the text. The output format of the first preset instruction is JSON format, that is, {'entity': [entity name]}.
[0055] 2) Learning the first seed instructions to generate a basic instruction set for the task.
[0056] The first seed instructions can be learned through a larger model with stronger capabilities (such as GPT4o) to generate a large number of instructions as the basic instruction set for the task.
[0057] It should be noted that, in order to ensure the rigor of the task, the instructions generated in step 2) may be manually reviewed, that is, after a large number of instructions are generated, they are manually reviewed to obtain a basic instruction set for the task.
[0058] Step 1012: for each instruction in the basic instruction set, input the instruction into the large language model, obtain the word-unit token output by each layer in the large language model, and visualize it.
[0059] For example, the instruction (prompt) input to the large language model is: Please extract entities from the text. \nText: The weather in Shanghai is really hot. \nPlease output the JSON format {'entity': [entity name]}. After that, the word unit token output by each layer in the large language model is visualized, as shown in Table 1 below:
[0060] Table 1
[0061]
[0062]
[0063] Step 1013: Based on the visualized tokens, obtain the blocks in the large language model that are strongly related to the task.
[0064] Here, experts can analyze the visualized tokens to identify and judge the blocks in the large language model that are strongly related to the task. This block can be defined as a strong capability block for the task. For example, in the above example, when the token of "Shanghai" is inferred, "things" is output at the 17th layer, "entities" are output at the 18th to 26th layers, "places" are output at the 27th to 29th layers, and "Shanghai" is output at the 30th to 40th layers. The reasoning process has a certain logic from abstract to concrete, so the 17th to 40th layers are defined as blocks that are strongly related to the task of this case (instruction). It should be noted that each instruction in the basic instruction set of the task is processed according to the above steps to obtain multiple blocks in the large language model that are strongly related to the task.
[0065] Step 1014: Determine a CoT block in the large language model corresponding to the task based on the obtained multiple blocks in the large language model that are strongly related to the task.
[0066] Among them, a common block can be determined based on a plurality of blocks strongly related to the task in the obtained large language model, and the common block can be determined as the CoT block corresponding to the task in the large language model.
[0067] The CoT block is strongly related to the task, reflecting the logical reasoning ability of the large language model on the task. Its reasoning process has a certain logic from abstract to concrete.
[0068] Step 102, based on the CoT block corresponding to each task, obtaining a CoT instruction set for each task, where the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process;
[0069] It should be noted that the instruction includes a CoT tag for indicating the reasoning process which is determined based on the CoT block corresponding to the task.
[0070] Step 103: train the large language model based on the CoT instruction set of all tasks to obtain a trained large language model.
[0071] The model training method of the present invention discovers different CoT blocks corresponding to different tasks in a large language model. The CoT blocks are strongly related to the tasks, reflecting the logical reasoning ability of the large language model on the tasks, and its reasoning process has a certain logic from abstract to concrete. Then, a CoT instruction set for the task is constructed based on the CoT blocks, and the CoT blocks in the large language model are subjected to intensive training. The CoT ability of the trained large language model is enhanced. In subsequent applications, the large language model trained by the present invention is used to perform reasoning on specific tasks, which can improve the accuracy of the obtained results.
[0072] In an optional embodiment, the above step 102, based on the CoT block corresponding to each task, obtains the CoT instruction set of each task, including:
[0073] Step 1021, for each of the multiple tasks, obtain a pre-constructed second seed instruction, where the second seed instruction includes a second preset text related to the task, a second preset instruction for the second preset text, a format for an output of the second preset instruction, and a CoT tag for indicating a reasoning process, where the reasoning process is determined based on a token output by the CoT block corresponding to the task;
[0074] It should be noted that the second sub-instruction may be an artificially constructed instruction to ensure the rigor of the task. Generally, there are multiple second sub-instructions.
[0075] In one example, the task is a NER task, and the second preset text is: The weather in Beijing is really hot. The second preset instruction for the second preset text is: Please extract the entities in the text. The output format for the second preset instruction is JSON format, that is, {'entity': [entity name]}. The CoT tag used to indicate the reasoning process is: the output of the second preset instruction is obtained according to the reasoning process indicated by the CoT tag, and the specific CoT tag is: {CoT: First, find the semantic space of "things", then convert from the semantic space of "things" to the semantic space of "entity", and then convert from the semantic space of "entity" to the semantic space of "place", and finally find the entity from the semantic space of "place"}. It can be seen that the CoT label is determined based on the token output by the CoT block; in this example, according to the reasoning process indicated by the CoT label, the entity "Beijing" is finally found from the semantic space of "place", entity: Beijing.
[0076] Step 1022: learn the second seed instruction to generate a CoT instruction set for the task.
[0077] A more powerful large model (such as GPT4o) can be used to learn the second seed instructions and generate a large number of instructions as the CoT instruction set for the task.
[0078] It should be noted that, in order to ensure the rigor of the task, the instructions generated in step 1022 may be manually reviewed, that is, after a large number of instructions are generated, they are manually reviewed to obtain the CoT instruction set of the task.
[0079] If the task is a NER task, case 1 is "The weather in Beijing is so hot!" and case 2 is "The weather in Shanghai is so hot!" In these two cases, the entities that can be extracted are "Beijing" and "Shanghai" respectively. The tokens output by the reasoning process of the two cases need to be consistent. The CoT instruction set that needs to be constructed is:
[0080] The instructions for case 1 are:
[0081] Input: Please extract entities from the following text. Text: The weather in Beijing is really hot! \nPlease output in JSON format.
[0082] output: CoT label: {CoT: First, find the semantic space of "things", then convert from the semantic space of "things" to the semantic space of "entity", then convert from the semantic space of "entity" to the semantic space of "place", and finally find the entity "Beijing" from the semantic space of "place", entity: Beijing}
[0083] The instructions for case 1 are:
[0084] Input: Please extract entities from the following text. Text: The weather in Shanghai is really hot! \nPlease output in JSON format.
[0085] output: CoT label: {CoT: First, we need to find the semantic space of "things", then convert from the semantic space of "things" to the semantic space of "entity", then convert from the semantic space of "entity" to the semantic space of "place", and finally find the entity "Shanghai" from the semantic space of "place", entity: Shanghai}
[0086] In an optional embodiment, the above step 103, based on the CoT instruction set of all tasks, trains the large language model to obtain the trained large language model, including:
[0087] The following steps are adopted to traverse each instruction in the CoT instruction set of all tasks, train the large language model, and obtain the trained large language model:
[0088] ① Input the instruction into the large language model, obtain the token output by each layer in the large language model, and calculate the loss value corresponding to the token output by each layer;
[0089] ② Based on the loss value corresponding to the token output by each layer, calculate the loss value of all layers in the large language model;
[0090] ③ Based on the loss values of all the layers, adjust the network parameters of the large language model to obtain an adjusted large language model, and update the large language model to the adjusted large language model.
[0091] It should be noted that steps ① to ③ are steps that must be executed for each instruction. For example, after instruction A executes steps ① to ③, it takes the next instruction B of instruction A and continues to execute ① to ③. And so on. After all the instructions in the CoT instruction set of all tasks are executed, the trained large language model is obtained.
[0092] Optionally, adjusting the network parameters of the large language model based on the loss values of all layers to obtain an adjusted large language model includes:
[0093] For the first instruction, when the specified layer in the large language model does not output a token that conforms to the reasoning process, the parameter weight of the specified layer is adjusted based on the loss values of all the layers to obtain the adjusted large language model; wherein the specified layer is at least one layer in the CoT block corresponding to the task to which the first instruction belongs, and the first instruction is the current instruction.
[0094] The present invention aggregates and calculates the loss of the model training process. For example, according to the example in Table 1 above:
[0095] Output "stuff" on layer 17
[0096] Output "entity" on layers 18 to 26
[0097] Output "location" on the 27th to 29th floors
[0098] Output "Shanghai" on the 30th to 40th floor
[0099] If the specified layer does not output the corresponding "token", the loss of this layer is aggregated into the loss passed down from the last layer (the lowest layer). For example, if the output of the last layer is "Shanghai", the loss of the top layer is 0. If the token output at the 17th layer is not "Dongxi", loss will be generated at the 17th layer, so the loss will change from 0 to the loss theoretically generated by this layer (the calculation logic is the same as the last layer). Similarly, the losses of all layers are calculated and aggregated, and then the parameter weights of the specified layer are optimized in the reverse gradient.
[0100] In this way, according to the above loss calculation logic, supervised learning is performed on the CoT instruction set of all tasks, so as to fine-tune and optimize the parameters of the specified block and explicitly inject the CoT capability of the task into the large language model.
[0101] It should be pointed out that the trained large language model can be used in specific tasks to strengthen the problem-solving ideas and logic of specific tasks, thereby improving the accuracy of the results obtained. The specific tasks of the present invention have a specific meaning of gathering knowledge with the same human cognition together, such as named entity recognition (NER), relation extraction (RE), event extraction (EE), etc.
[0102] like Figure 2 As shown, an embodiment of the present invention also provides a model training device, which includes:
[0103] The first acquisition module 201 is used to acquire multiple thought chain CoT blocks of the large language model, where different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task;
[0104] A second acquisition module 202, configured to acquire a CoT instruction set for each task based on the CoT block corresponding to each task, wherein the instructions in the CoT instruction set include a CoT tag for indicating an inference process;
[0105] The model training module 203 is used to train the large language model based on the CoT instruction set of all tasks to obtain a trained large language model.
[0106] Optionally, the first acquisition module 201 includes:
[0107] A first acquisition unit, configured to acquire, for each of the plurality of tasks, a basic instruction set of the task;
[0108] A first processing module is used for inputting each instruction in the basic instruction set into the large language model, obtaining a word unit token outputted by each layer in the large language model, and visually displaying the word unit token;
[0109] A second acquisition unit, configured to obtain a block in the large language model that is strongly related to the task based on the visually displayed token;
[0110] The second processing unit is used to determine the CoT block corresponding to the task in the large language model according to the obtained multiple blocks strongly related to the task in the large language model.
[0111] Optionally, the first acquiring unit is specifically used for:
[0112] Acquire a pre-built first seed instruction, wherein the first seed instruction includes a first preset text related to the task, a first preset instruction for the first preset text, and a format for outputting the first preset instruction;
[0113] The first seed instructions are learned to generate a basic instruction set for the task.
[0114] Optionally, the second acquisition module 202 includes:
[0115] a third acquisition unit, configured to acquire, for each of the plurality of tasks, a pre-constructed second seed instruction, wherein the second seed instruction includes a second preset text related to the task, a second preset instruction for the second preset text, a format for an output of the second preset instruction, and a CoT tag for indicating a reasoning process, wherein the reasoning process is determined based on a token output by the CoT block corresponding to the task;
[0116] The third processing unit is used to learn the second seed instruction and generate a CoT instruction set for the task.
[0117] Optionally, the model training module 203 includes:
[0118] The model training unit is used to traverse each instruction in the CoT instruction set of all tasks, train the large language model, and obtain the trained large language model by taking the following steps:
[0119] Input the instruction into the large language model, obtain the token output by each layer in the large language model, and calculate the loss value corresponding to the token output by each layer;
[0120] Based on the loss value corresponding to the token output by each layer, calculate the loss value of all layers in the large language model;
[0121] Based on the loss values of all the layers, the network parameters of the large language model are adjusted to obtain an adjusted large language model, and the large language model is updated to the adjusted large language model.
[0122] Optionally, the model training unit is specifically used for:
[0123] For the first instruction, when the specified layer in the large language model does not output a token that conforms to the reasoning process, the parameter weight of the specified layer is adjusted based on the loss values of all the layers to obtain the adjusted large language model; wherein the specified layer is at least one layer in the CoT block corresponding to the task to which the first instruction belongs, and the first instruction is the current instruction.
[0124] The model training device of the embodiment of the present invention obtains multiple thinking chain CoT blocks of the large language model, different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task; then, based on the CoT block corresponding to each task, a CoT instruction set for each task is obtained, and the instructions in the CoT instruction set include a CoT label for indicating the reasoning process; finally, based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model, so that by discovering different CoT blocks corresponding to different tasks in the large language model, the CoT block is strongly related to the task, reflecting the logical reasoning ability of the large language model on the task, and its reasoning process has a certain abstract to concrete logic; then, based on the CoT block to build the CoT instruction set of the task, the CoT block in the large language model is strengthened and trained, and the CoT ability of the trained large language model is strengthened. In subsequent applications, the large language model trained by the present invention is used to reason on specific tasks, which can improve the accuracy of the obtained results.
[0125] In order to better achieve the above goals, Figure 3 As shown, an embodiment of the present invention further provides a model training device, including a processor 300 and a transceiver 310, wherein the transceiver 310 receives and sends data under the control of the processor 300, and the processor 300 is used to perform the following process:
[0126] Obtain multiple thought chain CoT blocks of the large language model. Different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task;
[0127] Based on the CoT block corresponding to each task, a CoT instruction set of each task is obtained, where the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process;
[0128] Based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model.
[0129] Optionally, the processor 300 is further configured to:
[0130] For each of the multiple tasks, obtaining a basic instruction set of the task;
[0131] For each instruction in the basic instruction set, input the instruction into the large language model, obtain the word unit token output by each layer in the large language model, and visualize and display it;
[0132] Based on the visualized tokens, obtain the blocks in the large language model that are strongly related to the task;
[0133] According to the obtained multiple blocks strongly related to the task in the large language model, a CoT block corresponding to the task in the large language model is determined.
[0134] Optionally, the processor 300 is further configured to:
[0135] Acquire a pre-built first seed instruction, wherein the first seed instruction includes a first preset text related to the task, a first preset instruction for the first preset text, and a format for outputting the first preset instruction;
[0136] The first seed instructions are learned to generate a basic instruction set for the task.
[0137] Optionally, the processor 300 is further configured to:
[0138] For each of the multiple tasks, obtain a pre-constructed second seed instruction, where the second seed instruction includes a second preset text related to the task, a second preset instruction for the second preset text, a format for an output of the second preset instruction, and a CoT tag for indicating a reasoning process, where the reasoning process is determined based on a token output by a CoT block corresponding to the task;
[0139] The second seed instruction is learned to generate a CoT instruction set of the task.
[0140] Optionally, the processor 300 is further configured to:
[0141] The following steps are adopted to traverse each instruction in the CoT instruction set of all tasks, train the large language model, and obtain the trained large language model:
[0142] Input the instruction into the large language model, obtain the token output by each layer in the large language model, and calculate the loss value corresponding to the token output by each layer;
[0143] Based on the loss value corresponding to the token output by each layer, calculate the loss value of all layers in the large language model;
[0144] Based on the loss values of all the layers, the network parameters of the large language model are adjusted to obtain an adjusted large language model, and the large language model is updated to the adjusted large language model.
[0145] Optionally, the processor 300 is further configured to:
[0146] For the first instruction, when the specified layer in the large language model does not output a token that conforms to the reasoning process, the parameter weight of the specified layer is adjusted based on the loss values of all the layers to obtain the adjusted large language model; wherein the specified layer is at least one layer in the CoT block corresponding to the task to which the first instruction belongs, and the first instruction is the current instruction.
[0147] The model training device of the embodiment of the present invention obtains multiple thinking chain CoT blocks of the large language model, different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task; then, based on the CoT block corresponding to each task, a CoT instruction set for each task is obtained, and the instructions in the CoT instruction set include a CoT label for indicating the reasoning process; finally, based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model. In this way, by discovering different CoT blocks corresponding to different tasks in the large language model, the CoT block is strongly related to the task, reflecting the logical reasoning ability of the large language model on the task, and its reasoning process has a certain abstract to concrete logic; then, based on the CoT block to build the CoT instruction set of the task, the CoT block in the large language model is strengthened and trained, and the CoT ability of the trained large language model is strengthened. In subsequent applications, the large language model trained by the present invention is used to reason on specific tasks, which can improve the accuracy of the obtained results.
[0148] An embodiment of the present invention also provides a model training device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the various processes in the model training method embodiment described above are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.
[0149] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0150] An embodiment of the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the model training method as described above.
[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A device that specifies functions in one or more processes and / or one or more blocks.
[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable device executes a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0155] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A model training method, characterized in that: include: Obtain multiple thought chain CoT blocks of the large language model. Different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task; Based on the CoT block corresponding to each task, obtaining a CoT instruction set for each task, where the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process; Based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model.
2. The method according to claim 1, characterized in that The method of obtaining multiple thought chain CoT blocks of the large language model includes: For each of the multiple tasks, obtaining a basic instruction set of the task; For each instruction in the basic instruction set, input the instruction into the large language model, obtain the word unit token output by each layer in the large language model, and visualize and display it; Based on the visualized tokens, obtain the blocks in the large language model that are strongly related to the task; According to the obtained multiple blocks in the large language model that are strongly related to the task, a CoT block in the large language model corresponding to the task is determined.
3. The method according to claim 2, characterized in that Get the basic instruction set of the task, including: Acquire a pre-built first seed instruction, wherein the first seed instruction includes a first preset text related to the task, a first preset instruction for the first preset text, and a format for outputting the first preset instruction; The first seed instructions are learned to generate a basic instruction set for the task.
4. The method according to claim 1, characterized in that: The obtaining of a CoT instruction set for each task based on the CoT block corresponding to each task includes: For each of the multiple tasks, obtain a pre-constructed second seed instruction, where the second seed instruction includes a second preset text related to the task, a second preset instruction for the second preset text, a format for an output of the second preset instruction, and a CoT tag for indicating a reasoning process, where the reasoning process is determined based on a token output by a CoT block corresponding to the task; The second seed instruction is learned to generate a CoT instruction set of the task.
5. The method according to claim 1, characterized in that The CoT instruction set based on all tasks, training the large language model to obtain the trained large language model, includes: The following steps are adopted to traverse each instruction in the CoT instruction set of all tasks, train the large language model, and obtain the trained large language model: Input the instruction into the large language model, obtain the token output by each layer in the large language model, and calculate the loss value corresponding to the token output by each layer; Based on the loss value corresponding to the token output by each layer, calculate the loss value of all layers in the large language model; Based on the loss values of all the layers, the network parameters of the large language model are adjusted to obtain an adjusted large language model, and the large language model is updated to the adjusted large language model.
6. The method according to claim 5, characterized in that The adjusting the network parameters of the large language model based on the loss values of all the layers to obtain the adjusted large language model includes: For the first instruction, when the specified layer in the large language model does not output a token that conforms to the reasoning process, the parameter weight of the specified layer is adjusted based on the loss values of all the layers to obtain the adjusted large language model; wherein the specified layer is at least one layer in the CoT block corresponding to the task to which the first instruction belongs, and the first instruction is the current instruction.
7. A model training device, characterized in that: include: The first acquisition module is used to obtain multiple thought chain CoT blocks of the large language model. Different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task; A second acquisition module, configured to acquire a CoT instruction set for each task based on the CoT block corresponding to each task, wherein the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process; The model training module is used to train the large language model based on the CoT instruction set of all tasks to obtain the trained large language model.
8. A model training device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, characterized in that: The processor is configured to perform the following operations: Obtain multiple thought chain CoT blocks of the large language model. Different CoT blocks correspond to different tasks, and each CoT block is strongly related to the corresponding task; Based on the CoT block corresponding to each task, obtaining a CoT instruction set for each task, where the instructions in the CoT instruction set include a CoT tag for indicating a reasoning process; Based on the CoT instruction set of all tasks, the large language model is trained to obtain a trained large language model.
9. A model training device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the model training method as described in any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the model training method as described in any one of claims 1 to 6 are implemented.
11. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps in the model training method as described in any one of claims 1 to 6.
Citation Information
Cited By
Multi-task large language model training method oriented to biomedical field
CN120069086A