Large language model training method and device based on reinforcement learning and electronic equipment
By combining reinforcement learning algorithms and process reward models to evaluate the divergence of large language models, we encourage them to generate diverse outputs in creative design and inspiration search scenarios. This solves the problem of suppressing the divergence ability of models in scenarios without a single standard answer, and achieves logically consistent and creatively rich outputs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 特赞(上海)信息科技有限公司
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-05
AI Technical Summary
Existing large language models have their divergent thinking ability suppressed in creative design and inspiration search scenarios where there is no single standard answer, resulting in the model becoming more and more rigid as it is trained.
We employ a reinforcement learning-based approach, which uses training data and combines a proximal policy optimization algorithm and a process reward model to evaluate the divergence of each step in the thought chain and provide corresponding rewards. The goal is to train a large language model with the aim of maximizing cumulative rewards, thereby encouraging it to make leaps and generate diverse content in the semantic space.
It enables large language models to output highly creative and logically consistent results in creative design and inspiration search scenarios, and solves the problem of suppressing the divergence ability of the model in scenarios without a unique standard answer.
Smart Images
Figure CN121981195A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a method, apparatus, and electronic device for training large language models based on reinforcement learning. Background Technology
[0002] Currently, large language models (such as DeepSeek-R1 and OpenAI o1) excel in code generation and mathematical problem-solving. This superior performance is primarily attributed to reinforcement learning on datasets with definite correct answers. Reinforcement learning training for large language models mainly targets mathematical, programming, and natural science scenarios, which are characterized by convergence. Problems typically have only one correct answer, and the training objective is to make the large language model converge to this unique solution. However, if this training method is applied to creative design and inspiration search scenarios without a single correct answer, it inhibits the divergent thinking ability of the large language model, causing it to become increasingly rigid with further training.
[0003] There is currently no effective technical solution to the problem of suppressing the divergence ability of large language models when applied to creative design and inspiration search scenarios where there is no single correct answer. Summary of the Invention
[0004] The main purpose of this disclosure is to provide a method, apparatus, and electronic device for training large language models based on reinforcement learning, in order to solve the problem that the training methods for large language models in related technologies suppress the divergence ability of large language models when applied to creative design and inspiration search scenarios without a single standard answer.
[0005] To achieve the above objectives, the first aspect of this disclosure provides a method for training a large language model based on reinforcement learning, applicable to creative design and inspiration search scenarios, including: Obtain training data for a large language model, wherein the training data includes input data, thought chain, and output data of the large language model, and the thought chain includes all thought steps from input data to output data; By combining reinforcement learning algorithms and process reward models, the divergence of divergent association words generated by each thinking step in the thought chain is evaluated, and rewards are given to the corresponding thinking steps based on the divergence of the divergent association words. The reinforcement learning algorithm includes a proximal policy optimization algorithm. The rewards of all thinking steps in the thought chain are summed to obtain a cumulative reward. The large language model is trained with the goal of maximizing the cumulative reward, resulting in a trained large language model.
[0006] Optionally, the input data is the original design requirements, and the output data is the last round of divergent association words obtained from the input data through multiple rounds of semantic leaps. Specifically, the method of combining reinforcement learning algorithms and process reward models to evaluate the divergence of divergent associations generated at each thought step in the thought chain, and to reward the corresponding thought steps based on the divergence of the divergent associations, includes: Based on the original design requirements and the logical relationship between the multiple rounds of divergent association words, all words are divided into parent node words and corresponding child node words. Among them, all words include the original design requirements and the multiple rounds of divergent association words. Using the Embedding model, the vocabulary of parent nodes and child nodes is vectorized; By combining the aforementioned proximal policy optimization algorithm and the semantic distance-based process reward model, the degree of hopping between parent node words and child node words, as well as the degree of diversity among all child node words under the same parent node word, are evaluated. Based on the degree of leap between parent node vocabulary and child node vocabulary, vertical rewards are given to the corresponding thinking steps; Based on the degree of diversity among all child nodes under the same parent node, lateral rewards are given to the corresponding thought processes.
[0007] Furthermore, the vertical reward given to the corresponding thought process based on the degree of leap between the parent node vocabulary and the corresponding child node vocabulary includes: Determine the cosine distance between the vectorized parent node vocabulary and the child node vocabulary; Vertical rewards are given to the corresponding thought steps based on the cosine distance, wherein the larger the cosine distance, the larger the vertical reward.
[0008] Furthermore, the method of providing lateral rewards for corresponding thought steps based on the diversity among all child node words under the same parent node word includes: Determine the average distance between all child node words under the same parent node word after vectorization; Lateral rewards are given to the corresponding thought steps based on the average distance, wherein the greater the average distance, the greater the lateral reward.
[0009] Optionally, after obtaining the trained large language model, the method further includes: The trained large language model is used to receive user input design requirements and output creative results. Based on the semantic distance between the design requirements and the creative results, the divergence of the creative results relative to the design requirements is evaluated.
[0010] Furthermore, the step of using a trained large language model to receive user input design requirements and output creative results includes: The design requirement is to use a trained large language model to receive user input. Based on the aforementioned design requirements, multiple semantic leaps are performed to generate multiple rounds of divergent associative words; The last round of divergent associations from the multi-round divergent associations is output as the creative result.
[0011] Furthermore, the assessment of the divergence of the creative result relative to the design requirement based on the semantic distance between the design requirement and the creative result includes: Using the Embedding model, design requirements are mapped to vectors. Map the i-th creative word in the creative results to a vector. ; Based on vectors sum vector Cosine distance between The semantic divergence (SDS) of the creative result relative to the design requirements is determined according to the following formula:
[0012] Where N is the total number of creative words contained in the creative results.
[0013] The second aspect of this disclosure provides a large language model training device based on reinforcement learning, applicable to creative design and inspiration search scenarios, including: An acquisition unit is used to acquire training data of a large language model, wherein the training data includes input data, thought chain, and output data of the large language model, and the thought chain includes all thought steps from input data to output data; The reward unit is used to combine reinforcement learning algorithms and process reward models to evaluate the divergence of the divergent association words generated by each thinking step in the thought chain, and to reward the corresponding thinking steps according to the divergence of the divergent association words. The reinforcement learning algorithm includes a proximal policy optimization algorithm. The training unit is used to sum up the rewards of all thinking steps in the thought chain to obtain a cumulative reward, and to train the large language model with the goal of maximizing the cumulative reward, so as to obtain a trained large language model.
[0014] A third aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to execute the reinforcement learning-based large language model training method provided in any of the first aspects.
[0015] A fourth aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform a large language model training method based on reinforcement learning provided in any of the first aspects. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a schematic diagram of the training method for a large language model based on reinforcement learning provided in an embodiment of this disclosure; Figure 2 A block diagram of a large language model training device based on reinforcement learning provided in an embodiment of this disclosure; Figure 3 A block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0018] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.
[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0021] Currently, large language models (such as DeepSeek-R1 and OpenAI o1) excel in code generation and mathematical problem-solving. This superior performance is primarily attributed to reinforcement learning on datasets with definite correct answers. Reinforcement learning training for large language models mainly targets mathematical, programming, and natural science scenarios, which are characterized by convergence. Problems typically have only one correct answer, and the training objective is to make the large language model converge to this unique solution. However, if this training method is applied to creative design and inspiration search scenarios without a single correct answer, it inhibits the divergent thinking ability of the large language model, causing it to become increasingly rigid with further training.
[0022] To address the aforementioned issues, this disclosure provides a reinforcement learning-based method for training large language models, applicable to creative design and inspiration search scenarios, such as... Figure 1 As shown, the method includes the following steps S11 to S13: Step S11: Obtain the training data of the large language model. The training data includes the input data, thought chain, and output data of the large language model. The thought chain includes all the thought steps from the input data to the output data. The thought chain (CoT) is a technology that improves the ability of the large language model to solve complex problems through step-by-step logical reasoning. Step S12: Combining reinforcement learning algorithms and process reward models, evaluate the divergence of the divergent associations generated by each thinking step in the thought chain, and reward the corresponding thinking steps according to the divergence of the divergent associations. The reinforcement learning algorithm includes the proximal policy optimization algorithm; the proximal policy optimization algorithm is a divergent reinforcement learning strategy. Through the proximal policy optimization algorithm and process reward model, each thinking step is evaluated, and dense rewards are given to the thinking steps in the thought chain. In one optional implementation of this disclosure, the input data is the original design requirements, and the output data is the last round of divergent association words obtained from the multiple rounds of semantic leaps. The specific number of rounds of semantic leaps can be two, three, or more. A single semantic leap can be a divergent association of the original design requirements or the previous round of divergent association words from the perspective of the target audience, or it can be a divergent association of the original design requirements or the previous round of divergent association words according to logical thinking methods such as synonyms, related words, synesthesia, and metaphor. Among them, related words refer to words that have a relationship with a word, such as logical relationships such as parallel, intersection, opposition, or unity.
[0023] Step S12 includes: Based on the original design requirements and the logical relationship between the multiple rounds of divergent association words, all words are divided into parent node words and corresponding child node words. Among them, all words include the original design requirements and the multiple rounds of divergent association words. The parent node vocabulary and child node vocabulary are vectorized using the Embedding model; the Embedding model can be the BGE-M3 Embedding model, which vectorizes the node vocabulary to facilitate subsequent calculation of semantic distance between words; By combining the proximal policy optimization algorithm and the process reward model based on semantic distance, the degree of hopping between parent node words and child node words, as well as the degree of diversity among all child node words under the same parent node word are evaluated; the degree of divergence among words is evaluated by the degree of hopping between parent node words and child node words, as well as the degree of diversity among all child node words under the same parent node word. Based on the degree of leap between parent node vocabulary and child node vocabulary, vertical rewards are given to the corresponding thinking steps; Based on the degree of diversity among all child nodes under the same parent node, lateral rewards are given to the corresponding thought processes.
[0024] In response to the divergent nature of creative design and inspiration search scenarios and the fact that creative tasks do not have a single standard answer, this disclosure redefines the optimization objective, focusing not on correctness but on creativity and semantic leaps, and rewards based on the degree of leaps and diversity between words. This fills the technical gap in related technologies where reinforcement learning training methods cannot effectively train open-ended creative tasks.
[0025] In one optional implementation of this disclosure, vertical rewards are given to corresponding thought steps based on the degree of jump between parent node words and corresponding child node words, including: Determine the cosine distance between the vectorized parent node vocabulary and the child node vocabulary; Vertical rewards are assigned to corresponding thought processes based on cosine distance, with larger cosine distances resulting in larger vertical rewards. These vertical rewards encourage large language models to generate more leaping words rather than simple repetitive ones.
[0026] In one optional implementation of this disclosure, lateral rewards are given to corresponding thought processes based on the degree of diversity among all child node words under the same parent node word, including: Determine the average distance between all child node words under the same parent node word after vectorization; Lateral rewards are assigned to corresponding thought processes based on average distance, with a larger average distance resulting in a larger lateral reward. A larger lateral reward indicates a richer and more diverse range of divergent associations generated, rather than a single type.
[0027] This embodiment of the disclosure rewards based on the semantic distance of the embedding vector. By calculating the semantic offset between the parent node words and the child node words (vertical divergence) and the semantic mutual exclusion degree of the sibling node words (horizontal diversity), the large language model is rewarded for jumping in the semantic space. Furthermore, by simulating the perspectives of different target groups and different logical thinking methods to make reasonable jumps, the large language model can still maintain logical coherence when jumping in the semantic space.
[0028] Step S13: The rewards of all thinking steps in the thought chain are summed to obtain a cumulative reward. The large language model is trained with the goal of maximizing this cumulative reward, resulting in a fully trained large language model. By maximizing the cumulative reward, the large language model learns to automatically trigger its internal thought chain upon receiving design requirements. It then performs multiple semantic leaps by simulating the perspectives of different target groups and different logical thinking methods, ultimately outputting a highly creative and logically consistent result.
[0029] This disclosure combines a near-end policy optimization algorithm and a process reward model to score and update each step of the thought chain based on semantic distance. It solves the problem of suppressing the divergence ability of large language models when the training method of large language models is applied to creative design and inspiration search scenarios without a single standard answer.
[0030] In one optional embodiment of this disclosure, after obtaining the trained large language model in step S13, the method further includes: The trained large language model is used to receive user input design requirements and output creative results. Based on the semantic distance between design requirements and creative outcomes, assess the divergence of creative outcomes relative to design requirements.
[0031] In one optional implementation of this disclosure, a trained large language model is used to receive user-inputted design requirements and output creative results, including: The design requirement is to use a trained large language model to receive user input. Based on design requirements, perform multiple semantic leaps to generate multiple rounds of divergent associative words; The last round of divergent associations from multiple rounds of associations will be output as the creative result.
[0032] After receiving design requirements, the trained large language model can automatically trigger its internal thought chain, and perform multiple semantic leaps by simulating the perspectives of different target groups and different logical thinking methods, ultimately outputting highly creative and logically consistent results.
[0033] In one optional embodiment of this disclosure, the divergence of the creative outcome relative to the design requirements is evaluated based on the semantic distance between the design requirements and the creative outcome, including: Using the Embedding model, design requirements are mapped to vectors. Map the i-th creative word in the creative results to a vector. The embedding model can be the BGE-M3 embedding model. The BGE-M3 embedding model is used to vectorize the creative words in the design requirements and creative results, which makes it easier to calculate the semantic distance between words. Based on vectors sum vector Cosine distance between The semantic divergence (SDS) of the creative outcome relative to the design requirements is determined using the following formula:
[0034] Where N is the total number of creative words contained in the creative results. Semantic divergence (SDS) is used to quantify the leap in thinking between the creative results and the design requirements. The score ranges from [0,2]. The higher the score, the farther the generated creative results jump in the semantic space and the stronger the divergence.
[0035] As can be seen from the above description, this disclosure achieves the following technical effects: This disclosure combines a near-end policy optimization algorithm and a process reward model to score and update each step of the thought chain based on semantic distance. It solves the problem of suppressing the divergence ability of large language models when the training method of large language models is applied to creative design and inspiration search scenarios without a single standard answer.
[0036] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0037] This disclosure also provides a reinforcement learning-based large language model training device for implementing the above method embodiments, applicable to creative design and inspiration search scenarios, such as... Figure 2 As shown, the construction device 20 includes: The acquisition unit 21 is used to acquire the training data of the large language model. The training data includes the input data, thought chain and output data of the large language model. The thought chain includes all thought steps from the input data to the output data. Reward unit 22 is used to combine reinforcement learning algorithm and process reward model to evaluate the divergence of the divergent association words generated by each thinking step in the thinking chain, and to reward the corresponding thinking steps according to the divergence of the divergent association words. The reinforcement learning algorithm includes the proximal policy optimization algorithm. Training unit 23 is used to sum up the rewards of all thinking steps in the thought chain to obtain a cumulative reward. The training objective is to maximize the cumulative reward to train the large language model and obtain the trained large language model.
[0038] The specific methods of execution of each unit in the above device embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.
[0039] This disclosure also provides an electronic device, such as... Figure 3 As shown, the electronic device includes one or more processors 31 and a memory 32. Figure 3 Take a processor 31 as an example.
[0040] The controller may also include an input device 33 and an output device 34.
[0041] The processor 31, memory 32, input device 33, and output device 34 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.
[0042] Processor 31 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips. The general-purpose processor can be a microprocessor or any conventional processor.
[0043] The memory 32, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the control method in this embodiment. The processor 31 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 32, thereby implementing the reinforcement learning-based large language model training method of the above-described method embodiment.
[0044] The memory 32 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the processing device operated by the server. Furthermore, the memory 32 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 32 may optionally include memory remotely located relative to the processor 31, and these remote memories can be connected to a network connection device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0045] Input device 33 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the server's processing device. Output device 34 may include display devices such as a display screen.
[0046] One or more modules are stored in memory 32, and when executed by one or more processors 31, they perform actions such as... Figure 1 The method shown.
[0047] Those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes as described in the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory (FM), hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.
[0048] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for training a large language model based on reinforcement learning, characterized in that, Applied to creative design and inspiration search scenarios, including: Obtain training data for a large language model, wherein the training data includes input data, thought chain, and output data of the large language model, and the thought chain includes all thought steps from input data to output data; By combining reinforcement learning algorithms and process reward models, the divergence of divergent association words generated by each thinking step in the thought chain is evaluated, and rewards are given to the corresponding thinking steps based on the divergence of the divergent association words. The reinforcement learning algorithm includes a proximal policy optimization algorithm. The rewards of all thinking steps in the thought chain are summed to obtain a cumulative reward. The large language model is trained with the goal of maximizing the cumulative reward, resulting in a trained large language model.
2. The method according to claim 1, characterized in that, The input data is the original design requirements, and the output data is the last round of divergent association words obtained from the input data through multiple rounds of semantic leaps. Specifically, the method of combining reinforcement learning algorithms and process reward models to evaluate the divergence of divergent associations generated at each thought step in the thought chain, and to reward the corresponding thought steps based on the divergence of the divergent associations, includes: Based on the original design requirements and the logical relationship between the multiple rounds of divergent association words, all words are divided into parent node words and corresponding child node words. Among them, all words include the original design requirements and the multiple rounds of divergent association words. Using the Embedding model, the vocabulary of parent nodes and child nodes is vectorized; By combining the aforementioned proximal policy optimization algorithm and the semantic distance-based process reward model, the degree of hopping between parent node words and child node words, as well as the degree of diversity among all child node words under the same parent node word, are evaluated. Based on the degree of leap between parent node vocabulary and child node vocabulary, vertical rewards are given to the corresponding thinking steps; Based on the degree of diversity among all child nodes under the same parent node, lateral rewards are given to the corresponding thought processes.
3. The method according to claim 2, characterized in that, The method of providing vertical rewards for corresponding thought steps based on the degree of leap between parent node words and corresponding child node words includes: Determine the cosine distance between the vectorized parent node vocabulary and the child node vocabulary; Vertical rewards are given to the corresponding thought steps based on the cosine distance, wherein the larger the cosine distance, the larger the vertical reward.
4. The method according to claim 2, characterized in that, The method of providing lateral rewards for corresponding thought processes based on the diversity among all child node words under the same parent node word includes: Determine the average distance between all child node words under the same parent node word after vectorization; Lateral rewards are given to the corresponding thought steps based on the average distance, wherein the greater the average distance, the greater the lateral reward.
5. The method according to claim 1, characterized in that, After obtaining the trained large language model, the method further includes: The trained large language model is used to receive user input design requirements and output creative results. Based on the semantic distance between the design requirements and the creative results, the divergence of the creative results relative to the design requirements is evaluated.
6. The method according to claim 5, characterized in that, The process of receiving user input design requirements using a trained large language model and outputting creative results includes: The design requirement is to use a trained large language model to receive user input. Based on the aforementioned design requirements, multiple semantic leaps are performed to generate multiple rounds of divergent associative words; The last round of divergent associations from the multi-round divergent associations is output as the creative result.
7. The method according to claim 5, characterized in that, The assessment of the divergence of the creative result relative to the design requirement, based on the semantic distance between the design requirement and the creative result, includes: Using the Embedding model, design requirements are mapped to vectors. Map the i-th creative word in the creative results to a vector. ; Based on vectors sum vector Cosine distance between The semantic divergence (SDS) of the creative result relative to the design requirements is determined according to the following formula: Where N is the total number of creative words contained in the creative results.
8. A large language model training device based on reinforcement learning, characterized in that, Applied to creative design and inspiration search scenarios, including: An acquisition unit is used to acquire training data of a large language model, wherein the training data includes input data, thought chain, and output data of the large language model, and the thought chain includes all thought steps from input data to output data; The reward unit is used to combine reinforcement learning algorithms and process reward models to evaluate the divergence of the divergent association words generated by each thinking step in the thought chain, and to reward the corresponding thinking steps according to the divergence of the divergent association words. The reinforcement learning algorithm includes a proximal policy optimization algorithm. The training unit is used to sum up the rewards of all thinking steps in the thought chain to obtain a cumulative reward, and to train the large language model with the goal of maximizing the cumulative reward, so as to obtain a trained large language model.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the large language model training method based on reinforcement learning as described in any one of claims 1 to 7.
10. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the reinforcement learning-based large language model training method according to any one of claims 1 to 7.