Multi-style reinforcement learning strategy control method and system based on natural language

Abstract

The invention provides a multi-style reinforcement learning strategy control method and system based on a natural language. Firstly, a multi-style reinforcement learning strategy is trained in a required environment, the multi-style reinforcement learning strategy is based on a meta-behavior set, each meta-behavior is associated with a corresponding style parameter, the style parameter represents the tendency to execute the meta-behavior, and when each training period starts, a group of style parameters is randomly sampled, and the set of style parameters is set as a set; splicing the style parameter and the environment state, inputting the spliced style parameter and environment state into an intelligent agent, and training by using a reinforcement learning algorithm to obtain a strategy with multiple behavior styles; the natural language of the user is analyzed during reasoning, the natural language of the user is converted into the numerical value of the style parameter corresponding to the meta-behavior in combination with the prompt word, the converted numerical value is input into the intelligent agent to serve as the current style parameter of the intelligent agent, and then the behavior style specified by the user can be executed. According to the invention, highly diversified behavior performance of the intelligent agent can be realized, and the user is allowed to adjust the degree of each behavior style through texts or voices.

Description

Technical Field

[0001] The present invention relates to the field of AI technology, and in particular to a natural language-based multi-style reinforcement learning strategy control method and system. Background Art

[0002] The existing reinforcement learning strategies based on natural language control usually accept simple and direct natural language instructions, and the controlled strategies can usually only complete some simple tasks, such as moving to a specified location, grabbing and placing objects, etc. Even if some methods combine movement and object manipulation into long sequences of tasks, these tasks are still combinations of simple tasks and lack the ability to handle complex tasks.

[0003] There are two main types of reinforcement learning technologies based on natural language control: 1. Natural language embedding state training This type of method converts natural language instructions into embedding vectors through pre-trained language models or word vectors, and then trains by splicing the embedding vectors with the environment state, using whether the instruction is completed as a reward during the training period. Its limitation is that this type of method needs to construct a batch of natural language instructions as a training set, and define specific tasks or behaviors for each instruction. It also relies on rules or manual judgment to evaluate the success or failure of instruction execution and allocate rewards for training. It has poor generalization ability for changes in instruction expression (such as the use of synonyms and near-synonyms), and it is difficult to handle complex or abstract instructions.

[0004] 2. Layered training This type of method processes natural language instructions and task execution through a hierarchical structure, trains an independent lower-level strategy for each task, and allows the upper-level strategy to select the appropriate lower-level strategy based on the natural language instructions. The upper-level strategy (which can be a model trained based on the lower-level strategy or a pre-trained language model) generates corresponding goals for the lower-level strategy to execute based on the natural language instructions. Its limitation is that the goals of the lower-level strategy need to be manually defined and lack flexibility. During training, rules or manual judgment are still required to allocate rewards based on whether the instructions are executed successfully. It can only complete a limited number of predefined tasks and has difficulty in dealing with novel or complex instructions. There are challenges in coordinating and optimizing the upper and lower-level strategies.

[0005] Neither of the above two methods can support the control of reinforcement learning strategies in complex sequential decision-making environments. When training strategies in complex environments, it is difficult to use rules or manual methods to determine whether instructions are executed successfully. It is also difficult to align abstract high-level instructions with reinforcement learning strategies using only pre-trained language models or word vectors. The ability to understand abstract and complex instructions is limited, and the ability to adapt to changes in instruction expressions is limited, making it difficult to deal with new instructions outside the scope of predefined tasks.

[0006] Although reinforcement learning has demonstrated the ability to surpass human level in complex sequential decision-making environments such as RTS games, MOBA games, ball games, chess, and autonomous driving, there is no reliable technology that can control reinforcement learning strategies through natural language in such complex environments. The difficulties are as follows: 1) Complex instruction understanding problem: In a highly complex environment, executable instructions are no longer limited to simple tasks, but may involve complex, abstract, or even vague concepts. The system needs to have the ability to understand and parse these complex instructions.

[0007] 2) Alignment between strategies and instructions: In complex environments, the state space and action space are often extremely large, the decision sequence is long and complex, and the optional strategies are diverse. This makes it extremely difficult to accurately align the behavior style or goal specified in natural language with the specific reinforcement learning strategy.

[0008] 3) Instruction execution evaluation problem: During the training process, it is a major challenge to determine whether complex instructions are completed correctly. Both manual evaluation and rule-based methods have difficulty accurately evaluating the execution quality of complex instructions, especially when the instructions involve long-term strategies or abstract concepts. This makes it extremely difficult to train the model to have the ability to follow complex instructions. Summary of the invention

[0009] In view of the defects in the prior art, the purpose of the present invention is to provide a multi-style reinforcement learning strategy control method and system based on natural language.

[0010] A natural language-based multi-style reinforcement learning strategy control method provided by the present invention includes: Training step: training a multi-style reinforcement learning strategy in a desired environment, wherein the multi-style reinforcement learning strategy is based on a set of meta-behaviors, each meta-behavior is associated with a corresponding style parameter, and the style parameter represents the tendency to perform the meta-behavior. At the beginning of each training period, a set of style parameters is randomly sampled, and the style parameters and the environment state are concatenated and input into the agent, and the reinforcement learning algorithm is used for training to obtain a strategy with multiple behavior styles; Further, the style parameters include: floating point type and Boolean type; Floating point type parameters are continuous and range from 0 to 1, and Boolean type parameters are binary and take values ​​of 0 or 1; A smaller floating point value means multiplying the original reward by a lower scale, and a larger value means multiplying the original reward by a larger scale; Boolean type parameters have two preferences: disable or enable the reward.

[0011] Instruction-to-style conversion step: This step receives the user's natural language instructions and executes the corresponding behavior style. The instruction-to-style conversion step first introduces the application scenario and the concept of each style parameter to the large language model through prompt words, and gives a small number of example samples in the prompt words; during reasoning, the user's natural language is parsed, and the user's natural language is converted into the numerical value of the style parameter of the corresponding meta-behavior in combination with the prompt words, and the converted numerical value is input into the intelligent agent as the current style parameter of the intelligent agent to execute the behavior style specified by the user.

[0012] Furthermore, in the instruction-to-style conversion step, the structure of the prompt words given to the large model includes: scene description, style parameter type and meaning description, instruction parsing example samples, natural language instructions and output format requirements.

[0013] Furthermore, the construction method of the instruction parsing example sample in the prompt word module includes: Let the big model decompose the user's natural language and find out the meta-behaviors related to the natural language; Let the big model determine the numerical degree of each relevant meta-behavior based on the user's intention and tone.

[0014] Furthermore, after the instruction-to-style conversion step, a fine adjustment step is also included: Receive the user's subsequent natural language; The large language model identifies relevant style parameters through cue words; Infer user intent and determine the direction of style parameter adjustment; Determine the amount of adjustment based on user intent and tone.

[0015] A natural language-based multi-style reinforcement learning strategy control system provided by the present invention includes: Training module: training a multi-style reinforcement learning strategy in the required environment. The multi-style reinforcement learning strategy is based on a set of meta-behaviors. Each meta-behavior is associated with a corresponding style parameter. The style parameter represents the tendency to perform the meta-behavior. At the beginning of each training period, a set of style parameters is randomly sampled. The style parameters and the environment state are concatenated and input into the agent. The reinforcement learning algorithm is used for training to obtain a strategy with multiple behavior styles. Further, the style parameters include: floating point type and Boolean type; Floating point type parameters are continuous and range from 0 to 1, and Boolean type parameters are binary and take values ​​of 0 or 1; A smaller floating point value means multiplying the original reward by a lower scale, and a larger value means multiplying the original reward by a larger scale; Boolean type parameters have two preferences: disable or enable the reward.

[0016] Instruction-to-style conversion module: Introduce application scenarios and the types and concepts of each style parameter to the large language model through prompt words, and give a small number of example samples in the prompt words; parse the user's natural language during reasoning, and convert the user's natural language into the numerical value of the style parameter of the corresponding meta-behavior in combination with the prompt words, and input the converted numerical value into the intelligent agent as the current style parameter of the intelligent agent to execute the behavioral style specified by the user.

[0017] Furthermore, in the instruction-to-style conversion module, the prompt word structure includes: scene description, parameter type and meaning description, instruction parsing example sample, natural language instruction and output format requirements.

[0018] Furthermore, in the prompt word module, the construction system of the instruction parsing example sample includes: Disassemble the user's natural language, and the large language model finds meta-behaviors related to the natural language through prompt words; Determine the numerical degree of each relevant meta-behavior based on the user's intent and tone.

[0019] Furthermore, it also includes a fine adjustment module: Receive the user's subsequent natural language; The large language model identifies relevant style parameters through cue words; Infer user intent and determine the direction of style parameter adjustment; Determine the amount of adjustment based on user intent and tone.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The present invention enables users to adjust various behaviors and strategic tendencies of the intelligent agent during the game process through simple natural language (text or voice) input. It has the advantages of intuitiveness, flexibility, accuracy, immediacy and personalization, which can not only improve the user experience and reduce the user's learning and operation costs, but also open up new possibilities for the application of AI in sports games. This method has the potential to be extended to AI applications in other fields.

[0021] The present invention implements natural language control reinforcement learning strategies in complex environments, breaking through the limitations of previous methods that are mainly limited to simple object manipulation and navigation and movement tasks. It can be effectively implemented in complex sequence decision-making tasks such as games, expanding the scope of application. It can understand and execute abstract, high-level instructions, improve the intelligence level of the system, and enable AI to handle complex descriptions that are closer to human thinking.

[0022] The multi-style reinforcement learning training method used in this solution can achieve highly diverse behavioral performance of intelligent agents. Through the prompt word design of a large language model, precise alignment between natural language instructions and multi-style strategies is achieved. Allowing users to fine-tune the degree of each behavioral style through instructions provides personalization and customization capabilities to meet the specific needs of different users. Through natural language or voice interaction, the user's learning and operation costs are greatly reduced, allowing AI to more accurately understand and execute user intentions, generate behaviors that meet user needs, and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is the overall workflow diagram of the present invention; Figure 2 Schematic diagram of the instruction-to-style conversion method; Figure 3 Schematic diagram for fine-tuning the style; Figure 4 This is a schematic diagram of voice control. DETAILED DESCRIPTION

[0024] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0025] Reinforcement Learning (RL): Reinforcement learning is a machine learning method that allows an agent to learn how to make optimal decisions by interacting with the environment. The goal of reinforcement learning is to find a strategy that allows the agent to obtain the maximum cumulative reward in the long run.

[0026] Training environment: Or simply called environment, in reinforcement learning, the environment is the world that the agent interacts with. The environment receives the agent's actions and returns new states and rewards based on these actions. The environment can be any system that the agent can interact with, such as a game, the physical world of a robot, or even a stock market model.

[0027] Agent: In reinforcement learning, an agent is an entity that can perceive the environment and perform actions in the environment. The goal of the agent is to maximize the total reward by learning how to choose actions.

[0028] State: A state is a description of the environment at a certain moment in time, which contains all the necessary information that the agent needs to make a decision. For example, in a board game, the state may include the type of chess piece at each position on the board.

[0029] Action: An action is a basic operation that an agent can perform in a given state. An agent interacts with the environment by performing actions. For example, if the agent is a character in a game, possible actions include moving left, moving right, jumping, etc.

[0030] Reward: Reward is the feedback from the environment to the agent's actions. When the agent performs an action, the environment returns a reward to tell the agent whether the action is good or bad. Rewards can be positive (indicating encouragement of the action) or negative (indicating a desire to suppress the action). The goal of the agent is to maximize the total reward by learning how to choose actions.

[0031] Episode: The training environment does not run indefinitely, but starts and ends in units of training episodes. At the beginning of each training episode, the environment will be initialized to a certain state; at the end of each training episode, done will be used as a signal.

[0032] Reward Shaping: Also known as reward setting, reward shaping is a reinforcement learning technique that guides the agent to learn faster by designing the reward function.

[0033] Meta-behavior: The main behavior of the agent in the training environment. Usually, most of the agent's behaviors in the environment can be composed of meta-behaviors. For example, the behavior of the agent jumping to the left can be completed by the combination of the two meta-behaviors of the agent moving to the left and jumping.

[0034] Large Language Model (LLM): A large language model is a large-scale neural network trained on massive amounts of data that can understand and generate human language. Such models typically have billions to trillions of parameters and can perform a variety of language tasks such as text generation, question answering, translation, and summarization.

[0035] Natural Language: Natural language refers to the language that humans use naturally in daily life, such as Chinese, English, French, etc., for communication between people. It is different from artificially designed formal languages, such as programming languages ​​or logical symbol systems, which are usually used in specific technical fields.

[0036] Prompt: refers to the input text or instructions given to the model when using a large language model, which is used to guide the model to generate specific outputs or perform specific tasks. Prompt design can significantly improve the performance of the model, allowing it to more accurately understand user intent and produce the desired results.

[0037] Speech recognition (Automatic Speech Recognition, ASR): is a technology that automatically converts human speech into text and is a key technology for realizing human-computer voice interaction. It processes audio signals into corresponding text through components such as acoustic models, language models, and decoders.

[0038] Using natural language as an intuitive way for users to interact with AI models can significantly improve the user experience while reducing the user's learning cost and operating threshold. The present invention has been innovatively applied in the design of AI Bot. Specifically, the present invention has trained an AI Bot that can be controlled by natural language, allowing users to adjust the various behaviors and strategic tendencies of AI Bot during the game through simple voice input. For example, users can adjust the behavior pattern of AI Bot through instructions such as "increase the frequency of three-point shots in the bottom corner area" or "increase the enthusiasm for rebounding and reduce tip-in attempts." This method is not only limited to specifying the behavior of AI Bot, but also supports more refined adjustments. Users can control the different tendencies of AI Bot behavior through instructions such as "slightly increase the defensive intensity" or "completely give up offense and devote all their efforts to defense."

[0039] like Figure 1As shown, the multi-style reinforcement learning strategy control method based on natural language of the present invention first trains the multi-style reinforcement learning strategy in the target environment. The strategy is based on a series of preset meta-behaviors, each of which is associated with a specific style parameter. The style parameter represents the tendency to perform the behavior. Through the combination of different meta-behaviors and style parameters, the strategy can achieve a variety of behavioral styles. After the training is completed, the behavioral style of the strategy can be precisely controlled by adjusting the style parameters. The natural language understanding and reasoning capabilities of the large language model are used to convert the user's natural language instructions into corresponding style parameters. This process requires the design of prompt words based on the environment and meta-behaviors. The prompt word first introduces the target environment, and then details the meaning and function of each meta-behavior, as well as the type of corresponding style parameters. In order to make full use of the few-shot learning and chain of thought capabilities of the large language model, several instruction parsing example samples are added to the prompt word. These example samples show how to correspond natural language instructions to relevant style parameters through step-by-step reasoning, and specify the numerical degree of the style parameters. Considering the limitations of the large language model in processing precise numerical values, we use a discrete scale to represent the parameter degree, and then map this scale to the actual adjustable range of the style parameter. The user's natural language instructions are combined with the designed prompt words and then input into the large language model. After processing and mapping, the specific style parameters corresponding to the instructions can be obtained, and then handed over to the multi-style reinforcement learning strategy for execution. This method not only achieves the seamless connection between natural language and reinforcement learning strategies, but also improves the flexibility of the system and user ease of use.

[0040] The present invention trains a multi-style reinforcement learning strategy that can be controlled by parameter input. The strategy can perform well in complex sequential decision-making tasks. It eliminates the reliance on manual judgment or predetermined rules to determine whether the instruction is successful during the training process, and can automatically learn various behaviors. At the same time, the present invention has the ability to understand and execute complex, abstract high-level instructions. It achieves precise alignment of instructions and style parameters, and can accurately specify the degree of each style parameter. It is robust to different expressions (such as synonyms and sentence changes). It allows users to make real-time and refined adjustments to AI behaviors through natural language. It can access speech recognition and process inputs of other modalities such as speech.

[0041] Specifically, the natural language-based multi-style reinforcement learning strategy control method of the present invention includes: Training steps: Design a comprehensive set of meta-behaviors based on the characteristics of the environment and the requirements of the task. Each meta-behavior is associated with a specific style parameter. The size of the style parameter represents the scale of the reward after completing the meta-behavior, and also represents the policy's tendency towards the meta-behavior. Design a reward function for each meta-behavior to ensure that the scale of the reward for the meta-behavior changes with the value of the style parameter. Style parameters are divided into two types: floating point and Boolean. Floating point type parameters are continuous, ranging from 0 to 1; while Boolean type parameters are binary, with values ​​of 0 or 1. A smaller floating point value means that the original reward is multiplied by a lower scale, and a larger value means that the original reward is multiplied by a larger scale; Boolean type parameters have two preferences: disable or enable the reward. At the beginning of each training period, a set of style parameters is randomly sampled, and the style parameters and the environment state are concatenated and input into the agent, and the reinforcement learning algorithm is used for training to obtain a multi-style reinforcement learning strategy.

[0042] During training, the degree of task completion can be determined by style parameters and corresponding rewards, eliminating the need to write rules for each behavior or manually determine whether the instructions are completed during training. Since no human labeling is required, the agent can sample a large amount of data with different style parameters in the environment for training, allowing this method to train effective strategies in complex environments. This method can train a highly flexible and controllable multi-style reinforcement learning strategy. This strategy can exhibit diverse behavior patterns in complex environments according to different style parameter settings.

[0043] Instruction to style conversion step: This step receives the user's natural language instructions and executes the corresponding behavior style.

[0044] The trained multi-style reinforcement learning strategy can control its behavior style through style parameters. Using natural language instructions to control the strategy requires accurate conversion of natural language instructions into matching style parameters. Through the designed instruction-to-style conversion method, the natural language understanding and reasoning capabilities of the large language model are utilized to achieve accurate mapping of natural language instructions to multi-style reinforcement learning strategy parameters. The instruction-to-style conversion method first needs to construct prompt words, in which the application scenario and the concept and type of each style parameter are introduced to the large language model. In order to make full use of the thinking chain and few-shot learning capabilities of the large language model, the method gives several example samples in the prompt words. The construction form of the example samples is: a) disassemble the high-level instructions step by step to find the meta-behaviors related to the instruction; b) determine the numerical degree of each related meta-behavior according to the user's intention and tone. Considering the limitations of the large language model in processing precise numerical values, the conversion method allows it to output a scale of 0 to 10 to represent the degree of floating-point type style parameters, and then maps this scale to the actual adjustable range of the style parameters. For Boolean type style parameters, 0 or 1 is output. For example, the instruction "Full defense!" means that the user wants the AI ​​to fully defend. Therefore, meta-behaviors related to defense should have a high preference, while meta-behaviors related to offense should have a low preference. Finally, the generated preference is mapped to the actual interval of the style parameter and input to the agent as the current style parameter of the agent.

[0045] The implementation process of this instruction to style conversion method is as follows Figure 2 As shown: 1) The content required to construct the prompt words includes: a) Scenario description b) Parameter type and meaning c) Instruction parsing example d) Natural language instructions e) Output format requirements 2) The processing process of the large language model is: a) Parse the instructions and identify the relevant meta-behavior b) Evaluate the preference of each relevant meta-behavior c) Generate structured output 3) Post-processing: Map the 0-10 degree value or Boolean type parameter output by the large language model to the actual parameter range.

[0046] Fine-tuning steps: In a complex interactive environment, the same instruction may produce different expected results due to differences in user understanding. For example, in football, for the instruction "shrink defense", some users think that retreating to their own half is enough, while other users think that they need to retreat to their own penalty area. To solve this problem, this solution introduces a refined instruction adjustment mechanism that allows users to fine-tune the behavioral tendencies of AI through subsequent instructions. When the instruction does not meet the user's expectations, the user can continue to issue instructions to fine-tune some meta-behaviors of AI. We achieve this function by adding the current style parameters to the prompt words and adjusting the new style parameters based on the current style parameters. The process of this method is as follows: a) Receive the user's adjustment instruction (such as "step back a little more"). b) The large language model identifies the relevant style parameters (such as formation, position) through the designed prompt words. c) Infer the user's intention and determine the direction of parameter adjustment (such as reducing the relevant parameters). d) Select the appropriate change range according to the user's intention and tone. Since large language models are not very effective in processing precise numerical values, when it comes to adjusting the amplitude of style parameters, we do not directly let it output a specific change value, but discretize it into multiple change amplitudes, such as unchanged, slightly reduced, greatly reduced, slightly increased, and greatly increased. Each change amplitude level corresponds to a Gaussian distribution. The large language model selects the corresponding change amplitude based on the user's intention and tone, and then samples a change value from the Gaussian distribution corresponding to each change amplitude and adds it to the current style parameter. This method maps the adjustment amplitude of the user's instruction to a numerical value, and introduces a moderate amount of randomness, which increases the flexibility of the system. This process is as follows: Figure 3 shown.

[0047] Voice input control: In addition to being controlled by natural language in text form, the strategies in the present invention can also be controlled by voice input. Figure 4 As shown in the figure, during the process of receiving voice input, an external voice recognition module is used to process the voice input. The voice recognition module converts the received voice instructions into natural language instructions and then inputs them into the large language model. This method modularizes voice recognition separately, which is easy to integrate and upgrade. You can choose an ASR system optimized for vertical fields, and you can also perform post-processing such as adding hot word correction, context correction and other modules to improve recognition accuracy.

[0048] The present invention optimizes the execution capability of high-level instructions in complex environments: The present invention can accurately execute high-level abstract natural language instructions in complex sequential decision environments. This capability significantly improves the applicability and flexibility of the system, enabling AI to understand and execute more complex and abstract tasks.

[0049] The present invention has a training process that does not require manual judgment: the present invention breaks through the limitation of traditional methods that rely on manual annotation and rules to judge whether instructions are executed successfully. During the training stage, the system can autonomously learn various behaviors in complex scenarios, greatly improving the training efficiency and the system's autonomous learning ability.

[0050] The present invention has a strong command generalization capability: the present invention performs well in processing different expressions and can effectively understand and execute commands containing synonyms, near-synonyms, different tones, sentence inversions, etc. This strong generalization capability enables the system to better adapt to the language habits and expressions of different users.

[0051] The present invention has a refined real-time behavior adjustment capability: in the inference stage, users can use natural language to make real-time and refined adjustments to the behavior of AI. This dynamic adjustment mechanism enables the system to better adapt to the user's specific intentions and changing needs, improving the system's adaptability and user satisfaction.

[0052] The present invention can accept voice input in addition to natural language in text form, thereby reducing the cost of user learning and use and improving user experience.

[0053] The present invention also provides a multi-style reinforcement learning strategy control system based on natural language, which can be implemented by executing the process steps of the multi-style reinforcement learning strategy control method based on natural language, that is, those skilled in the art can understand the multi-style reinforcement learning strategy control method based on natural language as a preferred implementation of the multi-style reinforcement learning strategy control system based on natural language. The system includes: Training module: Train a multi-style reinforcement learning strategy in the required environment. The multi-style reinforcement learning strategy is based on a set of meta-behaviors. Each meta-behavior is associated with a corresponding style parameter. The style parameter represents the tendency to perform the meta-behavior. At the beginning of each training period, a set of style parameters is randomly sampled, and the style parameters and environmental states are concatenated and input into the intelligent agent for training using a reinforcement learning algorithm.

[0054] Instruction-to-style conversion module: Introduce the application scenario and the concept of each style parameter to the large language model through prompt words, and give samples in the prompt words; parse the user's natural language, and convert the user's natural language into the numerical value of the style parameter of the corresponding meta-behavior in combination with the prompt words, and input the converted numerical value into the intelligent agent as the current style parameter of the intelligent agent.

[0055] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0056] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multi-style reinforcement learning strategy control method based on natural language, characterized in that: include: Training step: training a multi-style reinforcement learning strategy in a desired environment, wherein the multi-style reinforcement learning strategy is based on a set of meta-behaviors, each meta-behavior is associated with a corresponding style parameter, and the style parameter represents the tendency to perform the meta-behavior. At the beginning of each training period, a set of style parameters is randomly sampled, and the style parameters and the environment state are concatenated and input into the agent, and the reinforcement learning algorithm is used for training to obtain a multi-style reinforcement learning strategy; Instruction-to-style conversion step: receiving the user's natural language instructions and executing the corresponding behavioral style; the instruction-to-style conversion step first introduces the application scenario and the concept of each style parameter to the large language model through prompt words, and gives example samples in the prompt words; during reasoning, the user's natural language instructions are parsed, and the user's natural language instructions are converted into numerical values ​​of the style parameters of the corresponding meta-behavior in combination with the prompt words, and the converted numerical values ​​are input into the intelligent agent as the current style parameters of the intelligent agent to execute the behavioral style specified by the user.

2. The multi-style reinforcement learning strategy control method based on natural language according to claim 1 is characterized in that: The style parameters include: floating point type and Boolean type; Floating point type parameters are continuous and range from 0 to 1, and Boolean type parameters are binary and take values ​​of 0 or 1; A smaller floating point value means multiplying the original reward by a lower scale, and a larger value means multiplying the original reward by a larger scale; Boolean type parameters have two preferences: disable or enable the reward.

3. The multi-style reinforcement learning strategy control method based on natural language according to claim 1 is characterized in that: In the instruction-to-style conversion step, the structure of the prompt word includes: scene description, parameter type and meaning description, instruction parsing example sample, natural language instruction and output format requirements.

4. The natural language-based multi-style reinforcement learning strategy control method according to claim 3 is characterized in that: The construction method of the instruction parsing example sample includes: The large model is used to decompose the user's natural language. The large language model uses prompt words to find out the meta-behaviors related to the natural language instructions. Let the big model determine the numerical degree of each relevant meta-behavior based on the user's intention and tone.

5. The multi-style reinforcement learning strategy control method based on natural language according to claim 1 is characterized in that: After the instruction-to-style conversion step, a fine-tuning step is also included: Receive subsequent natural language instructions from the user; The large language model identifies relevant style parameters through cue words; Infer user intent and determine the direction of style parameter adjustment; Determine the amount of adjustment based on user intent and tone.

6. A multi-style reinforcement learning strategy control system based on natural language, characterized in that: include: Training module: training a multi-style reinforcement learning strategy in the required environment. The multi-style reinforcement learning strategy is based on a set of meta-behaviors. Each meta-behavior is associated with a corresponding style parameter. The style parameter represents the tendency to execute the meta-behavior. At the beginning of each training period, a set of style parameters is randomly sampled. The style parameters and the environment state are concatenated and input into the agent. The reinforcement learning algorithm is used for training to obtain a multi-style reinforcement learning strategy. Instruction-to-style conversion module: receives the user's natural language instructions and executes the corresponding behavioral style; the instruction-to-style conversion module first introduces the application scenario and the concept of each style parameter to the large language model through prompt words, and gives example samples in the prompt words; parses the user's natural language instructions during reasoning, and converts the user's natural language instructions into the numerical value of the style parameter of the corresponding meta-behavior in combination with the prompt words, and inputs the converted numerical value into the intelligent agent as the current style parameter of the intelligent agent to execute the behavioral style specified by the user.

7. The natural language-based multi-style reinforcement learning strategy control system according to claim 6, characterized in that: The style parameters include: floating point type and Boolean type; Floating point type parameters are continuous and range from 0 to 1, and Boolean type parameters are binary and take values ​​of 0 or 1; A smaller floating point value means multiplying the original reward by a lower scale, and a larger value means multiplying the original reward by a larger scale; Boolean type parameters have two preferences: disable or enable the reward.

8. The natural language-based multi-style reinforcement learning strategy control system according to claim 6, characterized in that: In the instruction-to-style conversion module, the structure of the prompt word includes: scene description, parameter type and meaning description, instruction parsing example, natural language instruction and output format requirements.

9. The natural language-based multi-style reinforcement learning strategy control system according to claim 8, characterized in that: The construction method of the instruction parsing example sample includes: Disassemble the user's natural language, and the large language model finds the meta-behavior related to the natural language instruction through the prompt words; Determine the numerical degree of each relevant meta-behavior based on the user's intent and tone.

10. The natural language-based multi-style reinforcement learning strategy control system according to claim 6, characterized in that: Also includes fine-tuning modules: Receive subsequent natural language instructions from the user; The large language model identifies relevant style parameters through cue words; Infer user intent and determine the direction of style parameter adjustment; Determine the amount of adjustment based on user intent and tone.

Images