Method and apparatus for behavior decision of human-computer interaction device, device and storage medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a behavior decision-making method and device of a man-machine interaction device, equipment and a storage medium, wherein the method comprises the following steps: obtaining a reinforcement target of a target man-machine interaction device; constructing a target function with constraints according to the reinforcement target; alternately and iteratively training the target man-machine interaction device by using a preset reinforcement learning strategy and a preset learning strategy of imitating natural language to make a response; optimizing the target function until a preset reinforcement condition is met, so that the target man-machine interaction device is obtained after reinforcement is completed; identifying a user instruction type by using the target man-machine interaction device after reinforcement is completed; controlling the target man-machine device to execute corresponding optimal behavior decision-making based on the instruction type; and when no user instruction is obtained, controlling the target man-machine device to autonomously complete a task. Therefore, the application can help artificial intelligence to understand the intention of a speaker from human language, and lays a foundation for the wide application of future artificial intelligence.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for behavior decision-making of human-computer interaction devices. Background Technology

[0002] Building intelligent agents capable of assisting humans in performing complex tasks remains a long-term AI challenge. As the most widespread means of human-to-human communication, natural language has become a universal and crucial human-computer interaction interface. For decades, how to build AI systems capable of understanding and executing natural language instructions has been a key focus and research issue for the scientific community.

[0003] In recent years, with the widespread application of deep reinforcement learning technology, more and more researchers have attempted to solve the problem of understanding and executing natural language instructions as a reinforcement learning problem. More specifically, a reinforcement learning agent takes a natural language instruction describing a task objective as input and receives a positive reward upon task completion. Generally, a well-trained agent exhibits excellent generalization ability for various possible instructions. Although this method is concise and effective, it often requires a generator capable of continuously producing random instructions and an explicitly given reward function. Therefore, most related work can only be used to solve simple location and pathfinding problems. In these problems, instructions can generally be generated using a fixed set of templates, such as "go to the red box next to the blue wall." However, in most scenarios, natural language is usually abstract, and it is difficult to directly map natural language instructions into a reward function, making reinforcement learning training very challenging.

[0004] Another major research direction is collecting paired "behavior-instruction" human data to train agents capable of interpreting natural language instructions. This approach is often applied in fields such as robotics and localization / pathfinding. Since accurate instruction-based reward functions are no longer readily available, related work typically uses behavior cloning (BC) to directly mimic human behavior, or inverse reinforcement learning (IRL) to learn an instruction-based reward function. These methods often require collecting large amounts of human data, making them very expensive. More importantly, BC and IRL methods assume that human demonstrations are optimal, which is reflected in two aspects: (i) given a natural language instruction, the behavior in the demonstration is optimal; (ii) the natural language instruction in the demonstration is the optimal instruction to complete the corresponding task. In some fields, such as robotics and localization / pathfinding, this assumption may indeed hold true. However, in more complex problems, such as real-time strategy games, due to limitations in human players' game understanding and skill level, demonstration data from humans is often not optimal. Summary of the Invention

[0005] This application provides a behavior decision-making method, device, electronic device, and storage medium for human-computer interaction devices, which can help artificial intelligence understand the speaker's intention from human language and change its own behavior accordingly to meet the speaker's diverse needs, while autonomously seeking better execution methods, thereby laying the foundation for the widespread application of artificial intelligence in the future.

[0006] The first aspect of this application provides a behavior decision-making method for a human-computer interaction device, comprising the following steps: obtaining a reinforcement target for the target human-computer interaction device; constructing a constrained objective function based on the reinforcement target; alternately iteratively training the target human-computer interaction device using a preset reinforcement learning strategy and a preset learning strategy that mimics natural language responses to optimize the objective function until the preset reinforcement conditions are met, thereby obtaining a reinforced target human-computer interaction device; using the reinforced target human-computer interaction device to identify user instruction types; controlling the target human-computer interaction device to execute corresponding optimal behavior decisions based on the instruction types; and controlling the target human-computer interaction device to autonomously complete tasks when no user instructions are obtained.

[0007] Optionally, the step of alternately iteratively training the target human-computer interaction device using a preset reinforcement learning strategy and a preset learning strategy that mimics natural language responses to optimize the objective function until the preset reinforcement condition is met, thereby obtaining a reinforced target human-computer interaction device, includes: training the target human-computer interaction device using the preset reinforcement learning strategy to obtain behavioral trajectories; training the target human-computer interaction device using the preset learning strategy that mimics natural language responses to mimic real human responses to natural language instructions, while simultaneously performing self-distillation on the obtained behavioral trajectories; constructing a trajectory set based on all behavioral trajectories obtained through alternating iterations, and storing the trajectory set and all learning response data in the target human-computer interaction device to obtain a reinforced target human-computer interaction device.

[0008] Optionally, the formula for the preset reinforcement learning strategy is:

[0009] in, Describe the objective function. These are the parameters that the target model (i.e., the model of the decision function of the human-computer interaction device) needs to be trained on. These are the parameters of the model after the (k-1)th iteration. The learning rate represents the parameter update. express about The derivative, This represents the parameters obtained after reinforcement learning training in the k-th iteration.

[0010] Optionally, the formula for the preset learning strategy that mimics natural language responses is:

[0011] in, These are the parameters that the target model needs to train. These are the parameters of the model after the (k-1)th iteration. This represents the parameters obtained after reinforcement learning training in the k-th iteration. Indicates The decision function of a human-computer interaction device with parameters. A dataset representing the behavioral trajectories of real humans responding to natural language. The parameter is Decision function The collection of behavioral trajectories that successfully achieve the goal, where NA represents In collecting No natural language instructions are given at the time. Representing the trajectory The set goals were successfully achieved. The learning rate represents the parameter update. Represents the evaluation distance function Regarding parameters The gradient of, where The assessment focuses on behavioral trajectories. and On the union of, the parameter is The distance between the behavior generated by the decision function and the original behavior in the trajectory.

[0012] Optionally, the objective function of the enhanced target is:

[0013] in, These are the parameters used for model training. This represents the decision function of a human-computer interaction device, which determines the decision based on the state observed at time t. And the natural language given by humans Make a decision, For the actions to be taken, It is the objective function optimized by reinforcement learning, where NA represents the current time step without any instructions. Decisions are made solely based on the observed state of the human-computer interaction device. These are constraints. It is a dataset of real human responses to natural language.

[0014] A second aspect of this application provides a behavior decision-making device for a human-computer interaction device, comprising: an acquisition module for acquiring reinforcement targets of a target human-computer interaction device; a training module for constructing a constrained objective function based on the reinforcement targets, and alternately iteratively training the target human-computer interaction device using a preset reinforcement learning strategy and a preset learning strategy that imitates natural language responses to optimize the objective function until the preset reinforcement conditions are met, thereby obtaining a reinforced target human-computer interaction device; and a decision-making module for identifying user instruction types using the reinforced target human-computer interaction device, controlling the target human-computer interaction device to execute corresponding optimal behavior decisions based on the instruction types, and controlling the target human-computer interaction device to autonomously complete tasks when no user instructions are acquired.

[0015] Optionally, the training module is used to: train the target human-computer interaction device using the preset reinforcement learning strategy to obtain behavioral trajectories; train the target human-computer interaction device to imitate real human responses to natural language instructions using the preset learning strategy of mimicking natural language responses, and simultaneously perform self-distillation on the obtained behavioral trajectories; construct a trajectory set based on all behavioral trajectories obtained through alternating iterations, and store the trajectory set and all learning response data in the target human-computer interaction device to obtain a reinforced target human-computer interaction device.

[0016] Optionally, the formula for the preset reinforcement learning strategy is:

[0017] in, Describe the objective function. These are the parameters that the target model (i.e., the model of the decision function of the human-computer interaction device) needs to be trained on. These are the parameters of the model after the (k-1)th iteration. The learning rate represents the parameter update. express about The derivative, This represents the parameters obtained after reinforcement learning training in the k-th iteration.

[0018] Optionally, the formula for the preset learning strategy that mimics natural language responses is:

[0019] in, These are the parameters that the target model needs to train. These are the parameters of the model after the (k-1)th iteration. This represents the parameters obtained after reinforcement learning training in the k-th iteration. Indicates The decision function of a human-computer interaction device with parameters. A dataset representing the behavioral trajectories of real humans responding to natural language. The parameter is Decision function The collection of behavioral trajectories that successfully achieve the goal, where NA represents In collecting No natural language instructions are given at the time. Representing the trajectory The set goals were successfully achieved. The learning rate represents the parameter update. Represents the evaluation distance function Regarding parameters The gradient of, where The assessment focuses on behavioral trajectories. and On the union of, the parameter is The distance between the behavior generated by the decision function and the original behavior in the trajectory.

[0020] Optionally, the objective function of the enhanced target is:

[0021] in, These are the parameters used for model training. This represents the decision function of a human-computer interaction device, which determines the decision based on the state observed at time t. And the natural language given by humans Make a decision, For the actions to be taken, It is the objective function optimized by reinforcement learning, where NA represents the current time step without any instructions. Decisions are made solely based on the observed state of the human-computer interaction device. These are constraints. It is a dataset of real human responses to natural language.

[0022] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the behavior decision-making method for a human-computer interaction device as described in the above embodiments.

[0023] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the behavior decision method for a human-computer interaction device as described in the above embodiments.

[0024] Therefore, this application has at least the following beneficial effects: With the widespread application of reinforcement learning technology, artificial intelligence can help people complete a variety of complex tasks. However, current reinforcement learning-based AI can only complete some specific pre-set tasks. To better assist humans, the next generation of reinforcement learning-based AI needs to interact more with humans and better meet the diverse needs of humanity. Natural language is the bridge for human communication; people communicate, cooperate, express their views, and articulate their intentions through natural language. It can be said that natural language is a very important interface for human-computer interaction. In current technologies, training an intelligent agent that obeys natural language requires accurately mapping natural language to a reward function, or assuming that human behavior in human demonstration data is optimal.

[0025] To this end, this application proposes an iterative training framework that enables the trained agent to: a) strictly follow instructions when given a command, even if the command's objective contradicts the pursuit of higher rewards; b) choose the optimal behavior to follow a more abstract command, even if multiple behavioral patterns exist to achieve the desired effect, thereby maximizing rewards while complying; and c) autonomously complete tasks and earn higher rewards even without human input. This framework helps artificial intelligence understand the speaker's intent from human language and adjust its behavior accordingly to meet the speaker's diverse needs, while autonomously seeking better execution methods, thus laying the foundation for the widespread application of artificial intelligence in the future.

[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0027] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the behavior decision-making method for a human-computer interaction device according to an embodiment of this application; Figure 2 This is an example diagram of a behavior decision-making device for a human-computer interaction device provided according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0029] This application aims to train an agent that can both obtain higher rewards and obey natural language commands. In practice, while agents trained using RL can obtain higher rewards, they cannot obey natural language commands. Furthermore, in reality, when humans follow natural language commands, their actions are often not optimal and exhibit strong biases. This is because natural language is abstract; different people may be executing the same command, but their specific actions can vary significantly, leading to the failure of simple behavioral cloning. Additionally, human commands themselves are highly biased, resulting in a conflict between obeying commands and obtaining high rewards for the agent.

[0030] Based on the above points, the main problems to be solved by the embodiments of this application are: a) When a human issues an instruction, the intelligent agent should strictly follow the instruction and take action, even if the goal of the instruction contradicts obtaining a higher reward; b) When a human issues a relatively abstract instruction, there are multiple behavioral patterns that can achieve the effect of obeying the instruction, and the intelligent agent should choose the optimal way to obey the instruction, thereby obtaining a higher reward while obeying the instruction; c) When no human issues any instruction, the intelligent agent is capable of autonomously completing the task and obtaining a higher reward.

[0031] Therefore, this application proposes a behavior decision-making method, apparatus, electronic device, and storage medium for human-computer interaction devices. The behavior decision-making method, apparatus, electronic device, and storage medium of the human-computer interaction device according to the embodiments of this application will be described below with reference to the accompanying drawings.

[0032] Specifically, Figure 1 This is a flowchart illustrating a behavior decision-making method for a human-computer interaction device provided in an embodiment of this application.

[0033] like Figure 1 As shown, the behavior decision-making method of this human-computer interaction device includes the following steps: In step S101, the enhanced target of the target human-computer interaction device is obtained.

[0034] Among them, the target human-computer interaction device is an intelligent agent that can perform human-computer interaction; the reinforcement target can be trained to be an intelligent agent that can both obtain higher rewards and obey natural language commands.

[0035] In step S102, a constrained objective function is constructed based on the reinforcement objective. The target human-computer interaction device is then iteratively trained using a preset reinforcement learning strategy and a preset learning strategy that mimics natural language responses, in order to optimize the objective function until the preset reinforcement conditions are met, thus obtaining the reinforced target human-computer interaction device.

[0036] Among them, the preset enhancement conditions can be set according to the actual situation without specific limitations.

[0037] It is understood that the embodiments of this application can model the above-mentioned problem based on the determination of reinforcement objectives as a constrained reinforcement learning problem. The objective of reinforcement learning is to maximize the reward function, and the constraint is that, given natural language instructions, the agent's behavior should be not significantly different from human behavior. To this end, the embodiments of this application propose a two-stage reinforced demonstration distillation (RED) algorithm to solve the above-mentioned constrained reinforcement learning problem. RED iteratively trains the agent through reinforcement learning and imitation learning. Through reinforcement learning, the agent learns how to obtain high rewards, while through imitation learning, the agent learns how humans obey instructions.

[0038] In this embodiment, a target human-computer interaction device is iteratively trained using a preset reinforcement learning strategy and a preset learning strategy that mimics natural language responses to optimize the objective function until a preset reinforcement condition is met, resulting in a reinforced target human-computer interaction device. This includes: training the target human-computer interaction device using the preset reinforcement learning strategy to obtain behavioral trajectories; training the target human-computer interaction device using the preset learning strategy that mimics natural language responses to mimic real human responses to natural language instructions, while simultaneously performing self-distillation on the obtained behavioral trajectories; constructing a trajectory set based on all behavioral trajectories obtained through iterative training, and storing the trajectory set and all learning response data in the target human-computer interaction device to obtain a reinforced target human-computer interaction device.

[0039] Specifically, embodiments of this application can incorporate real human responses to natural language into reinforcement learning training, modeling the entire problem as a constrained reinforcement learning problem:

[0040] in These are the parameters that the model needs to train. This represents the decision function of the agent, which will make decisions based on the state observed at time t. And the natural language given by humans Make a decision and decide on the action to be taken. .

[0041] In the above formula, It is the objective function optimized using reinforcement learning. When optimizing this objective function, no natural language is given to the agent as input. NA indicates that no instructions are given. At the current time step... Decisions are made solely based on the observed state of the human-computer interaction device. These are constraints, among which This is a dataset of real human responses to natural language. The actions of these humans are typically not optimal, but they reflect how humans obey natural language commands. Under this constraint, the embodiments of this application aim to train an agent that, while maximizing its reward function, can respond correctly to commands issued by natural language, much like a human.

[0042] To optimize the constrained objective function described above, this application proposes an iterative training framework. The training process can be divided into two stages, which are performed alternately during iteration. Here, k represents the model parameters at the k-th iteration. (1) Using reinforcement learning to learn winning strategies

[0043] This stage is consistent with conventional reinforcement learning, involving optimizing the objective function. The agent learns how to obtain more rewards. During this process, the agent is not constrained by any language. Simultaneously, embodiments of this application save the trajectories that yield higher rewards during this process; the set of these high-reward trajectories is represented as...

[0044] (2) Use imitation learning to enable the agent to learn to respond correctly to instructions issued by natural language.

[0045]

[0046] To enable the intelligent agent to correctly respond to natural language instructions, embodiments of this application can utilize imitation learning to learn how real humans respond to natural language from human datasets. Simultaneously, embodiments of this application also aim to ensure the intelligent agent retains winning strategies learned through reinforcement learning; therefore, embodiments of this application can also simultaneously process the behavioral trajectories saved during reinforcement learning. Self-distillation was performed.

[0047] In step S103, the enhanced target human-machine interaction device is used to identify the type of user instruction, and the target human-machine device is controlled to execute the corresponding optimal behavior decision based on the instruction type. When no user instruction is obtained, the target human-machine device is controlled to complete the task autonomously.

[0048] It is understood that the embodiments of this application enable the trained intelligent agent to: a) strictly follow instructions when a human issues a command, even if the goal of the command contradicts obtaining a higher reward; b) choose the optimal method to follow the command when a human issues a relatively abstract command with multiple behavioral patterns to achieve the effect of obeying the command, thereby obtaining a higher reward while obeying the command; and c) complete tasks autonomously and obtain a higher reward when no human issues any command.

[0049] In some embodiments, RED was evaluated in a strategy game. Through comprehensive experimental and human-computer interaction evaluations, it was found that RED does indeed respond better to natural language commands and achieves a higher win rate compared to other baseline algorithms.

[0050] According to the behavior decision-making method for human-computer interaction devices proposed in this application, through an iterative training framework, the trained agent a) should strictly follow an instruction when a human issues one, even if the goal of the instruction contradicts obtaining a higher reward; b) when a human issues a more abstract instruction, and multiple behavioral patterns exist to achieve the effect of obeying the instruction, the agent should choose the optimal way to obey the instruction, thereby obtaining a higher reward while obeying the instruction; c) when no human instruction is issued, the agent is capable of autonomously completing the task and obtaining a higher reward. This can help artificial intelligence understand the speaker's intention from human language and change its behavior accordingly to meet the speaker's diverse needs, while autonomously seeking better execution methods, thus laying the foundation for the widespread application of artificial intelligence in the future.

[0051] Next, the behavior decision-making device for a human-computer interaction device according to an embodiment of this application is described with reference to the accompanying drawings.

[0052] Figure 2 This is a block diagram of the behavior decision-making device of the human-computer interaction device according to an embodiment of this application.

[0053] like Figure 2 As shown, the behavior decision-making device 10 of the human-computer interaction device includes: an acquisition module 100, a training module 200, and a decision-making module 300.

[0054] The acquisition module 100 is used to acquire the reinforcement target of the target human-computer interaction device; the training module 200 is used to construct a constrained objective function based on the reinforcement target, and alternately iterate the target human-computer interaction device using a preset reinforcement learning strategy and a preset learning strategy that imitates natural language to make responses, so as to optimize the objective function until the preset reinforcement conditions are met, and obtain the reinforced target human-computer interaction device; the decision module 300 is used to identify the user instruction type using the reinforced target human-computer interaction device, control the target human-computer interaction device to execute the corresponding optimal behavior decision based on the instruction type, and control the target human-computer interaction device to complete the task autonomously when no user instruction is acquired.

[0055] In this embodiment, the training module 200 is used to: train the target human-computer interaction device using a preset reinforcement learning strategy to obtain behavioral trajectories; train the target human-computer interaction device using the preset reinforcement learning strategy to obtain behavioral trajectories; train the target human-computer interaction device using the preset learning strategy of mimicking natural language responses to mimic real human responses to natural language instructions, and simultaneously perform self-distillation on the obtained behavioral trajectories; construct a trajectory set based on alternating iterations of all behavioral trajectories, and store the trajectory set and all learning response data in the target human-computer interaction device to obtain a reinforced target human-computer interaction device.

[0056] Optionally, the formula for the preset reinforcement learning strategy is:

[0057] in, Describe the objective function. These are the parameters that the target model (i.e., the model of the decision function of the human-computer interaction device) needs to be trained on. These are the parameters of the model after the (k-1)th iteration. The learning rate represents the parameter update. express about The derivative, This represents the parameters obtained after reinforcement learning training in the k-th iteration.

[0058] Optionally, the formula for the learning strategy that mimics natural language responses is:

[0059] in, These are the parameters that the target model needs to train. These are the parameters of the model after the (k-1)th iteration. This represents the parameters obtained after reinforcement learning training in the k-th iteration. Indicates The decision function of a human-computer interaction device with parameters. A dataset representing the behavioral trajectories of real humans responding to natural language. The parameter is Decision function The collection of behavioral trajectories that successfully achieve the goal, where NA represents In collecting No natural language instructions are given at the time. Representing the trajectory The set goals were successfully achieved. The learning rate represents the parameter update. Represents the evaluation distance function Regarding parameters The gradient of, where The assessment focuses on behavioral trajectories. and On the union of, the parameter is The distance between the behavior generated by the decision function and the original behavior in the trajectory.

[0060] Optionally, the objective function for strengthening the target is:

[0061] in, These are the parameters used for model training. This represents the decision function of a human-computer interaction device, which determines the decision based on the state observed at time t. And the natural language given by humans Make a decision, For the actions to be taken, It is the objective function optimized by reinforcement learning, where NA represents the current time step without any instructions. Decisions are made solely based on the observed state of the human-computer interaction device. These are constraints. It is a dataset of real human responses to natural language.

[0062] It should be noted that the foregoing explanation of the behavior decision-making method embodiment for human-computer interaction devices also applies to the behavior decision-making device of the human-computer interaction device in this embodiment, and will not be repeated here.

[0063] According to the behavioral decision-making device for human-computer interaction proposed in this application, through an iterative training framework, the trained intelligent agent a) should strictly follow an instruction when a human issues one, even if the goal of the instruction contradicts obtaining a higher reward; b) when a human issues a more abstract instruction, and multiple behavioral patterns exist to achieve the effect of obeying the instruction, the intelligent agent should choose the optimal way to obey the instruction, thereby obtaining a higher reward while obeying the instruction; c) when no human issues any instruction, the intelligent agent is capable of autonomously completing tasks and obtaining a higher reward. Therefore, this can help artificial intelligence understand the speaker's intention from human language and change its behavior accordingly to meet the speaker's diverse needs, while autonomously seeking better execution methods, thus laying the foundation for the widespread application of artificial intelligence in the future.

[0064] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 301, the processor 302, and the computer program stored on the memory 301 and capable of running on the processor 302.

[0065] When the processor 302 executes the program, it implements the behavior decision-making method for the human-computer interaction device provided in the above embodiments.

[0066] Furthermore, electronic devices also include: Communication interface 303 is used for communication between memory 301 and processor 302.

[0067] The memory 301 is used to store computer programs that can run on the processor 302.

[0068] The memory 301 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0069] If the memory 301, processor 302, and communication interface 303 are implemented independently, then the communication interface 303, memory 301, and processor 302 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0070] Optionally, in a specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on a single chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through an internal interface.

[0071] Processor 302 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0072] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described behavior decision-making method for a human-computer interaction device.

[0073] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0074] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0075] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0076] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0077] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0078] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A behavior decision-making method for a human-computer interaction device, characterized in that, Includes the following steps: Acquire the enhanced target of the target human-computer interaction device; Based on the reinforcement objective, a constrained objective function is constructed. The target human-computer interaction device is then iteratively trained using a preset reinforcement learning strategy and a preset learning strategy that mimics natural language responses to optimize the objective function until the preset reinforcement conditions are met, resulting in a fully reinforced target human-computer interaction device. The formula for the preset reinforcement learning strategy is: in, Describe the objective function. These are the parameters that the target model needs to be trained on. The target model is the model of the decision function of the human-computer interaction device. These are the parameters of the model after the (k-1)th iteration. The learning rate represents the parameter update. express about The derivative, This represents the parameters obtained after reinforcement learning training in the k-th iteration; The formula for the preset learning strategy that mimics natural language responses is: in, These are the parameters that the target model needs to train. These are the parameters of the model after the (k-1)th iteration. This represents the parameters obtained after reinforcement learning training in the k-th iteration. A dataset representing the behavioral trajectories of real humans responding to natural language. The parameter is Decision function The collection of behavioral trajectories that successfully achieve the goal, where NA represents In collecting No natural language instructions are given at the time. Representing the trajectory The set goals were successfully achieved. The learning rate represents the parameter update. Represents the evaluation distance function Regarding parameters The gradient of, where The assessment focuses on behavioral trajectories. and On the union of, the parameter is The distance between the behavior generated by the decision function and the original behavior in the trajectory; The objective function for the enhanced target is: in, These are the parameters used in model training. Indicated by The decision function of the human-computer interaction device is defined by parameters. The human-computer interaction device will make decisions based on the state observed at time t. And the natural language given by humans Make a decision, For the actions to be taken, It is the objective function optimized by reinforcement learning, where NA represents the current time step without any instructions. Decisions are made solely based on the observed state of the human-computer interaction device. These are constraints; The enhanced target human-computer interaction device is used to identify the type of user command, and the target human-computer interaction device is controlled to execute the corresponding optimal behavior decision based on the command type. When no user command is obtained, the target human-computer interaction device is controlled to complete the task autonomously.

2. The method according to claim 1, characterized in that, The method of alternately and iteratively training the target human-computer interaction device using a preset reinforcement learning strategy and a preset learning strategy that mimics natural language responses to optimize the objective function until the preset reinforcement condition is met, thereby obtaining a fully reinforced target human-computer interaction device, includes: The target human-computer interaction device is trained using the preset reinforcement learning strategy to obtain the behavior trajectory; The target human-computer interaction device is trained to mimic real human responses to natural language commands using the preset learning strategy of mimicking natural language, and the obtained behavioral trajectory is self-distilled at the same time. A trajectory set is constructed by obtaining all behavioral trajectories through alternating iterations, and the trajectory set and all learning response data are stored in the target human-machine interaction device to obtain the enhanced target human-machine interaction device.

3. A behavior decision-making device for a human-computer interaction device, characterized in that, include: The acquisition module is used to acquire the enhanced targets of the target human-computer interaction device; The training module is used to construct a constrained objective function based on the reinforcement objective, and to iteratively train the target human-computer interaction device using a preset reinforcement learning strategy and a preset learning strategy that mimics natural language responses, in order to optimize the objective function until the preset reinforcement conditions are met, thus obtaining a reinforced target human-computer interaction device. The formula for the preset reinforcement learning strategy is: in, Describe the objective function. These are the parameters that the target model needs to be trained on. The target model is the model of the decision function of the human-computer interaction device. These are the parameters of the model after the (k-1)th iteration. The learning rate represents the parameter update. express about The derivative, This represents the parameters obtained after reinforcement learning training in the k-th iteration; The formula for the preset learning strategy that mimics natural language responses is: in, These are the parameters that the target model needs to train. These are the parameters of the model after the (k-1)th iteration. This represents the parameters obtained after reinforcement learning training in the k-th iteration. A dataset representing the behavioral trajectories of real humans responding to natural language. The parameter is Decision function The collection of behavioral trajectories that successfully achieve the goal, where NA represents In collecting No natural language instructions are given at the time. Representing the trajectory The set goals were successfully achieved. The learning rate represents the parameter update. Represents the evaluation distance function Regarding parameters The gradient of, where The assessment focuses on behavioral trajectories. and On the union of, the parameter is The distance between the behavior generated by the decision function and the original behavior in the trajectory; The objective function for the enhanced target is: in, These are the parameters used in model training. Indicated by The decision function of the human-computer interaction device is defined by parameters. The human-computer interaction device will make decisions based on the state observed at time t. And the natural language given by humans Make a decision, For the actions to be taken, It is the objective function optimized by reinforcement learning, where NA represents the current time step without any instructions. Decisions are made solely based on the observed state of the human-computer interaction device. These are constraints; The decision module is used to identify the type of user instruction using the enhanced target human-computer interaction device, control the target human-computer device to execute the corresponding optimal behavior decision based on the instruction type, and control the target human-computer device to complete the task autonomously when no user instruction is obtained.

4. The apparatus according to claim 3, characterized in that, The training module is used for: The target human-computer interaction device is trained using the preset reinforcement learning strategy to obtain the behavior trajectory; The target human-computer interaction device is trained to mimic real human responses to natural language commands using the preset learning strategy of mimicking natural language, and the obtained behavioral trajectory is self-distilled at the same time. A trajectory set is constructed by obtaining all behavioral trajectories through alternating iterations, and the trajectory set and all learning response data are stored in the target human-machine interaction device to obtain the enhanced target human-machine interaction device.

5. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the behavior decision method for a human-computer interaction device as described in any one of claims 1-2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the behavior decision method for the human-computer interaction device as described in any one of claims 1-2.

Images