Behavior decision method and device for human-computer interaction, equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a behavior decision-making method and device for human-computer interaction, equipment and a storage medium, wherein the method comprises the following steps: acquiring at least one target event in a human-computer interaction scene; randomly searching a target reward function space of the at least one target event to obtain a plurality of behavior preferences, generating a bias strategy set based on the plurality of behavior preferences, sequentially selecting different types of bias strategies from the bias strategy set according to a preset behavior index, and constructing a strategy pool with strategy diversity according to the different types of bias strategies and a preset reward strategy; training an adaptive strategy of a target intelligent agent by randomly sampling a strategy from the strategy pool, deciding a behavior of human-computer interaction with the target intelligent agent based on the adaptive strategy, and obtaining a behavior decision-making result. Thus, the embodiment of the application can realize strong interaction and collaborative generalization centered on people, and group behavior decision-making generalization of large-scale multi-intelligent agents in an open scene.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for behavioral decision-making in human-computer interaction. Background Technology

[0002] Despite decades of research, building intelligent agents capable of interacting, collaborating, and assisting humans remains a long-term AI challenge. Classical approaches are typically model-based, building an effective behavioral model from human data and using this model for planning. While this approach has achieved considerable success, this model-based paradigm requires an expensive and time-consuming data collection process, which is ill-suited to the complex problems that current AI technologies address, and may also raise privacy concerns.

[0003] Recently, multi-agent reinforcement learning (MARL) has emerged as a promising approach for solving many challenging decision-making problems. Particularly in competitive environments, AI trained using MARL algorithms based on self-play (SP) has outperformed human experts across various fields. This empirical evidence suggests that new directions in developing powerful AI can involve direct collaboration with humans in a similar "model-free" manner, namely through self-play.

[0004] Unlike zero-sum games where a simple Nash equilibrium strategy suffices, a significant problem with training cooperative agents through self-play is convention overfitting. Because cooperative games offer a large number of possible optimal strategies, agents trained through self-play can easily converge to a specific optimum and make decisions solely based on a particular behavioral pattern (i.e., consensus) of their co-trainers, resulting in poor generalization to unknown partners. Summary of the Invention

[0005] This application provides a behavior decision-making method, device, electronic device, and storage medium for human-computer interaction, which can achieve strong human-centered interaction and collaborative generalization, as well as the generalization of group behavior decision-making of large-scale multi-agents in open scenarios.

[0006] The first aspect of this application provides a behavior decision-making method for human-computer interaction, comprising the following steps: acquiring at least one target event in a human-computer interaction scenario; randomly searching the target reward function space of the at least one target event to obtain multiple behavior preferences; generating a bias policy set based on the multiple behavior preferences; sequentially selecting different types of biased policies from the bias policy set according to a preset behavior index; constructing a policy pool with diversified policies based on the different types of biased policies and the preset reward policy; randomly sampling policies from the policy pool to train an adaptive policy for a target agent; and making a behavior decision for human-computer interaction with the target agent based on the adaptive policy to obtain a behavior decision result.

[0007] Optionally, the step of randomly searching the target reward function space of the at least one target event to obtain multiple behavioral preferences, and generating a set of biased policies based on the multiple behavioral preferences, includes: sampling multiple samples of the feature weights of the target reward function space, and obtaining a set of hidden reward functions based on the multiple samples; for each hidden reward function, finding an approximate Nash equilibrium solution of the hidden potential Markov game derived from it through self-play, and obtaining a set of biased policies covering multiple behavioral preferences based on the approximate Nash equilibrium solution.

[0008] Optionally, the step of sequentially selecting different types of biased strategies from the biased strategy set according to preset behavioral indicators includes: obtaining an initial strategy from the initial pool; and selecting a biased strategy different from the initial strategy from the biased strategy set according to preset behavioral indicators.

[0009] Optionally, before randomly searching the target reward function space of the at least one target event, the method includes: identifying event features of the at least one target event; and generating the target reward function space based on the linear function of the event features and the bounds of the feature weights.

[0010] A second aspect of this application provides a behavior decision-making device for human-computer interaction, comprising: an acquisition module for acquiring at least one target event in a human-computer interaction scenario; a construction module for randomly searching the target reward function space of the at least one target event to obtain multiple behavior preferences, generating a bias policy set based on the multiple behavior preferences, sequentially selecting different types of biased policies from the bias policy set according to a preset behavior index, and constructing a policy pool with diversified policies based on the different types of biased policies and the preset reward policy; and a decision-making module for randomly sampling policies from the policy pool to train an adaptive policy for a target agent, and making a decision on the behavior of human-computer interaction with the target agent based on the adaptive policy, thereby obtaining a behavior decision result.

[0011] Optionally, the construction module is further configured to: sample multiple samples of the feature weights of the target reward function space, and obtain a set of hidden reward functions based on the multiple samples; for each hidden reward function, find an approximate Nash equilibrium solution of the hidden potential energy Markov game derived from it through self-play, and obtain a set of biased policies covering multiple behavioral preferences based on the approximate Nash equilibrium solution.

[0012] Optionally, the construction module is further configured to: obtain an initial policy from the initial pool; and select a biased policy that is different from the initial policy from the biased policy set according to a preset behavioral metric.

[0013] Optionally, the building module is further configured to: identify event features of the at least one target event before randomly searching the target reward function space of the at least one target event; and generate the target reward function space based on the linear function of the event features and the bounds of the feature weights.

[0014] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the behavior decision method for human-computer interaction as described in the above embodiments.

[0015] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the behavior decision method for human-computer interaction as described in the above embodiments.

[0016] Therefore, this application has at least the following beneficial effects:

[0017] Next-generation reinforcement learning-based artificial intelligence requires multi-machine, multi-person collaborative interaction in complex interactive scenarios. Traditional multi-agent reinforcement learning employs a self-game algorithm system, optimizing the reward function of the group or individual agents using traditional reinforcement learning algorithms. This traditional framework does not meet the generalization requirement, especially in scenarios involving human interaction. Traditional self-games do not consider diversity, often resulting in specialized strategies. However, humans choose diverse behaviors, and due to different preferences, human strategies can deviate significantly from the highest reward strategy. Therefore, to achieve efficient collaboration with humans, it is necessary to find multiple possible optimal strategies, and even non-optimal human-like strategies, in the self-game solution. Next-generation multi-agent reinforcement learning algorithms must fully consider the existence and potential preferences of humans, theoretically incorporating human behavioral preferences into the self-game, and adaptively adjusting their behavior according to human intentions to achieve human-centered strong interaction and collaborative generalization. Furthermore, based on diversity theory, it should achieve generalization of group behavior decisions for large-scale multi-agents in open scenarios.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0020] Figure 1 This is a flowchart of a behavior decision-making method for human-computer interaction provided according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of the adaptive agent training process provided according to an embodiment of this application;

[0022] Figure 3 This is an example diagram of a behavior decision-making device for human-computer interaction provided according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0025] To address the problems in the background technology, a two-stage framework has been proposed in related technologies. First, a diverse policy pool consisting of multiple self-playing trained policies is constructed, which may cover different consensuses. Then, an adaptive policy is further trained for this policy pool.

[0026] While this two-stage framework has been experimentally successful, it suffers from a fundamental flaw: even if the policy pool prevents consensus from overfitting, each policy trained in the self-game within the pool remains an optimal or suboptimal solution to the fixed reward function specified by the underlying cooperative game. This implies a crucial generalization assumption: that any tester will aim to maximize the game reward. This assumption often fails when working with humans. Human behavior has been extensively studied in cognitive science, economics, and game theory. Scientific research shows that even with a clear objective, the human potential function can be significantly biased, suggesting that human behavior may be influenced by an unknown reward function that is entirely different from the game reward. This fact reveals the algorithmic limitations of existing self-game-based methods.

[0027] Therefore, embodiments of this application propose a behavior decision-making method, apparatus, electronic device, and storage medium for human-computer interaction. The behavior decision-making method, apparatus, electronic device, and storage medium for human-computer interaction of embodiments of this application will be described below with reference to the accompanying drawings.

[0028] Specifically, Figure 1 This is a flowchart illustrating a behavior decision-making method for human-computer interaction provided in an embodiment of this application.

[0029] like Figure 1 As shown, the behavioral decision-making method for human-computer interaction includes the following steps:

[0030] In step S101, at least one target event in the human-computer interaction scenario is acquired.

[0031] In step S102, the target reward function space of at least one target event is randomly searched to obtain multiple behavioral preferences. A set of biased strategies is generated based on the multiple behavioral preferences. Different types of biased strategies are selected from the set of biased strategies according to preset behavioral indicators. A strategy pool with diversified strategies is constructed based on different types of biased strategies and preset reward strategies.

[0032] Among them, preset behavioral indicators and preset reward strategies can be selected according to the actual situation, without specific limitations.

[0033] Understandably, this application's embodiments can introduce biased human assumptions into the self-play stage of a two-stage framework, proposing a hidden-utility self-play (HSP) algorithm. HSP explicitly models human biases by adding a hidden reward function to the self-play training objective. This generalized modeling approach can express any non-adaptive human policy. Furthermore, a tractable approximation of the hidden reward function space is proposed, and a random search is performed on this approximation space when constructing the policy pool in the first stage. Therefore, the enhanced policy pool can capture a wide range of human biases beyond the conventional and game reward-related skill levels. Consequently, the final adaptive policy derived in the second stage can be more adaptable to unknown humans.

[0034] In this embodiment of the application, before randomly searching the target reward function space of at least one target event, the method includes: identifying event features of at least one target event; and generating the target reward function space based on the linear function of the event features and the bounds of the feature weights.

[0035] Specifically, regarding the event-based reward function space, the core lies in designing a suitable hidden reward function space. The effective reward space is very large. Inspired by the fact that human preferences are typically event-centric, this application's embodiments can represent the hidden reward function space as a linear function of event features, i.e. C max It is the boundary of the feature weight w, and This refers to the occurrence of different game events when a combined action (a1, a2) is taken in state s.

[0036] In this embodiment of the application, a target reward function space of at least one target event is randomly searched to obtain multiple behavioral preferences. A set of biased policies is generated based on the multiple behavioral preferences. This includes: sampling multiple samples of the feature weights of the target reward function space, and obtaining a set of hidden reward functions based on the multiple samples; for each hidden reward function, finding an approximate Nash equilibrium solution of the hidden potential Markov game derived from it through self-play, and obtaining a set of biased policies covering multiple behavioral preferences based on the approximate Nash equilibrium solution.

[0037] Specifically, a set of bias strategies is generated: In this embodiment of the application, a set of different behaviors can be obtained simply by performing a random search on the feature weights w, as follows:

[0038] First, N samples {w} are sampled for the feature weights. (i)} i∈[N] ,in From a set of values ​​C j Uniform sampling yields a set of hidden reward functions. For each hidden reward function, an approximate Nash equilibrium solution for the hidden potential Markov game derived from it can be found through self-play. The above process generates a policy pool covering a wide range of behavioral preferences.

[0039] In this embodiment of the application, different types of biased strategies are selected sequentially from the biased strategy set according to preset behavioral indicators, including: obtaining the initial strategy in the initial pool; and selecting a biased strategy different from the initial strategy from the biased strategy set according to the preset behavioral indicators.

[0040] Specifically, regarding policy selection: This embodiment of the application notes that the policy pool typically contains many similar policies. This is because the same policy can reach its optimum under a set of reward functions, and duplicate policies only slow down training without contributing to learning adaptive policies. For more efficient training, this embodiment of the application uses a behavioral metric, i.e., based on the diversity of events, only policies different from the initial pool are retained. This embodiment of the application employs a greedy method to progressively select policies, as shown in the pseudocode: Algorithm 1: Greedy Policy Selection, S <- {i_0}, where i_0 is randomly selected from 0 to N, and N is a positive integer.

[0041] In practical applications, the policy pool consists of half biased policies and the other half MEP policies trained under game rewards. This makes the overall pool more biased towards game rewards, thereby improving experience performance.

[0042] In step S103, an adaptive policy for the target agent is obtained by randomly sampling policies from the policy pool and training the policy. Based on the adaptive policy, the target agent is used to make human-computer interaction decisions, and the behavior decision results are obtained.

[0043] It is understood that embodiments of this application can train a final adaptive policy by randomly sampling policies from a pool, and embodiments of this application can add the ID of each biased policy as an additional feature to the observations of the value function network. The pseudocode for training is as follows: Figure 2 As shown, the details are as follows:

[0044] (1) Train pi_w^(i) and pi_a^(i) under the sampled reward function R_W^(i), where i is a positive integer;

[0045] (2) Run Algorithm 1 to extract K strategies;

[0046] (3) Initialize strategy pj_A;

[0047] (4) Repeatedly use pi_A and the sampled pi_w^(i) to collect trajectory data;

[0048] (5) Update the strategy pi_A until the actual number of iterations meets the set number of iterations, and end the training.

[0049] In some embodiments, this application evaluates HSP in the popular human-computer collaboration simulator Overcooked, conducting comprehensive ablation studies and comparing it with baseline algorithms that do not explicitly model human bias. Experimental results show that the HSP method achieves superior performance through a behavioral model learned from human data. Furthermore, this application also constructs a set of manually written biased policies, ensuring they are sufficiently differentiated from the policy pool; experiments show that HSP achieves a greater performance improvement than the baseline algorithm. Finally, human-computer interaction experiments were conducted, and the collected feedback indicates that human experimenters consistently perceived HSP as having a higher degree of assistance.

[0050] The behavioral decision-making method for human-computer interaction proposed in the embodiments of this application fully considers the existence and possible preferences of humans, and comprehensively introduces human behavioral preferences into self-game from a theoretical level. It adaptively adjusts its own behavior according to human intentions to achieve strong human-centered interaction and collaborative generalization. Furthermore, based on diversity theory, it realizes the generalization of group behavioral decision-making of large-scale multi-agents in open scenarios.

[0051] Next, referring to the accompanying drawings, a behavior decision-making device for human-computer interaction proposed according to an embodiment of this application is described.

[0052] Figure 3 This is a block diagram of a behavior decision device for human-computer interaction according to an embodiment of this application.

[0053] like Figure 3 As shown, the behavior decision-making device 10 for human-computer interaction includes: an acquisition module 100, a construction module 200, and a decision-making module 300.

[0054] The acquisition module 100 is used to acquire at least one target event in the human-computer interaction scenario; the construction module 200 is used to randomly search the target reward function space of at least one target event to obtain multiple behavioral preferences, generate a set of biased policies based on the multiple behavioral preferences, select different types of biased policies from the set of biased policies according to preset behavioral indicators, and construct a policy pool with diversified policies based on different types of biased policies and preset reward policies; the decision module 300 is used to randomly sample policies from the policy pool to train the adaptive policies of the target agent, and make decisions on the behavior of human-computer interaction with the target agent based on the adaptive policies to obtain the behavior decision results.

[0055] In this embodiment of the application, the construction module 200 is further configured to: sample multiple samples of the feature weights of the target reward function space, and obtain a set of hidden reward functions based on the multiple samples; for each hidden reward function, find the approximate Nash equilibrium solution of the hidden potential energy Markov game derived from it through self-play, and obtain a set of biased policies covering multiple behavioral preferences based on the approximate Nash equilibrium solution.

[0056] In this embodiment of the application, the construction module 200 is further configured to: obtain the initial policy in the initial pool; and select a biased policy that is different from the initial policy from the biased policy set according to a preset behavioral index.

[0057] In this embodiment of the application, the construction module 200 is further configured to: identify event features of at least one target event before randomly searching the target reward function space of at least one target event; and generate the target reward function space based on the linear function of the event features and the bounds of the feature weights.

[0058] It should be noted that the foregoing explanation of the behavioral decision-making method embodiment for human-computer interaction also applies to the behavioral decision-making device for human-computer interaction in this embodiment, and will not be repeated here.

[0059] The behavioral decision-making device for human-computer interaction proposed in the embodiments of this application fully considers the existence and possible preferences of humans, and comprehensively introduces human behavioral preferences into self-game from a theoretical level. It adaptively adjusts its own behavior according to human intentions, realizing strong interaction and collaborative generalization centered on humans. Furthermore, based on diversity theory, it realizes the generalization of group behavioral decision-making of large-scale multi-agents in open scenarios.

[0060] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0061] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0062] When the processor 402 executes the program, it implements the behavior decision-making method for human-computer interaction provided in the above embodiments.

[0063] Furthermore, electronic devices also include:

[0064] Communication interface 403 is used for communication between memory 401 and processor 402.

[0065] The memory 401 is used to store computer programs that can run on the processor 402.

[0066] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0067] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0068] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0069] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.

[0070] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described behavior decision-making method for human-computer interaction.

[0071] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0072] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0073] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0074] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0075] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0076] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A behavioral decision-making method for human-computer interaction, characterized in that, Includes the following steps: Acquire at least one target event in a human-computer interaction scenario; Randomly searching the target reward function space of the at least one target event yields multiple behavioral preferences. A biased policy set is generated based on these preferences. This includes sampling multiple samples of the feature weights of the target reward function space and obtaining a set of hidden reward functions based on these samples. For each hidden reward function, an approximate Nash equilibrium solution of the hidden potential Markov game derived from it is found through self-play. A biased policy set covering multiple behavioral preferences is obtained based on the approximate Nash equilibrium solution. Different types of biased policies are sequentially selected from the biased policy set according to preset behavioral indicators. A policy pool with diverse strategies is constructed based on the different types of biased policies and the preset reward strategy. Before randomly searching the target reward function space of the at least one target event, the process includes identifying the event characteristics of the at least one target event and generating the target reward function space based on the linear function of the event characteristics and the boundaries of the feature weights. An adaptive policy for the target agent is obtained by randomly sampling policies from the policy pool and training them. Based on the adaptive policy, the behavior of human-computer interaction with the target agent is determined, and the behavior decision result is obtained.

2. The method according to claim 1, characterized in that, The step of sequentially selecting different types of biased strategies from the set of biased strategies based on preset behavioral indicators includes: Retrieve the initial policy from the initial pool; Based on preset behavioral indicators, a biased strategy that is different from the initial strategy is selected from the set of biased strategies.

3. A behavioral decision-making device for human-computer interaction, characterized in that, include: The acquisition module is used to acquire at least one target event in a human-computer interaction scenario; A construction module is used to randomly search the target reward function space of the at least one target event to obtain multiple behavioral preferences, and generate a set of biased policies based on the multiple behavioral preferences. This includes sampling multiple samples of the feature weights of the target reward function space, obtaining a set of hidden reward functions based on the multiple samples, finding an approximate Nash equilibrium solution of the hidden potential Markov game derived from each hidden reward function through self-play for each hidden reward function, and obtaining a set of biased policies covering multiple behavioral preferences based on the approximate Nash equilibrium solution; sequentially selecting different types of biased policies from the set of biased policies according to preset behavioral indicators, and constructing a policy pool with diversified strategies based on the different types of biased policies and preset reward strategies. Before randomly searching the target reward function space of the at least one target event, this includes identifying the event features of the at least one target event, and generating the target reward function space based on the linear function of the event features and the boundaries of the feature weights. The decision module is used to randomly sample policies from the policy pool to train an adaptive policy for the target agent, and to make decisions on the human-computer interaction behavior with the target agent based on the adaptive policy, thereby obtaining the behavior decision result.

4. The apparatus according to claim 3, characterized in that, The construction module is further used to: obtain the initial policy in the initial pool; and select a biased policy that is different from the initial policy from the biased policy set according to a preset behavioral index.

5. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the behavioral decision-making method for human-computer interaction as described in any one of claims 1-2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the behavioral decision-making method for human-computer interaction as described in any one of claims 1-2.

Images