Intelligent agent control method, intelligent agent control device and electronic equipment

Abstract

The invention provides an intelligent agent control method and device, electronic equipment and a non-instantaneous computer readable storage medium. The control method comprises the steps that state data and executable action data of an intelligent agent are acquired; determining the next execution action of the agent according to the state data and the executable action data; and controlling the intelligent agent based on the next execution action. According to the embodiment of the invention, the executable action data is used as the input data for determining the next execution action, so that the problem of action space change caused by multi-environment change of the intelligent agent is solved, and the method can adapt to complex and changeable scenes.

Description

Technical Field

[0001] The present invention relates to the field of intelligent bodies, and in particular to a control method and control device for an intelligent body, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Reinforcement learning is an important branch of machine learning and has achieved remarkable results in recent years. Unlike traditional supervised learning, reinforcement learning not only relies on existing data samples, but also obtains feedback through interaction with the environment during the process of exploration and trial and error. With this, reinforcement learning has gradually demonstrated excellent adaptability and decision-making capabilities in the fields of robot control, game AI, and autonomous driving, giving machines the potential to perform tasks in complex environments.

[0003] However, although reinforcement learning has demonstrated strong generalization and adaptability, it still faces challenges in dealing with dynamic and mutable state spaces and action spaces. Traditional reinforcement learning algorithms usually assume that the environment is relatively fixed and the state and action sets are static. However, in the real world, the state space and action space of many problems change over time. For example, a robot walking in different environments will cause changes in the state space, and different tasks or goals will also increase or decrease the action set. Since the structure of the neural network is relatively fixed, when the state space or action space changes, if you want to adapt to such changes, you need to modify the network structure and retrain the model. Therefore, in these cases, it is difficult for traditional reinforcement learning models to adapt to such dynamic changes. Summary of the invention

[0004] The present application aims to propose a control method and control device for an intelligent agent, an electronic device and a non-transient computer-readable storage medium to solve the problem that the existing technology cannot adapt to the changes in the action space caused by the changes in multiple environments of the intelligent agent.

[0005] According to one aspect of the present application, a method for controlling an intelligent agent is proposed, comprising: acquiring state data and executable action data of the intelligent agent; determining a next execution action of the intelligent agent based on the state data and the executable action data; and controlling the intelligent agent based on the next execution action.

[0006] According to some embodiments, determining the next execution action of the agent based on the state data and the executable action data includes: fusing the state data and the executable action data to generate a fused feature vector; and determining the next execution action of the agent using the fused feature vector.

[0007] According to some embodiments, the state data and the executable action data are fused to generate a fused feature vector, including: performing feature extraction and fusion processing on the state data to obtain a situation feature vector of the state data; performing feature extraction and fusion processing on the executable action data to obtain an action feature vector of the executable action data; and fusing the situation feature vector and the action feature vector to generate the fused feature vector.

[0008] According to some embodiments, using the fused feature vector to determine the next action to be performed by the agent includes: determining the action probability distribution of the executable action data based on the fused feature vector; and determining the next action to be performed by the agent based on the action probability distribution.

[0009] According to some embodiments, the control method further comprises: performing a value estimation on the state data according to the situation feature vector to provide strategy support for the next execution action of the agent.

[0010] According to some embodiments, the state data is value estimated based on the situation feature vector to provide strategy support for the next execution action of the intelligent agent, including: performing nonlinear transformation processing on the situation feature vector; using the situation feature vector that has been processed by nonlinear transformation to perform value estimation to provide strategy support for the strategy training of the intelligent agent.

[0011] According to some embodiments, before determining the next execution action of the agent based on the state data and the executable action data, the control method further includes: adding agent type data to the state data.

[0012] According to one aspect of the present application, a control device for an intelligent agent is proposed, comprising: a data acquisition unit for acquiring state data and executable action data of the intelligent agent; an execution action determination unit for determining the next execution action of the intelligent agent based on the state data and the executable action data; and a control unit for controlling the intelligent agent based on the next execution action.

[0013] According to one aspect of the present application, an electronic device is provided, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor executes the control method as described in any of the previous embodiments.

[0014] According to one aspect of the present application, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor executes the control method described in any of the previous embodiments.

[0015] According to the embodiments of the present application, by using executable action data as input data for determining the next execution action, the problem of changes in the action space of the intelligent agent due to changes in multiple environments is solved, thereby being able to adapt to complex and changing scenarios.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. By describing the exemplary embodiments in detail with reference to the drawings, the above and other objectives, features and advantages of the present invention will become more apparent.

[0018] Figure 1 A flow chart of a control method for an intelligent agent according to an exemplary embodiment of the present application is shown.

[0019] Figure 2 A flow chart of another method for controlling an intelligent agent according to an exemplary embodiment of the present application is shown.

[0020] Figure 3 A schematic diagram of a multi-head attention principle according to an example embodiment of the present application is shown.

[0021] Figure 4 A block diagram of a control device of an intelligent agent according to an exemplary embodiment of the present application is shown.

[0022] Figure 5 An electronic device according to an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same figures in the drawings represent the same or similar parts, and thus their repeated description will be omitted.

[0024] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices or operations may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0025] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0026] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0027] The specific embodiments according to the present invention are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flow chart of a control method of an intelligent agent according to an exemplary embodiment of the present application is shown. Figure 1 The control method shown includes steps S101, S103 and S105. Figure 1 Taking as an example, a control method of an intelligent agent according to an exemplary embodiment of the present application is described in detail.

[0029] like Figure 1 As shown, in step S101, the state data and executable action data of the agent are obtained.

[0030] According to the embodiments of the present application, the intelligent agent of the present application refers to an agent that can perceive the environment and take actions to achieve specific goals, and has autonomy, adaptability and interaction capabilities to complete tasks or adapt to complex environments. In specific embodiments, the intelligent agent includes but is not limited to a robot, a drone, an automated system or a game character.

[0031] In the embodiments of the present application, due to changes in the external environment, the action space (including action data) that the same agent can operate and the corresponding state space (including state data) after the action is executed will also change. Taking the action space as an example, when the agent walks to a crossroads, the corresponding action space includes walking forward, walking right, walking backward, and walking left. When the agent walks to a three-way intersection, the corresponding action space includes walking forward, walking right, and walking backward.

[0032] In step S103, the next action to be performed by the agent is determined based on the state data and the executable action data.

[0033] According to an embodiment of the present application, in step S103, first, the state data and the executable action data are fused to generate a fused feature vector; then, the fused feature vector is used to determine the next execution action of the agent.

[0034] In a specific embodiment, when the state data and the executable action data are fused to generate a fused feature vector, first, feature extraction and fusion processing are performed on the state data to obtain a situation feature vector of the state data; and feature extraction and fusion processing are performed on the executable action data to obtain an action feature vector of the executable action data; finally, the situation feature vector and the action feature vector are fused to generate a fused feature vector. In this embodiment, by using the action state data, the sensitivity and adaptability of the agent to the change of the state data when determining the next execution action are improved.

[0035] According to an embodiment of the present application, when using the fused feature vector to determine the next action to be performed by the agent, the action probability distribution of the executable action data is first determined based on the fused feature vector; then, the next action to be performed by the agent is determined based on the action probability distribution. For example, the action corresponding to the largest probability value in the action space is selected as the next action to be performed.

[0036] According to an embodiment of the present application, when multiple agents need to be controlled, before step S103, agent type data needs to be added to the state data to identify the agent and determine the next execution action of the corresponding agent.

[0037] In step S105, the agent is controlled based on the next execution action.

[0038] According to an embodiment of the present application, based on the next execution action, the agent is controlled so that the agent performs the determined next execution action.

[0039] In some embodiments, after step S105, it is also necessary to collect state data after the agent performs the next determined execution action to provide data preparation for subsequent execution actions.

[0040] According to an embodiment of the present application, Figure 1 The method shown also includes performing a value estimation on the state data according to the situation feature vector, wherein the value estimation is used to perform a global value estimation on the current state of the current agent, so as to evaluate the next execution action determined in step S103, so as to provide policy support for the policy training of the agent.

[0041] In a specific embodiment, when performing value estimation on state data based on a situation feature vector, first, the situation feature vector is subjected to nonlinear transformation processing; then, the situation feature vector subjected to nonlinear transformation processing is used to perform value estimation to provide strategic support for the next execution action of the intelligent agent.

[0042] according to Figure 1 The illustrated embodiment solves the problem of changes in the action space of the intelligent agent due to changes in multiple environments by using executable action data as input data for determining the next execution action, thereby being able to adapt to complex and changing scenarios.

[0043] Figure 2 A flow chart of another method for controlling an intelligent agent according to an exemplary embodiment of the present application is shown. Figure 2 The first unit shown is to extract and fuse the memory features of the input state data to obtain the situation feature vector and value estimation. The second unit first extracts and fuses the action data, and further extracts and fuses the features with the situation feature vector, and finally obtains the logical value corresponding to each action in the executable action space after linear processing, which is used to generate the probability value corresponding to each action, so as to realize the selection of the next execution action. Among them, the first unit includes N identical modules and fully connected layers stacked. Among them, each module includes a multi-head attention module and a corresponding residual connection module, as well as a forward propagation and a corresponding residual connection unit. Among them, the residual connection module includes execution layer regularization and residual connection operations. Among them, layer regularization helps to stabilize distribution changes and improve the generalization ability of the network, while residual connection ensures that the original information can still be effectively transmitted after deep stacking, which helps to prevent gradient disappearance and information loss.

[0044] like Figure 2 As shown, after the state information is input, the situation embedding vector is obtained and added to the type vector representing the type information corresponding to the agent.

[0045] The added vector is then input into the multi-head attention module for processing. In this embodiment, multiple different attention heads are introduced at the same level at the same time, and each attention head performs independent attention calculations and is stacked through multiple self-attention mechanisms to enhance its feature extraction capabilities. That is, given an input sequence, each head generates three vectors of query, key, and value based on the input data, and performs attention calculations. Through the parallel calculation of multiple heads, the multi-head attention module can extract the relationship features between each position in the sequence from different angles, as well as the association between the state information of the captured state input agent, and generate a fused feature representation. This attention mechanism can perform multi-dimensional weighted analysis of the state information of the agent in space and time, which is equivalent to global feature extraction of the input state information. That is, the multi-head attention mechanism can capture the diverse patterns and dependencies of the input data. Each attention head will focus on different features in the input sequence. Some heads may pay more attention to local relationships, while others may pay more attention to remote dependencies, so that the model has higher flexibility and expression ability when capturing information. Finally, the outputs of all heads are concatenated and linearly transformed to form the final multi-head attention output, which provides the model with the ability to understand information from multiple angles and improves the richness of feature expression.

[0046] For example, given a query, key, and value matrix, they are converted into h sub-queries, sub-keys, and sub-values, which are then fed into the corresponding self-attention models respectively, and finally the heads are connected and combined with the final weight matrix, as shown in Formula (1).

[0047] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (1)

[0048] in Attention is the self-attention function. Corresponding to query, key and value matrices respectively, W O is the corresponding head weight value, Concat is the combination function, and MultiHead(Q,K,V) is the fusion result output.

[0049] like Figure 3 As shown, it is a schematic diagram of a multi-head attention principle according to an exemplary embodiment of the present application, such as Figure 3 The multi-head attention shown in the figure uses each position in the sequence to pay attention to other positions in the same sequence, dynamically adjusting the weights of each element in the sequence to obtain a more global feature representation. For example, given a sequence of elements x1, x2, ..., x n, the self-attention mechanism calculates the similarity between each pair of elements to determine the weight of each element in the sequence.

[0050] Specifically, first, three vectors, query (Q), key (K), and value (V), are generated according to each element in the input sequence; then, the dot product is calculated between the query vector of each element and the key vector of other elements in the sequence. The higher the similarity, the stronger the correlation between the two positions. The similarity result is processed by Softmax to obtain a weight distribution, as shown in formula (2), which realizes the dynamic adjustment of the weight of the feature vector, so that the model can flexibly focus on the key information in the situation; then, these weights are used to weight and sum each value vector, and finally form a new representation of each element. This method allows the model to dynamically adjust the attention of each position to other parts of the sequence, so that the elements in the sequence can be directly associated with each other, thereby capturing longer-distance dependencies in the feature representation, which is suitable for global information extraction in long sequences. This mechanism enables the model to not only focus on local information, but also extract effective features from the overall structure of the sequence.

[0051]

[0052] Among them, Q, K, and V correspond to the query matrix, key matrix, and value matrix respectively, d k Compute the dot product of the query vector of each element with the key vectors of other elements in the sequence, with T being the transpose operation.

[0053] like Figure 3 As shown in the figure, after the output data of the multi-head attention module passes through the residual connection module, nonlinear changes are performed in the feedforward network (including feedforward propagation and corresponding residual connections) to capture the complex relationship between parameters.

[0054] In this implementation, the situation feature vector is obtained through the feedforward network. The situation feature vector is further extracted and abstracted through the fully connected layer of the first module, and the nonlinear characteristics of the model are increased through the activation function to capture complex patterns and relationships, and obtain value estimates to evaluate the next execution action to provide policy support for the strategy training of the intelligent agent. For example, in the subsequent policy update, the direction of the gradient update is guided. At the same time, the situation feature vector is used as the input of the second module.

[0055] like Figure 2As shown in FIG. 1 , similar to the first module, the second module also includes N identical modules and fully connected layers. In the second module, first, the optional action set is vectorized, and each action is represented as a vector and then input into the multi-head attention module for feature extraction and fusion, so that the action features are correlated with each other, thereby achieving efficient information transmission in the action sequence, so that each action can refer to the feature representation of other actions, thereby obtaining more comprehensive feature information, and making the representation of action features more comprehensive and smooth.

[0056] According to an embodiment of the present application, when feature extraction and fusion are performed on action vectors, when invalid actions need to be processed, they are eliminated from the candidate action group to fully utilize the variable characteristics of the action space, thereby filtering out invalid actions and reducing the computational load of action reasoning.

[0057] After that, after being processed by the corresponding residual module, it is used as the query vector input of the next multi-head attention module. The situation feature vector generated in the first module is used as the key (K) and value (V) vector input of the multi-head attention module to achieve the feature fusion of the agent state information and action information, so that the action feature can flexibly refer to the situation feature information, improve the sensitivity and adaptability to the change of the agent state, and provide data support for the final action decision. Through this layer, the model can use the situation information as background or context, so as to more accurately calculate the role and value of each action in the current environment, and provide support for subsequent action selection. And through the corresponding residual connection module, the stability of the model and the integrity of feature transfer are ensured.

[0058] like Figure 3 As shown, the fused feature matrix passes through a fully connected layer, and the output of the fully connected layer generates a logical value for each action, which is used to indicate the selection tendency of the action. In this embodiment, the obtained logical value needs to be normalized by softmax to obtain the probability distribution corresponding to each action. Through this processing method, an accurate action probability distribution can be generated based on the current situation and action features, so as to determine the next execution action based on the generated action probability distribution. For example, the action corresponding to the largest probability value in the action space is selected as the next execution action.

[0059] In the embodiment of the present application, the calculated value estimate can be used to evaluate the determined next execution action, thereby providing strategic support for the determination of the next execution action of the agent. In a specific embodiment, the PPO algorithm is used to update the strategy of the agent, and the value estimate is used as input data during the strategy update.

[0060] exist Figure 2In the embodiment shown, the state information is feature extracted through the first module, and the action set is deeply integrated with the state feature in the second module to generate an action probability distribution that meets the situation requirements, thereby strengthening the prominent representation of the action feature, making the intelligent agent more robust in the uncertain action space, and providing strong support for the decision-making of the intelligent agent. In addition, when the network architecture does not change, since the action set can be arbitrarily added or reduced when the environment changes, this embodiment has stronger scalability and generalization, and can better cope with complex and changing scenarios.

[0061] In this embodiment, when performing multi-head attention calculations, it does not rely on the sequence order, and can process all elements in the sequence at the same time, thereby supporting highly parallel calculations and significantly speeding up the training speed. This parallel computing function also decouples the elements in the sequence, and there is no fixed requirement for the number of elements in the sequence, that is, as long as the shape of the elements is the same, the number of elements can be increased or decreased arbitrarily, providing an architectural foundation for the scalability of the state space and action space.

[0062] The above mainly introduces the embodiments of the present application from the perspective of the method. Those skilled in the art should easily appreciate that, in combination with the operations or steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Those skilled in the art can use different ways to implement the described functions for each specific operation or method, and such implementation should not be considered to be beyond the scope of the present application.

[0063] The following describes the device embodiments of the present application. For details not described in the device embodiments of the present application, reference can be made to the method embodiments of the present application.

[0064] Figure 4 A block diagram of a control device of an intelligent body according to an exemplary embodiment of the present application is shown. Figure 4 The control device shown includes a data acquisition unit 401, an execution action determination unit 403 and a control unit 405. The data acquisition unit 401 is used to acquire the state data and executable action data of the agent; the execution action determination unit 403 is used to determine the next execution action of the agent according to the state data and the executable action data; and the control unit 405 is used to control the agent based on the next execution action.

[0065] Figure 5 An electronic device according to an exemplary embodiment of the present application is shown. Figure 5 The electronic device 200 according to this embodiment of the present application is described. Figure 5The electronic device 200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0066] like Figure 5 As shown, the electronic device 200 is in the form of a general computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0067] The storage unit stores program codes, which can be executed by the processing unit 210, so that the processing unit 210 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 210 can execute the following: Figure 1 The method shown in .

[0068] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 2201 and / or a cache memory unit 2202 , and may further include a read-only memory unit (ROM) 2203 .

[0069] The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0070] Bus 230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0071] The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 200, and / or communicate with any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 250. Furthermore, the electronic device 200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0072] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation method of the present application.

[0073] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0074] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0075] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0076] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.

[0077] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0078] According to an embodiment of the present application, a computer program is provided, including a computer program or an instruction. When the computer program or the instruction is executed by a processor, the method described above can be executed.

[0079] This application significantly improves the accuracy and dynamic response capability of network load management through the collaborative work of terminals and servers. Compared with the prior art, this application no longer relies on the terminal to make independent base station selection decisions, but instead uses the server to integrate multi-terminal data, generate the optimal strategy in real time, and push it in advance to the terminal that is about to enter the relevant area. This method avoids decision conflicts between terminals, reduces the frequency of data blocking, and realizes dynamic balancing of base station loads, making network resource allocation more intelligent. In addition, terminal devices can significantly reduce poor network experience caused by data blocking and improve user satisfaction. At the same time, by intelligently distributing network loads, the overload of network infrastructure can be effectively reduced. The more terminals are used, the greater the benefits.

[0080] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present invention, the specific implementation methods and application scope of the present invention, all belong to the scope of protection of the present invention. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A control method for an intelligent agent, characterized in that: include: Obtaining state data and executable action data of the agent; Determine the next execution action of the agent according to the state data and the executable action data; Based on the next execution action, the agent is controlled.

2. The control method according to claim 1, characterized in that: Determining the next execution action of the agent according to the state data and the executable action data includes: Fusing the state data and the executable action data to generate a fused feature vector; The fused feature vector is used to determine the next execution action of the agent.

3. The control method according to claim 2, characterized in that: The state data and the executable action data are fused to generate a fused feature vector, including: Performing feature extraction and fusion processing on the state data to obtain a situation feature vector of the state data; Performing feature extraction and fusion processing on the executable action data to obtain an action feature vector of the executable action data; The posture feature vector and the action feature vector are fused to generate the fused feature vector.

4. The control method according to claim 3, characterized in that: Determining the next execution action of the agent by using the fused feature vector includes: Determining an action probability distribution of the executable action data according to the fused feature vector; Determine the next action to be performed by the agent according to the action probability distribution.

5. The control method according to claim 3, characterized in that: Also includes: The value of the state data is estimated according to the situation feature vector to provide strategic support for the next execution action of the agent.

6. The control method according to claim 5, characterized in that: The state data is value estimated according to the situation feature vector to provide strategy support for the next execution action of the agent, including: Performing nonlinear transformation processing on the situation feature vector; The situation feature vector processed by nonlinear transformation is used to perform value estimation to provide strategic support for determining the next execution action of the intelligent agent.

7. The control method according to claim 1, characterized in that: Before determining the next execution action of the agent according to the state data and the executable action data, the control method further includes: Add agent type data to the state data.

8. A control device for an intelligent agent, characterized in that: include: A data acquisition unit, used to acquire state data and executable action data of the agent; An execution action determination unit, used to determine the next execution action of the agent according to the state data and the executable action data; A control unit is used to control the agent based on the next execution action.

9. An electronic device, comprising: processor; as well as A memory storing a computer program, which, when executed by the processor, enables the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 12.

Images