Intelligent agent inference method and related device

Abstract

The invention provides an agent inference method and a related device. The method comprises the steps of obtaining information of a robot and an environment, and constructing a generation model of an intelligent agent according to the information of the robot and the environment; the intelligent agent interacts with the environment through the generative model, the generative model deduces the real state and outputs the prior state distribution, and the generative model is adjusted according to the prior state distribution to minimize the surprise degree; wherein the free energy is obtained according to the expected value of the logarithm of the KL divergence of the variation distribution and the prior state distribution and the quotient of the variation probability and the conditional probability, and the free energy is minimized to minimize the surprise degree; the intelligent agent has more efficient perception ability in an incomplete information environment, the model training efficiency is improved, environment interaction develops towards an interested state, and falling into a local optimal solution is avoided.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning, and more particularly to agent inference methods and related apparatus. Background Technology

[0002] For reinforcement learning in environments with incomplete information, in many real-world scenarios such as human-computer interaction, game theory, and decision-making in complex environments, agents often cannot obtain complete environmental information. Therefore, methods tailored to incomplete information are needed to help agents make optimal decisions in these situations. Currently, the main bottleneck of reinforcement learning methods with incomplete information lies in their low efficiency in perceiving the environment. Agents typically require a long period of exploration to obtain effective feedback, leading to slow training processes and a tendency to get trapped in local optima. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] The purpose of this application is to at least partially solve one of the technical problems existing in the related technologies. The embodiments of this application provide an intelligent agent inference method and related devices, enabling intelligent agents to have more efficient perception capabilities in environments with incomplete information.

[0005] An embodiment of the first aspect of this application provides an agent inference method, comprising: Acquire information about the robot and its environment, and construct a generative model of the intelligent agent based on this information; The agent interacts with the environment through a generative model, which infers the real state and outputs a prior state distribution. The agent adjusts the generative model according to the prior state distribution to minimize the surprise factor, which represents the difference between the prior environment model and the agent's perception of the environment. Wherein, adjusting the generative model according to the prior state distribution to minimize surprise factor includes: The free energy is obtained by taking the expected value of the logarithm of the quotient of the variational probability and the conditional probability, based on the KL divergence between the variational distribution and the prior state distribution. The free energy is minimized to minimize the surprise factor, whereby the free energy is the variational upper bound of the surprise factor.

[0006] According to certain embodiments of the first aspect of this application, the generative model of the intelligent agent includes an internal state space, an external state space, an action space, and a perception state space; the step of constructing the generative model of the intelligent agent based on the robot and environment model includes: The internal state space is obtained based on the map information determined by the robot's perception of the environment; The perception state space is obtained based on the robot's position; The external state space is obtained based on the robot's reasoning about the target point; The action space is obtained based on the action.

[0007] According to certain embodiments of the first aspect of this application, the degree of surprise is expressed as: ;in, Information observed by the agent. P ( o ) for the event The probability of occurrence To enhance the sense of surprise.

[0008] According to certain embodiments of the first aspect of this application, the free energy is expressed as: ;in, For free energy, s For environmental conditions, Information observed by the agent. C For the prior state set, For the agent's action strategy, It is a variational distribution. For the prior state distribution, Let KL be the KL divergence between the variational distribution and the prior state distribution. For variational probability, For conditional probability, Let be the expected value of the logarithm of the quotient of the variational probability and the conditional probability.

[0009] According to certain embodiments of the first aspect of this application, minimizing the free energy includes: Based on the agent's observations of the interaction between the agent and the environment, the prior state set is updated to minimize the expected free energy. The update formula for the prior state set is expressed as: ; This is the updated set of prior states. The prior state set before the update. For coefficients, For gradient operators, Let the free energy be the gradient of the prior state set before the update. For the agent's action strategy, For free energy.

[0010] According to certain embodiments of the first aspect of this application, during the process of updating the prior state set, an action strategy is used to make the environmental state approximate the prior state set, and an action that maximizes the conditional probability is selected according to the action strategy.

[0011] According to certain embodiments of the first aspect of this application, the action that maximizes the second conditional probability is selected based on the action strategy using the following formula: ; a For action, This is the second conditional probability. The action strategy of the intelligent agent.

[0012] According to a second aspect of this application, a robot performs task inference based on environmental information using an agent inference method as described in the first aspect of this application.

[0013] According to a third aspect of this application, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the agent inference method as described in the first aspect of this application.

[0014] According to a fourth aspect of this application, a computer storage medium stores computer-executable instructions for performing the agent inference method as described in the first aspect of this application.

[0015] The above scheme has at least the following beneficial effects: by acquiring information about the robot and the environment, a generative model of the agent is constructed based on this information; the agent interacts with the environment through the generative model, the generative model infers the real state and outputs the prior state distribution, and the generative model is adjusted according to the prior state distribution to minimize surprise; the free energy is obtained by taking the expected value of the logarithm of the quotient of the variational distribution and the prior state distribution and the variational probability and the conditional probability, and the free energy is minimized to minimize surprise; this enables the agent to have more efficient perception capabilities in environments with incomplete information, improves model training efficiency, and allows environmental interactions to develop towards states of interest, avoiding getting trapped in local optima. Attached Figure Description

[0016] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0017] Figure 1 This is a flowchart illustrating the steps of an agent inference method. Figure 2 This is a schematic diagram of the intelligent agent framework; Figure 3 This is a schematic diagram illustrating the application scenario of robots in Frozen Lake; Figure 4 This is a schematic diagram of the generative model. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] For reinforcement learning in environments with incomplete information, in many real-world scenarios such as human-computer interaction, game theory, and decision-making in complex environments, agents often cannot obtain complete environmental information. Therefore, methods tailored to incomplete information are needed to help agents make optimal decisions in these situations. Currently, the main bottleneck of reinforcement learning methods with incomplete information lies in their low efficiency in perceiving the environment. Agents typically require a long period of exploration to obtain effective feedback, leading to slow training processes and a tendency to get trapped in local optima.

[0021] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0022] Embodiments of this application provide a robot. Based on the theoretical framework of the free energy principle, an active inference agent is constructed; the behavioral pattern of the active inference agent is: minimizing free energy. This makes the active inference agent highly adaptable to environments with incomplete information.

[0023] For example, the robot can be applied to the Frozen Lake application scenario. Of course, in other embodiments, the robot can also be applied to other scenarios.

[0024] In the application scenarios of Frozen Lake, the incomplete information is constructed as follows: two locations are selected, and the treasure point is repeatedly switched between the two locations. After a certain number of games, the treasure point will switch to the other of the two locations.

[0025] The robot is equipped with an intelligent agent that performs task inferences based on environmental information according to the following agent inference method. This agent inference method is based on an agent framework suitable for environments with incomplete information, and its construction method is based on free energy from the field of biological intelligence.

[0026] Reference Figure 2The agent is equipped with a generative model. The agent observes data from the external environment and uses the observed data as input to the generative model. The generative model outputs an action strategy based on the observed data. The agent takes action according to the action strategy, and the action affects the environment, which in turn affects the observed data. This process is repeated iteratively.

[0027] Reference Figure 1 The agent inference method includes the following steps: Step S100: Obtain information about the robot and the environment, and construct a generative model of the intelligent agent based on the information about the robot and the environment; In step S200, the agent interacts with the environment through a generative model. The generative model infers the real state and outputs the prior state distribution. The generative model is adjusted according to the prior state distribution to minimize the surprise factor.

[0028] Perform POMDP modeling on the environment, define the POMDP model of the environment, and give the state space, action space and reward value.

[0029] Reference Figure 3 In the Frozen Lake application scenario, the current position of the robot is represented by an integer in the range [0, 63]. The 64 integers correspond one-to-one with each grid point in order from top to bottom and from left to right.

[0030] The robot's motion space is an integer in the interval [0,3], representing movement to the left, down, right, and up, respectively.

[0031] The reward value is set as follows: if you reach the treasure point, the reward value is 1; if you reach the ordinary ice square, the reward value is 0; if you reach the ice cave, the reward value is -1.

[0032] The true location of the treasure point is represented by a two-dimensional array. If the treasure point is located at 47, the array is [1,0]; if the treasure point is located at 63, the array is [0,1]. Initially, the robot does not know the true value of the array.

[0033] Using Markov blankets, we construct generative models of intelligent agents.

[0034] The definition of a Markov blanket is: Given the universal set of random variables A, for a given random variable X, there is a set A such that A = f(x) * ... The smallest set B is the Markov blanket of X.

[0035] Generative models contain 4-tuples ,in S represents the external state space, which cannot be directly perceived by the brain; A represents the perceptual state space, which can be directly perceived by the brain; R represents the action space; and R represents the internal state space, which is the brain's intrinsic property.

[0036] All states can be divided into internal and external states, separated by a Markov blanket comprising a perceptual state space and an action space. This segmentation intuitively describes the brain's actions and perceptions; the self-organization of internal states corresponds to the brain's perceptions, while the action space couples the brain states with the external states.

[0037] The generative model of an intelligent agent includes an internal state space, an external state space, an action space, and a perception state space. Constructing a generative model of an intelligent agent based on a model of the robot and its environment includes the following steps: obtaining the internal state space based on map information determined by the robot's perception of the environment; obtaining the perception state space based on the robot's position; obtaining the external state space based on the robot's reasoning about the target point; and obtaining the action space based on the actions.

[0038] Reference Figure 4 Specifically, for robots in the Frozen Lake application scenario, the perception state space is their own position in the maze; the hidden state (external state) space is a probability distribution representing the reasoning about the location of the treasure point; the action space is the action space under the POMDP paradigm; and the internal state space is the map information of the entire maze, representing the robot's cognition of the environment.

[0039] The brain is essentially a "reasoning machine," possessing a priori model of the world. The behavioral patterns of organisms in the real world can always be explained as an attempt to minimize the discrepancy between the priori world model and their perception of the world. This discrepancy is called surprise level in the field of biological intelligence, and the formula for calculating surprise level is: .in, Information observed by the agent. P ( o ) for the event The probability of occurrence To enhance the sense of surprise.

[0040] If the probability is high, it means that the event in the world is within the agent's expectations, so the surprise value is low; if the probability is low, it means that the event in the world is beyond the agent's expectations, so the surprise value is high.

[0041] The agent interacts with the POMDP environment through a generative model. By continuously adjusting the generative model, it minimizes surprise. For observations with incomplete information, the generative model can infer the true state and provide a prior marginal probability distribution (belief).

[0042] The variational upper bound of surprise level is called free energy. Therefore, minimizing free energy can achieve the goal of minimizing surprise level.

[0043] Free energyF It is a measure of free energy that describes a future trajectory, derived from observations with incomplete information, and can be used to determine action strategies. The marginal likelihood function is estimated.

[0044] Free energy is expressed as: ;in, For free energy, s This refers to the environmental state. Information observed by the intelligent agent. C This is a set of prior states, which typically contains certain events of interest. The action strategy of the intelligent agent. It is a variational distribution. This represents the prior state distribution. This is the KL divergence between the variational distribution and the prior state distribution, which measures the inconsistency between the two types of information. Let be the variational probability. This represents the conditional probability. This is the expected value of the logarithm of the quotient of the variational probability and the conditional probability, which measures the inconsistency between the two types of information.

[0045] Essentially, C This represents a certain expectation of the agent. By minimizing the free energy, the agent's action strategy will approach a state that moves as close as possible to the desired outcome. C It is developing in that direction.

[0046] For example, in the Frozen Lake application scenario, the proactive inferential agent contains a prior state set C, representing the agent's inferences about a two-dimensional array. The hidden state is represented as... The array contains 64 positions and the locations of 2 treasure points. The hidden state is the agent's belief, where integers in the interval [0, 63] represent the agent's perceived precise location. The second two-dimensional array... That is C This refers to the agent's inference about the location of the treasure. The information observed by the agent... o It contains its exact location and a reward value. The location in the hidden state is an identity mapping to the location in the observed information. That is, if the agent believes it is at position 10, then 10 is its true location, and it observes that its actual location is also 10. The reward value depends on... The truth value of .

[0047] right C and Perform iterative updates to minimize the free energy.

[0048] Minimizing the free energy includes the following steps: updating the prior state set based on the agent's observations of the agent's interaction with the environment in order to minimize the expected free energy.

[0049] The update formula for the prior state set is expressed as: ; This is the updated set of prior states. The prior state set before the update. For coefficients, For gradient operators, Let the free energy be the gradient of the prior state set before the update. For the agent's action strategy, For free energy.

[0050] The gradient calculation method described above can simultaneously optimize action strategies and inference information.

[0051] According to the formula for calculating free energy, in Of the free energy, only the term and variables C Relevant. After one round of simulation, the prior state set is iterated based on the observations. C In essence, C This represents the agent's inferences or expectations regarding unknown factors in the environment. During the process of updating the prior state set, action policies are used... Make the environment state s To the prior state set C Approximation, based on the action strategy, selects the conditional probability. Maximize Action a .

[0052] Choose the action that maximizes the second conditional probability based on the action strategy using the following formula: ; a For action, This is the second conditional probability. The action strategy of the intelligent agent.

[0053] For example, for robots in the Frozen Lake application scenario, the explicit reward value is replaced with the inverse of the surprise factor. The optimal action aims to minimize surprise. When the agent is located at the treasure point or the ice cave, the value is either 1 or... 1; When the agent is in a normal ice grid, the value is 0. This represents the agent's prior preferences: to reach the target point to obtain a good score, rather than wandering in ordinary ice squares. The proactive inference agent transforms the POMDP into an inference problem: as the location of the treasure point changes, the proactive inference agent needs to continuously update its state set C of interest in order to make the optimal action. C is initialized to [0.5, 0.5]. After a game, if the result indicates that the treasure point is at 47, C will be updated to [1, 0]; if the treasure point is at 63, C will be updated to [0, 1]. The probability of observing information o is essentially a conditional probability based on C. In each game, the agent first predicts the location of the target point from state C, and then outputs the optimal action.

[0054] For traditional agents, the Beta distribution is used as the prior probability. The Beta distribution is a commonly used probability model in uncertain situations. The parameters are initialized to Beta(1,1), denoted as... To update the vector, the discount factor is set to 0.9. In a game, the prior probability is assigned the value Beta. If the treasure is at location 47, then Updated to: If the treasure is at location 63, then Updated to: ; Updated Forming the posterior probability Beta of this game This is also the prior probability for the next game. In each game, the agent uses the prior probability Beta... Generate a sample to predict the location of the treasure point, and then output the optimal action.

[0055] In the Frozen Lake scenario, each agent plays 1000 rounds of a maze game. The initial location of the treasure point is 47, at the [missing information - likely a typo]. During each game, the location of the treasure point will change. In each game, 1 point is awarded if and only if the agent reaches the treasure point within the maximum number of moves; otherwise, 0 points are awarded.

[0056] Table 1. Comparison of Traditional Intelligent Agents and Active Inference Intelligent Agents Provided in the Examples

[0057] Table 1 compares the performance of the two types of agents. It can be seen that the traditional agent has an average score of 0.923, demonstrating good perception of environmental changes and effectively updating the parameters of the Beta distribution within a limited number of game rounds, accurately predicting the treasure location in subsequent games. The proactive inference agent has an average score of 0.990, outperforming the traditional agent and effectively updating the prior state within a very small number of game rounds.C In subsequent games, this allows for accurate prediction of treasure location locations.

[0058] Embodiments of this application provide an electronic device. The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described agent inference method.

[0059] This electronic device can be any smart terminal, including computers.

[0060] In general, for the hardware structure of electronic devices, the processor can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solutions provided in the embodiments of this application.

[0061] The memory can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called and executed by the processor.

[0062] Input / output interfaces are used to implement information input and output.

[0063] The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0064] The bus transmits information between various components of a device, such as the processor, memory, input / output interfaces, and communication interfaces. The processor, memory, input / output interfaces, and communication interfaces communicate with each other within the device via the bus.

[0065] Embodiments of this application provide a computer storage medium. The computer storage medium stores computer-executable instructions for performing the agent inference method described above.

[0066] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0067] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0068] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms. Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0072] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. An agent inference method, characterized in that, include: Acquire information about the robot and its environment, and construct a generative model of the intelligent agent based on this information; The agent interacts with the environment through a generative model, which infers the real state and outputs a prior state distribution. The agent adjusts the generative model according to the prior state distribution to minimize the surprise factor, which represents the difference between the prior environment model and the agent's perception of the environment. Wherein, adjusting the generative model according to the prior state distribution to minimize surprise factor includes: The free energy is obtained by taking the expected value of the logarithm of the quotient of the variational probability and the conditional probability, based on the KL divergence between the variational distribution and the prior state distribution. The free energy is minimized to minimize the surprise factor, whereby the free energy is the variational upper bound of the surprise factor.

2. The agent inference method according to claim 1, characterized in that, The generative model of the intelligent agent includes an internal state space, an external state space, an action space, and a perception state space; the construction of the generative model of the intelligent agent based on the robot and environment model includes: The internal state space is obtained based on the map information determined by the robot's perception of the environment; The perception state space is obtained based on the robot's position; The external state space is obtained based on the robot's reasoning about the target point; The action space is obtained based on the action.

3. The agent inference method according to claim 1, characterized in that, The level of surprise is expressed as: ;in, P(o) represents the information observed by the agent. The probability of occurrence To enhance the sense of surprise.

4. The agent inference method according to claim 1, characterized in that, The free energy is expressed as: ;in, Let s be the free energy and s be the environmental state. C represents the information observed by the agent, and C represents the prior state set. For the agent's action strategy, It is a variational distribution. For the prior state distribution, Let KL be the KL divergence between the variational distribution and the prior state distribution. For variational probability, For conditional probability, Let be the expected value of the logarithm of the quotient of the variational probability and the conditional probability.

5. The agent inference method according to claim 1, characterized in that, Minimizing free energy includes: Based on the agent's observations of the interaction between the agent and the environment, the prior state set is updated to minimize the expected free energy. The update formula for the prior state set is expressed as: ; This is the updated set of prior states. The prior state set before the update. For coefficients, For gradient operators, Let the free energy be the gradient of the prior state set before the update. For the agent's action strategy, For free energy.

6. The agent inference method according to claim 5, characterized in that, During the process of updating the prior state set, the environmental state is made to approximate the prior state set through action strategies, and the action that maximizes the conditional probability is selected according to the action strategies.

7. The agent inference method according to claim 6, characterized in that, The action that maximizes the second conditional probability is selected according to the action strategy using the following formula: ; a represents action, This is the second conditional probability. The action strategy of the intelligent agent.

8. A robot, characterized in that, The agent inference method according to any one of claims 1 to 7 performs task inference based on environmental information.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the agent inference method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The system stores computer-executable instructions for performing the agent inference method as described in any one of claims 1 to 7.

Images