Data processing method and apparatus, electronic device, computer readable medium

Abstract

The present disclosure provides a data processing method and device, and relates to the technical fields of artificial intelligence, automatic driving and the like. A specific implementation of the method comprises: taking an environment state of an environment in which an intelligent agent is located at a current time as a selected state; determining a set number of environment states in sequence before the selected state; determining a representation formula for estimating a local intrinsic dimension of the selected state based on the selected state and the set number of environment states, the local intrinsic dimension being a mathematical quantity for measuring the dimension of a state space; and calculating an intrinsic reward for guiding the action of the intelligent agent based on the representation formula. This implementation improves the performance of reinforcement learning.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the fields of artificial intelligence, autonomous driving, and other related technologies, and in particular to data processing methods and apparatus, electronic devices, computer-readable media, and computer program products. Background Technology

[0002] Currently, mainstream exploration methods based on intrinsic rewards in sparse reward environments are mainly based on self-supervised methods. Self-supervised methods typically use a trained deep learning model to record the state space explored by the agent. When the agent makes a choice again, it can distinguish the state space that has been experienced many times based on the records of the deep learning model, thus avoiding repeated exploration. However, current self-supervised methods still have many shortcomings, such as the inability to perform quantitative analysis and the lack of consideration for the characteristics of reinforcement learning itself. Summary of the Invention

[0003] Embodiments of this disclosure provide data processing methods and apparatus, electronic devices, computer-readable media, and computer program products.

[0004] In a first aspect, embodiments of this disclosure provide a data processing method, the method comprising: taking the environmental state of the environment in which the agent is located at the current moment as a selected state; determining a predetermined number of environmental states sequentially prior to the selected state; determining a representation for estimating the local intrinsic dimension of the selected state based on the selected state and the predetermined number of environmental states, wherein the local intrinsic dimension is a mathematical quantity for measuring the dimension of the state space; and calculating an intrinsic reward guiding the agent's actions based on the representation.

[0005] In some embodiments, the above-mentioned calculation of the intrinsic reward guiding the agent's actions based on the representation formula includes: estimating the number of times the agent accesses the selected state to obtain a pseudo-count value of the selected state; calculating the local intrinsic dimension estimate of the selected state based on the pseudo-count value and the representation formula; and using the local intrinsic dimension estimate as the intrinsic reward guiding the agent's actions.

[0006] In some embodiments, the above-mentioned estimation of the number of times the agent accesses the selected state to obtain a pseudo-count value of the selected state includes: estimating the probability of the selected state occurring within a set time period using a trained first density model to obtain a first density value, wherein the first density model is trained based on a set number of environmental states; estimating the probability of the selected state occurring within a set time period using a trained second density model to obtain a second density value, wherein the second density model is trained based on the selected state and a set number of environmental states; and obtaining a pseudo-count value of the selected state based on the first density value and the second density value.

[0007] In some embodiments, obtaining the pseudo-count value of the selected state based on the first density value and the second density value includes: calculating the information gain of the first density value and the second density value; and calculating the pseudo-count value of the selected state based on the information gain.

[0008] In some embodiments, the above-mentioned calculation of the local intrinsic dimension estimate of the selected state based on the pseudo-count value and the representation includes: splitting the environmental state corresponding to the representation into pseudo-count value selected states and difference value environmental states, the difference being the difference between a set number and the pseudo-count value; splitting the representation into expressions including a first representative expression and a second representative expression, wherein the first representative expression corresponds to the pseudo-count value selected states and the second representative expression corresponds to the difference value environmental states; and calculating the second representative expression based on the difference value environmental states to obtain the local intrinsic dimension estimate of the selected state.

[0009] In some embodiments, the above representation is:

[0010]

[0011] Among them, s t+1 To select a state, S B =[s t-1+1 s t-k+2 , ..., s t-1 s t [r] represents the sequence of k sequential environmental states preceding the selected state. i (s t+1 S B ) represents the environmental state sequence S B In the middle, away from the selected state s t+1 From the i-th environment state among the k most recent environment states to the selected state s t+1 distance; r max (s t+1 S B ) represents the environmental state sequence S B In the middle, away from the selected state s t+1 Among the most recent k environmental states and the selected state s t+1 The furthest environmental state to the selected state s t+1 The distance between and , where k and i are natural numbers, and i ≤ k.

[0012] In some embodiments, the above representation is:

[0013]

[0014] Among them, s t+1 To select a state, S B =[s t-k+1 s t-k+2 , ..., st-1 s t ] represents the sequence of k sequential environmental states prior to the selected state, ψ(s) t+1 ) represents the selection feature for the selected state, r i (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the context of the k closest environmental features to the selected feature, the distance from the furthest environmental feature to the selected feature is calculated, where the environmental feature represents the environmental state. max (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the equation, the distance from the selected feature to the k closest environmental features is the distance from the selected feature to the environmental feature furthest from the selected feature, where k and i are natural numbers and i ≤ k.

[0015] In some embodiments, the selected features and environmental features are obtained through a trained feature transformation model. The feature transformation model is trained using the following steps: performing feature transformation on the environmental states adjacent to the selected state in the sequence of selected and environmental states to obtain selected features and adjacent features; sending the environmental states adjacent to the selected state to the agent to obtain the behavioral state output by the agent; obtaining the predicted value of the selected features based on the behavioral state and adjacent features; obtaining the predicted value of the behavioral state based on the adjacent features and selected features; and adjusting the parameters of the feature transformation network based on the predicted value of the selected features, the predicted value of the behavioral state, and the behavioral state to obtain the feature transformation model.

[0016] In some embodiments, the intelligent agent includes an autonomous vehicle, and the environmental state includes the operating state of the autonomous vehicle; the method further includes determining the behavioral state of the autonomous vehicle based on the selected state and intrinsic reward.

[0017] Secondly, embodiments of this disclosure provide a data processing apparatus, comprising: a selection unit configured to take the environmental state of the environment in which the agent is located at the current moment as a selection state; a determination unit configured to determine a predetermined number of environmental states sequentially before determining the selection state; a dimension representation unit configured to determine a representation for estimating the local intrinsic dimension of the selection state based on the selection state and the predetermined number of environmental states, wherein the local intrinsic dimension is a mathematical quantity for measuring the dimension of the state space; and a calculation unit configured to calculate an intrinsic reward guiding the agent's actions based on the representation.

[0018] In some embodiments, the dimension representation unit includes: an estimation subunit configured to estimate the number of times the agent accesses the selected state to obtain a pseudo-count value of the selected state; a calculation subunit configured to calculate a local intrinsic dimension estimate of the selected state based on the pseudo-count value and the representation formula; and a subunit configured to use the local intrinsic dimension estimate as an intrinsic reward to guide the agent's actions.

[0019] In some embodiments, the estimation subunit includes: a first estimation module configured to estimate the probability of a selected state occurring within a set time period using a trained first density model to obtain a first density value, wherein the first density model is trained based on a set number of environmental states; a second estimation module configured to estimate the probability of a selected state occurring within a set time period using a trained second density model to obtain a second density value, wherein the second density model is trained based on the selected state and a set number of environmental states; and a obtaining module configured to obtain a pseudo-count value of the selected state based on the first density value and the second density value.

[0020] In some embodiments, the above-mentioned obtaining module includes: a gain submodule configured to calculate the information gain of the first density value and the second density value; and a calculation submodule configured to calculate the pseudo-count value of the selected state based on the information gain.

[0021] In some embodiments, the above-mentioned calculation subunit includes: a state splitting module, configured to split the environmental state corresponding to the representation into pseudo-count value selected states and difference value environmental states, the difference being the difference between a set number and the pseudo-count value; a formula splitting module, configured to split the representation into an expression including a first representative formula and a second representative formula, wherein the first representative formula corresponds to the pseudo-count value selected states and the second representative formula corresponds to the difference value environmental states; and a calculation module, configured to calculate the second representative formula based on the difference value environmental states to obtain the local intrinsic dimension estimate of the selected states.

[0022] In some embodiments, the above representation is:

[0023]

[0024] Among them, s t+1 To select a state, S B =[s t-k+1 s t-k+2 , ..., s t-1 s t [r] represents the sequence of k sequential environmental states preceding the selected state. i (s t+1 S B ) represents the environmental state sequence S BIn the middle, away from the selected state s t+1 From the i-th environment state among the k most recent environment states to the selected state s t+1 distance; r max (s t+1 S B ) represents the environmental state sequence S B In the middle, away from the selected state s t+1 Among the most recent k environmental states and the selected state s t+1 The furthest environmental state to the selected state s t+1 The distance between and , where k and i are natural numbers, and i ≤ k.

[0025] In some embodiments, the above representation is:

[0026]

[0027] Among them, s t+1 To select a state, S B =[s t-k+1 s t-k+2 , ..., s t-1 s t ] represents the sequence of k sequential environmental states prior to the selected state, ψ(s) t+1 ) represents the selection feature for the selected state, r i (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the context of the k closest environmental features to the selected feature, the distance from the furthest environmental feature to the selected feature is calculated, where the environmental feature represents the environmental state. max (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the equation, the distance from the selected feature to the k closest environmental features is the distance from the selected feature to the environmental feature furthest from the selected feature, where k and i are natural numbers and i ≤ k.

[0028] In some embodiments, the selected features and environmental features are obtained through a trained feature transformation model. The feature transformation model is trained using the following steps: performing feature transformation on the environmental states adjacent to the selected state in the sequence of selected and environmental states to obtain selected features and adjacent features; sending the environmental states adjacent to the selected state to the agent to obtain the behavioral state output by the agent; obtaining the predicted value of the selected features based on the behavioral state and adjacent features; obtaining the predicted value of the behavioral state based on the adjacent features and selected features; and adjusting the parameters of the feature transformation network based on the predicted value of the selected features, the predicted value of the behavioral state, and the behavioral state to obtain the feature transformation model.

[0029] In some embodiments, the intelligent agent includes an autonomous vehicle, and the environmental state includes the operating state of the autonomous vehicle; the device is further configured to determine the behavioral state of the autonomous vehicle based on the selected state and intrinsic reward.

[0030] Thirdly, embodiments of this disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any implementation of the first aspect.

[0031] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0032] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any implementation of the first aspect.

[0033] The data processing method and apparatus provided in the embodiments of this disclosure first take the current environmental state of the agent's environment as the selected state; second, determine a predetermined number of environmental states sequentially prior to the selected state; third, based on the selected state and the predetermined number of environmental states, determine a representation for estimating the local intrinsic dimension of the selected state, where the local intrinsic dimension is a mathematical quantity that measures the dimension of the state space; finally, based on the representation, calculate the intrinsic reward guiding the agent's actions. Thus, when the agent interacts with the environment, it estimates the dimension of its current state space, and based on the estimated dimension of the state space, determines the agent's intrinsic reward, thereby encouraging the agent to explore higher-dimensional state spaces, helping the agent to better explore the environment, and improving the efficiency of reinforcement learning. Attached Figure Description

[0034] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0035] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0036] Figure 2 This is a flowchart of an embodiment of the data processing method according to the present disclosure;

[0037] Figure 3 This is a schematic diagram of the feature model training framework structure according to an embodiment of this disclosure;

[0038] Figure 4a This is a schematic diagram of an external real reward according to an embodiment of the present disclosure;

[0039] Figure 4b This is a schematic diagram of an intrinsic reward according to an embodiment of the present disclosure;

[0040] Figure 5 This is a schematic diagram of the structure of an embodiment of the data processing apparatus according to the present disclosure;

[0041] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0042] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] Figure 1 An exemplary system architecture 100 to which the data processing methods of this disclosure can be applied is shown.

[0045] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the data processing methods of this application can be applied.

[0046] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, typically including wireless communication links, etc.

[0047] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications, such as instant messaging tools and email clients, can be installed on terminal devices 101, 102, and 103.

[0048] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be data acquisition devices with communication and control functions for collecting environmental status. These data acquisition devices can communicate with server 105. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned terminals. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to collect environmental status) or as a single software program or software module. No specific limitations are made here.

[0049] Server 105 can be a server that provides various services, such as a reward server that supports agents on terminal devices 101, 102, and 103. The reward server can analyze and process relevant information from each terminal in the network and feed back the processing results (such as intrinsic rewards) to the terminal devices.

[0050] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0051] It should be noted that the data processing method provided in the embodiments of this disclosure is generally executed by server 105.

[0052] Reinforcement learning (RL), also known as reward learning, evaluation learning, or reinforcement learning, is a paradigm and methodology in machine learning used to describe and solve problems where an agent learns strategies to maximize rewards or achieve specific goals during interactions with its environment. In the exploration problem of reinforcement learning for autonomous driving, environments can generally be divided into sparse reward environments and non-sparse reward environments. A sparse reward environment refers to an environment where the reward signal for the agent's actions in reinforcement learning is typically very scarce. Only in rare cases can the agent, within a few steps, encounter the reward signal by completing a series of specific actions. For a long period prior, the agent receives no feedback from the environment to guide its actions, thus failing to effectively adjust its strategy and increase its chances of obtaining further reward signals, ultimately leading to a predicament. In non-sparse reward environments, the environment can usually provide the agent with more concentrated and effective reward signals as feedback. This allows the agent to effectively adjust its behavioral strategy based on the currently collected reward signals each time it trains itself. The adjusted strategy can usually help the agent obtain more and denser reward signals, and these further reward signals, in turn, help the agent to better adjust its strategy, establishing a virtuous cycle.

[0053] For complex robot control tasks, current reinforcement learning methods typically require adding a great deal of domain knowledge to the specific task and meticulously designing the reward function and training method in order to achieve satisfactory performance. When the agent enters a new environment, it usually needs to be redesigned according to the task itself.

[0054] To address the aforementioned shortcomings, the data processing method disclosed herein proposes an intrinsic reward based on the state space dimension for sparse reward environments: at each time step, the local intrinsic dimension of the state space surrounding the agent is estimated, and the intrinsic reward of the agent during the exploration of the environment is determined based on the estimated dimension. This helps the agent to better explore the environment and improves the efficiency of reinforcement learning. Figure 2 The diagram illustrates a flow 200 of an embodiment of a data processing method according to the present disclosure, the data processing method comprising the following steps:

[0055] Step 201: Select the current environmental state of the agent's environment as the selected state.

[0056] In this embodiment, the agent is an agent running a reinforcement learning method. After performing reinforcement learning, the agent sends the information to the controlled object (e.g., ...). Figure 1The terminal devices 101, 102, and 103 in the diagram implement actions to maximize and effectively control the controlled object. In practical applications, reinforcement learning for intelligent agents faces a significant challenge: the agent may not receive sufficient and effective rewards, or it may receive sparse rewards, leading to slow and ineffective learning. It's important to note that the data processing method operates on an entity that can be either the aforementioned intelligent agent or an entity outside the agent. Calculating intrinsic rewards through this entity can effectively support the agent's actions. For example, under the sparse reward conditions of autonomous driving, providing effective intrinsic rewards helps the agent execute effective actions towards the controlled object.

[0057] In this embodiment, the selected state can be the environmental state of the environment in which the agent is located during reinforcement learning. The environmental state varies depending on the task the agent performs. For example, for autonomous driving tasks, the agent's environmental state can include the vehicle's operating state (e.g., speed, fuel consumption) and surrounding environmental information (e.g., congestion coefficient). For virtual game control, the agent's environmental state can include the virtual game's operating state (e.g., the speed of the virtual vehicle). The current moment can be any moment in the agent's current environment. By calculating the intrinsic reward of the environmental state at each moment, the intrinsic reward of the state space surrounding the agent can be further estimated at each moment, helping the agent to better explore the environment.

[0058] Step 202: Determine the number of environment states to be set sequentially before selecting a state.

[0059] In most reinforcement learning environments, the environment is typically turn-based. That is, similar tasks are completed within a single turn, and at the end of the turn, the agent in the reinforcement learning method terminates its action after completing the task for that turn. Many existing attempts at traditional exploration problems are usually turn-based exploration methods, where the metric for whether an agent needs to explore a given environment state further is based on comparisons of data across all turns used during training.

[0060] The environmental state used in the data processing method disclosed herein can be used only as a sample to calculate the intrinsic reward when estimating the local intrinsic dimension. This is an in-round exploration, which can bring about the generalization of intrinsic reward estimation.

[0061] Optionally, when estimating the local intrinsic dimension, the intrinsic reward can also be calculated using the environmental state during multiple round cycles and the environmental state within a round cycle. This takes into account both inter-round exploration and intra-round exploration. Inter-round exploration, in a fixed specific environment, has better performance in terms of overfitting to the environment.

[0062] In this embodiment, the set number of environmental states prior to the selected state are collected sequentially in chronological order. The collected environmental states can be related to the collection cycle of the executing entity, and the set number can be determined based on the intrinsic reward calculation accuracy. For example, the set number can be 5000.

[0063] In this embodiment, the set quantity can also be the number of environmental states collected within a set time period. The set time period can correspond to the task execution round cycle of the agent's reinforcement learning. For example, the set time period is one round cycle of reinforcement learning, or the set time period includes multiple round cycles of reinforcement learning.

[0064] Step 203: Based on the selected state and a set number of environmental states, determine the representation of the local intrinsic dimension of the selected state.

[0065] In this embodiment, the local intrinsic dimension is a mathematical quantity that measures the dimension of the state space. When the selected state and the state of the environment are in different state spaces, the representation formulas that can be used to represent the local intrinsic dimension can be different; for example, in a low-dimensional state space, the representation formula can use a linear relationship to represent the correspondence between the selected state, the state of the environment and the local intrinsic dimension. In a high-dimensional state space, the representation formula shown in equation (2) is used.

[0066] In this embodiment, the introduction of local intrinsic dimension during the agent's reinforcement learning process aims to estimate the dimensionality of the state space of the agent's environment as the agent interacts with it. Based on the obtained local intrinsic dimension of the selected state, the agent's intrinsic reward is determined. This encourages the agent to explore higher-dimensional environmental state spaces during interaction, as these spaces are likely more worthwhile to explore and more important within the overall environmental state space. Conversely, the state spaces of environments lacking change and offering little distinction from their neighbors typically have lower estimated local intrinsic dimensions, indicating lower importance and less information. This approach also helps the agent better explore the environment even in the absence of genuine external rewards, preventing it from consistently failing to learn meaningful action strategies in the absence of such rewards.

[0067] In Euclidean space, the volume of a two-dimensional sphere is πr. 2 The volume of a three-dimensional sphere is 4 / 3πr. 3 Similarly, for a D-dimensional sphere, its volume is proportional to r. D Therefore, based on the ratio of the change in radius to the change in volume, the dimensions of the sphere can be estimated as follows:

[0068]

[0069] In equation (1), V1 and r1 are a volume value of the sphere and the radius corresponding to that volume value, respectively, and V2 and r2 are another volume value of the sphere after the volume change and the radius corresponding to that volume value.

[0070] The estimation method for the eigendimensionality of a sphere is transferred to probability density. The formal definition of local eigendimensionality is as follows: For a given data sample x∈X, where X is the entire dataset, we define r>0 as a random variable representing the distance of x to other sample data in the entire dataset. The cumulative probability density function F(r) is positive and continuously differentiable. For a sample x at a distance r, its local eigendimensionality is as shown in equation (2):

[0071]

[0072] If the limit exists, the local eigendimensional dimension at point x can be defined as:

[0073] In equation (2), ε is the change in distance from x to other sample data.

[0074] Estimating local intrinsic dimensions is impractical when performing calculations on the entire dataset. Alternatively, a subset of samples can be sampled from the entire dataset. Through partial sample X B The local intrinsic dimension is estimated as shown in equation (2), where g(x) represents the feature transformation of the sample, and r i (g(x), g(X)) B )) represents a portion of sample X B Let r be the distance from sample x to the i-th sample among the k (k > 0 and k is a natural number) samples closest to sample x. max (g(x), g(X)) B )) represents a portion of sample X B In the equation, the distance from x is the distance of the k nearest samples to x to x.

[0075]

[0076] Step 204: Based on the representation, calculate the intrinsic reward that guides the agent's actions.

[0077] In this embodiment, the rewards guiding the agent's actions in the reinforcement learning method include external real rewards and intrinsic rewards. External real rewards are used to measure the agent's performance and are generally immutable. Intrinsic rewards are internal to the agent and are used to train the agent's action strategy throughout its lifecycle. These intrinsic rewards can be used to update the agent's parameters. The intrinsic rewards obtained through the data processing method of this embodiment can help the agent to effectively explore the current environment and determine effective action strategies.

[0078] In reinforcement learning tasks for agents, the local intrinsic dimensions estimated from several consecutive environmental states can be used as intrinsic rewards to guide the agent's actions. Even without external real rewards, the agent can achieve satisfactory control effects, thus learning meaningful action strategies when the agent does not receive effective external real rewards.

[0079] In this embodiment, based on different representation formulas, the selected state and a set number of environmental states can be input into the representation formula to calculate the local intrinsic dimension, which is then used as the intrinsic reward for the corresponding selected state. Optionally, feature transformation can be performed on the selected state and the set number of environmental states, and the representation formula can be calculated using the transformed features to obtain the local intrinsic dimension, which is then used as the intrinsic reward for the corresponding selected state.

[0080] Optionally, after obtaining the local intrinsic dimensions, the local intrinsic dimensions can be weighted or have coefficients added to obtain the intrinsic reward for the corresponding selected state.

[0081] In this embodiment, for the agent at time t before the current time t+1, the observed environmental state is s. t Based on the agent's own policy π(a|s) (as shown in the Π model in Figure 4), the corresponding environmental state s is obtained. t Behavioral state a t Receive external rewards Simultaneously, the environment transitions to the current environmental state s at time t+1. t+1 At this point, the current environmental state can be taken as the selected state. Based on the representation of the local intrinsic dimension as shown in equation (2) or (3), the local intrinsic dimension of the selected state can be estimated, i.e., the local intrinsic dimension estimate. The local intrinsic dimension estimate is then used as the intrinsic reward of the selected state.

[0082] In some optional implementations of this embodiment, the estimation of the local intrinsic dimension can be performed using the representation shown in equation (4), which involves sampling a portion of the data from the entire dataset. x represents the sample that needs to be estimated, r i (x, X) B ) represents a portion of the sample X B In the equation, r is the distance from the i-th sample among the k closest samples to sample x to sample x. max (x, X) B ) represents a portion of the sample X B In the equation, the distance from x is the distance from the k nearest samples to x to the sample farthest from x.

[0083] The local intrinsic dimension estimate for sample x is:

[0084]

[0085] for For computational purposes, all past environmental states experienced by the agent constitute the total data. A subset of samples needs to be extracted from this dataset, and then selected based on the chosen state s. t+1 Estimating the local intrinsic dimension using the k nearest samples, we can choose the agent's k past environmental states in the current round to represent the selected state s. t+1 The most recent k samples, i.e., the environmental state sequence S B =[s t-k+1 s t-k+2 , ..., s t-1 s t From this, we can obtain information about the selected state s. t+1 Local eigendimensional estimation, i.e.

[0086]

[0087] Where k is a hyperparameter.

[0088] In equation (5), r i (s t+1 S B ) represents the environmental state sequence S B In the middle, away from the selected state s t+1 From the i-th environment state among the k most recent environment states to the selected state s t+1 distance; r max (s t+1 S B ) represents the environmental state sequence S B In the middle, away from the selected state s t+1 Among the most recent k environmental states and the selected state s t+1 The furthest environmental state to the selected state st+1 The distance between and , where k and i are natural numbers, and i ≤ k.

[0089] In an optional implementation of this embodiment, equation (5) is used as a representation of the local intrinsic dimension of the selected state. The local intrinsic dimension obtained by calculating this representation can be directly used as the intrinsic reward corresponding to the selection and loading, providing reliable guidance for the agent's actions.

[0090] Optionally, the environmental state sequence S B The environmental state can also be the environmental state of the agent across multiple rounds.

[0091] When calculating intrinsic rewards in a high-dimensional state space, directly estimating the intrinsic reward using the environmental state is inappropriate. A method is needed to transform the features of the original environmental state input. In another optional implementation of this embodiment, the representation can be:

[0092]

[0093] In equation (6), s t+1 To select a state, S B =[s t-k+1 s t-k+2 , ..., s t-1 s t ] represents the sequence of environment states formed by the k (k>0) sequentially preceding the selected state, ψ(s) t+1 ) represents the selection feature for the selected state, r i (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the context of the k closest environmental features to the selected feature, the distance from the furthest environmental feature to the selected feature is calculated, where the environmental feature represents the environmental state. max (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the equation, the distance from the selected feature to the k closest environmental features is the distance from the selected feature to the environmental feature furthest from the selected feature, where k and i are natural numbers and i ≤ k.

[0094] Based on the selected state s t+1 The local intrinsic dimension is re-estimated using the k nearest samples, where the agent's k past states in the current round or across multiple rounds can be chosen to represent the relationship with s. t+1 The most recent k samples, i.e., state S B =[s t-k+1 s t-k+2 , ..., st-1 s t From this, we can obtain information about the selected state s. t+1 The local eigendimensionality estimation. Equation (6) differs from equation (5) in that it selects state s. t+1 With S B =[s t-k+1 s t-k+2 , ..., s t-1 s t They respectively became ψ(s) t+1 ) and ψ(s B )=[ψ(s t-k+1 ), ψ(s t-k+2 ), ..., ψ(s) t-1 ), ψ(s t )).

[0095] In this optional implementation, the selected features and environmental features can be obtained directly through feature engineering.

[0096] Alternatively, a pre-trained feature transformation model can be used to directly extract features from the selected state and the environmental state sequence to obtain the selected features and the environmental features of each environmental state in the environmental state sequence.

[0097] In an optional implementation of this embodiment, equation (6) is used as a representation for estimating the local intrinsic dimension of the selected state. The local intrinsic dimension obtained by calculating this representation can be directly used as the intrinsic reward corresponding to the selection and loading, providing a reliable guidance method for guiding the action of the agent under the state space conditions of a high-dimensional environment.

[0098] In another optional implementation of this embodiment, the selected features and environmental features are obtained through a trained feature transformation model, which is trained using the following steps:

[0099] A feature transformation network is used to transform the environmental states adjacent to the selected state in the sequence of selected and environmental states, resulting in selected features and adjacent features. The environmental states adjacent to the selected state are sent to the agent to obtain the behavioral state output by the agent. Based on the behavioral state and adjacent features, the predicted value of the selected feature is obtained. Based on the adjacent features and selected features, the predicted value of the behavioral state is obtained. Based on the predicted value of the selected feature, the predicted value of the behavioral state, and the behavioral state, the parameters of the feature transformation network are adjusted to obtain the feature transformation model.

[0100] Specifically, such as Figure 3 As shown, for environmental state s t and select state s t+1 Feature transformation is performed using a feature transformation network T to obtain adjacent features ψ(s) respectively.t ) and the selected feature ψ(s) t+1 Meanwhile, a forward network F and a reverse network I are defined. The forward network F processes adjacent features ψ(s). t ) and behavioral state a t Make a prediction to obtain the selected feature ψ(s) t+1 Predicted value Define the forward loss of a forward network The reverse network processes adjacent features ψ(s) t ) and the selected feature ψ(s) t+1 To predict the corresponding behavioral state a, we can obtain the prediction. t Predicted value Define the reverse loss of the reverse network Based on the predicted values ​​of selected features, the predicted values ​​of behavioral states, and the positive loss L, a positive loss L is calculated. fw and reverse loss L inv Adjust the parameters of the feature transformation network until the positive loss L fw and reverse loss L inv In the equilibrium phase, the parameters of the current feature transformation network are no longer changed, and the current feature transformation network T becomes the desired feature transformation model. The feature transformation model obtained through this optional implementation effectively avoids the influence of environmental factors unrelated to the agent and task on the agent.

[0101] The data processing method provided in this embodiment can be used in all reinforcement learning scenarios, such as games, human-computer interaction, virtual control, etc. The environment in which the agent is located includes a virtual environment, and the environment state includes the virtual environment running state. The agent determines the current game running strategy based on the virtual environment running state and the intrinsic reward.

[0102] The data processing method provided in this embodiment can be applied to an autonomous driving environment. In some optional implementations of this embodiment, the intelligent agent includes an autonomous driving vehicle, and the environmental state includes the operating state of the autonomous driving vehicle, which includes the vehicle speed, fuel consumption, vehicle temperature, etc. The above data processing method also includes: determining the behavioral state of the autonomous driving vehicle based on the selected state and intrinsic reward, which includes the behavioral state of the autonomous driving vehicle such as left turn, right turn, straight, stop, etc.

[0103] Optionally, the environmental conditions may also include: the in-vehicle and out-of-vehicle conditions of the autonomous vehicle. For example, the in-vehicle conditions include: the number of people, the weight of the cargo, etc.; the out-of-vehicle conditions may include: congested roads, highways, rainy weather, etc.

[0104] In this optional implementation, the data processing method is applied to the autonomous driving scenario, providing a reliable means for the intelligent agent in the autonomous vehicle to obtain intrinsic rewards, thus ensuring the reliability of the autonomous vehicle control.

[0105] The data processing method provided in this embodiment provides a lower local intrinsic dimensionality estimate (lower intrinsic reward) for state spaces that lack change compared to their surroundings, while providing a higher dimensionality estimate (higher intrinsic reward) for state spaces that exhibit more significant and unique changes compared to their surroundings. This method encourages the agent to explore state spaces that are more unique than their surroundings. The estimated dimensionality provides some explanation for the agent's behavior.

[0106] The data processing method provided in the embodiments of this disclosure first takes the current environmental state of the agent's environment as the selected state; second, it determines a predetermined number of environmental states sequentially preceding the selected state; third, based on the selected state and the predetermined number of environmental states, it determines a representation for estimating the local intrinsic dimension of the selected state, where the local intrinsic dimension is a mathematical quantity that measures the dimension of the state space; finally, based on the representation, it calculates the intrinsic reward that guides the agent's actions. Thus, when the agent interacts with the environment, it estimates the dimension of its current state space, and based on the estimated dimension, determines the agent's intrinsic reward, thereby encouraging the agent to explore higher-dimensional state spaces as much as possible, achieving better exploration of the environment and improving the efficiency of reinforcement learning.

[0107] Since reinforcement learning environments are typically round-based, similar tasks will be completed within a single round. At the end of the round, the agent terminates its action. In another embodiment of this disclosure, a pseudo-counting method can be used to sample both the intra-round and inter-round environmental states simultaneously, and the selected state can be used to estimate the local environmental dimension to obtain the intrinsic reward. Thus, for different tasks, both inter-round exploration and intra-round exploration are considered. Intra-round exploration can lead to better generalization performance, while inter-round exploration can better perform against overfitting to the environment.

[0108] In continuous space problems, directly counting state actions will fail. Exploration algorithms based on pseudo-counting evaluate the frequency of state occurrence by designing a density model, which can make the calculated pseudo-count values ​​replace the real counts.

[0109] In another embodiment of this disclosure, the above-mentioned calculation of the intrinsic reward guiding the agent's actions based on the representation formula includes: estimating the number of times the agent accesses the selected state to obtain a pseudo-count value of the selected state; calculating the local intrinsic dimension estimate of the selected state based on the pseudo-count value and the representation formula; and using the local intrinsic dimension estimate as the intrinsic reward guiding the agent's actions.

[0110] In this optional implementation, the estimation of the number of times the selected state is accessed can be performed in the same reinforcement learning round of the same task or in multiple reinforcement learning rounds of different tasks. Estimating the number of times the agent accesses the selected state refers to estimating the number of times the agent accesses the selected state. By estimating the number of times the selected state is accessed, the number of times the selected state is experienced in the sample set can be determined.

[0111] In this optional implementation, estimating the number of times the agent accesses the selected state to obtain a pseudo-count value for the selected state includes: determining the trained density model corresponding to the current reinforcement learning task; and using the density model to estimate the probability of the selected state occurring within a set time period to obtain a pseudo-count value for the selected state.

[0112] In this embodiment, when estimating the local intrinsic dimension of the selected state using the representation, in order to encourage the agent to avoid the state space that has been repeatedly experienced before, the predicted pseudo-count value needs to be zero for the state space that only has the selected state. That is, when calculating the representation, the state space corresponding to each selected state needs to be discarded.

[0113] The embodiments of this disclosure provide a method for calculating the intrinsic reward that guides the action of an agent. The method estimates the number of times the agent accesses a selected state to obtain a pseudo-count value. Based on the pseudo-count value and the representation, the method calculates the local intrinsic dimension estimate and uses the local intrinsic dimension estimate as the intrinsic reward that guides the action of the agent. This method can save the computational cost of calculating the intrinsic reward and improve the efficiency of obtaining the intrinsic reward.

[0114] In some optional implementations of this embodiment, the estimation of the number of times the agent accesses the selection state to obtain a pseudo-count value for the selection state includes:

[0115] The probability of the selected state occurring within a set time period is estimated using a first density model that has been trained, resulting in a first density value. The first density model is trained based on a set number of environmental states. The probability of the selected state occurring within a set time period is estimated using a second density model that has been trained, resulting in a second density value. The second density model is trained based on the selected state and a set number of environmental states. Based on the first density value and the second density value, a pseudo-count value for the selected state is obtained.

[0116] In this optional implementation, obtaining the pseudo-count value of the selected state based on the first density value and the second density value includes: calculating the average of the first density value and the second density value, and using the average value as the pseudo-count value of the selected state.

[0117] Optionally, obtaining the pseudo-count value of the selected state based on the first density value and the second density value includes: taking the maximum value of the first density value and the second density value as the pseudo-count value of the selected state.

[0118] This optional implementation provides a pseudo-count value for the selected state. It estimates the probability of the selected state occurring within a set time period using a first density model and a second density model, respectively. Based on the first density value and the second density value, it obtains the pseudo-count value for the selected state, thereby improving the reliability of obtaining the pseudo-count value.

[0119] In some optional implementations of this embodiment, obtaining the pseudo-count value of the selected state based on the first density value and the second density value includes: calculating the information gain of the first density value and the second density value; and calculating the pseudo-count value of the selected state based on the information gain.

[0120] In this optional implementation, the information gain is asymmetric and is used to measure the difference between the first density value and the second density value of the two probability distributions. The information gain describes the difference when the first density value is used for encoding and then the second density value is used for encoding.

[0121] In this optional implementation, information gain PG t (s t+1 )=logρ t ′(s t+1 )-logρ t (s t+1 ), where the first density value is ρ t (s t+1 The second density value is ρ t ′(s t+1 ).

[0122] In this optional implementation, the pseudo-count value Substituting information gain The pseudo count value can then be obtained.

[0123] In this optional implementation, the pseudo-count value is obtained by calculating the difference in probability distributions of the first density value and the second density value, thus ensuring the reliability of the obtained pseudo-count value.

[0124] In some optional implementations of this embodiment, the above-mentioned calculation of the local intrinsic dimension estimate of the selected state based on the pseudo-count value and the representation includes: splitting the environmental state corresponding to the representation into pseudo-count value selected states and difference value environmental states, where the difference is the difference between a set number and the pseudo-count value; splitting the representation into an expression including a first representative expression and a second representative expression, wherein the first representative expression corresponds to the pseudo-count value selected states and the second representative expression corresponds to the difference value environmental states; and calculating the second representative expression based on the difference value environmental states to obtain the local intrinsic dimension estimate of the selected state.

[0125] In this optional implementation, based on the pseudo-count value, the representation expression is split into an expression including a first representative expression and a second representative expression. The first representative expression is the first representative expression corresponding to the selected state, and the second representative expression is the second representative expression corresponding to the difference of the environmental states. The numerator of the representation expression is the distance from the environmental state to the selected state, and the first representative expression is the representative expression of the selected state. Since the distance is zero, the value obtained by the first representative expression must be zero.

[0126] In this optional implementation, for the selected state s t+1 The k environmental states corresponding to the representation are changed from the original S k =[s t-k+1 s t-k+2 , ..., s t-1 s t It became a pseudo-count value. Selected states s t+1 , With sequence Environmental status That is, from the original state s t+1 The previous k environmental states changed sequentially to the following sequence. The states and the values ​​estimated by the density model Selected states s t+1 Thus, in calculating the selected state s t+1 When the local intrinsic dimension is zero, the intrinsic reward becomes (where... Abbreviated as )

[0127]

[0128] because Only sample s t+1 Therefore, we can know then,

[0129]

[0130] In this optional implementation, the first representative expression can be: The second representative form is k is a hyperparameter.

[0131] In this optional implementation, by splitting the representation, while referencing the local intrinsic dimension of the selected state, the agent is also encouraged to avoid state spaces that have been repeatedly experienced. For environmental states that the agent has repeatedly experienced, a smaller intrinsic reward is obtained when calculating the local intrinsic dimension; for environmental states that have been experienced less frequently, a relatively larger intrinsic reward is given when calculating the local intrinsic dimension. This embodiment also considers the novelty within reinforcement learning rounds and the novelty between reinforcement learning rounds, making better use of mathematical concepts and providing better interpretability for the agent's intrinsic reward.

[0132] In a practical example, the bottom of a wooden stick is connected to a black object below. The agent moves the object left and right to keep the stick upright as much as possible. The longer the stick remains upright in a round, the more external real-world reward it receives. like Figure 4(a) and 4(b) As shown, Figure 4(a) only uses external real rewards. Figure 4(b) shows only the intrinsic reward generated by estimation through local intrinsic dimensions. The horizontal axis of the graph represents the number of rounds in the training process, and the vertical axis represents the actual cumulative reward for each round. As can be seen from the graph, for the results in Figure 4(a), the agent experienced a relatively long period with almost no reward in the early stages, and then quickly mastered an effective strategy. In contrast, for the results in Figure 4(b), the agent, guided by intrinsic reward, was able to form some meaningful strategies at a relatively early stage and achieved a higher cumulative reward for each round in the early stages.

[0133] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a data processing apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0134] like Figure 5As shown, embodiments of this disclosure provide a data processing apparatus 500, which includes: a selection unit 501, a determination unit 502, a dimension representation unit 503, and a calculation unit 504. The selection unit 501 can be configured to take the current environmental state of the agent's environment as the selection state. The determination unit 502 can be configured to sequentially determine a predetermined number of environmental states before determining the selection state. The dimension representation unit 503 can be configured to determine a representation formula for estimating the local intrinsic dimension of the selection state based on the selection state and the predetermined number of environmental states, where the local intrinsic dimension is a mathematical quantity that measures the dimension of the state space. The calculation unit 504 can be configured to calculate the intrinsic reward guiding the agent's actions based on the representation formula.

[0135] In this embodiment, the specific processing and technical effects of the selection unit 501, determination unit 502, dimension representation unit 503, and calculation unit 504 in the data processing device 500 can be found in references to [reference needed]. Figure 2 The corresponding steps are 201, 202, 203, and 204 in the embodiment.

[0136] In some embodiments, the dimension representation unit 503 includes: an estimation subunit (not shown in the figure), a calculation subunit (not shown in the figure), and an as-subunit (not shown in the figure). The estimation subunit can be configured to estimate the number of times the agent accesses the selected state, obtaining a pseudo-count value for the selected state. The calculation subunit can be configured to calculate a local intrinsic dimension estimate of the selected state based on the pseudo-count value and the representation formula. The as-subunit can be configured to use the local intrinsic dimension estimate as an intrinsic reward guiding the agent's actions.

[0137] In some embodiments, the estimation subunit includes: a first estimation module (not shown in the figure), a second estimation module (not shown in the figure), and a obtaining module (not shown in the figure). The first estimation module can be configured to estimate the probability of a selected state occurring within a set time period using a trained first density model, obtaining a first density value. The first density model is trained based on a set number of environmental states. The second estimation module can be configured to estimate the probability of a selected state occurring within a set time period using a trained second density model, obtaining a second density value. The second density model is trained based on the selected state and a set number of environmental states. The obtaining module can be configured to obtain a pseudo-count value of the selected state based on the first density value and the second density value.

[0138] In some embodiments, the above-described obtaining module includes: a gain submodule (not shown in the figure) and a calculation submodule (not shown in the figure). The gain submodule can be configured to calculate the information gain of the first density value and the second density value. The calculation submodule can be configured to calculate the pseudo-count value of the selected state based on the information gain.

[0139] In some embodiments, the above-mentioned calculation subunit includes: a state splitting module (not shown in the figure), a formula splitting module (not shown in the figure), and a calculation module (not shown in the figure). The state splitting module can be configured to split the environmental state corresponding to the representation into pseudo-count values ​​of selected states and difference values ​​of environmental states, where the difference is the difference between a set number and the pseudo-count value. The formula splitting module can be configured to split the representation into expressions including a first representative expression and a second representative expression, where the first representative expression corresponds to the pseudo-count values ​​of selected states, and the second representative expression corresponds to the difference values ​​of environmental states. The calculation module can be configured to calculate the second representative expression based on the difference values ​​of environmental states to obtain an estimate of the local intrinsic dimension of the selected states.

[0140] In some embodiments, the above representation is:

[0141]

[0142] Among them, s t+1 To select a state, S B =[s t-k+1 s t-k+2 , ..., s t-1 s t [r] represents the sequence of k sequential environmental states preceding the selected state. i (s t+1 S B ) represents the environmental state sequence S B In the middle, away from the selected state s t+1 From the i-th environment state among the k most recent environment states to the selected state s t+1 distance; r max (s t+1 S B ) represents the environmental state sequence S B In the middle, away from the selected state s t+1 Among the most recent k environmental states and the selected state s t+1 The furthest environmental state to the selected state s t+1 The distance between and , where k and i are natural numbers, and i ≤ k.

[0143] In some embodiments, the above representation is:

[0144]

[0145] Among them, s t+1 To select a state, S B =[s t-k+1 s t-k+2 , ..., s t-1 s t ] represents the sequence of k sequential environmental states prior to the selected state, ψ(s) t+1 ) represents the selection feature for the selected state, r i (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the context of the k closest environmental features to the selected feature, the distance from the furthest environmental feature to the selected feature is calculated, where the environmental feature represents the environmental state. max (ψ(s t+1 ), ψ(S B ) represents the environmental state sequence S B In the equation, the distance from the selected feature to the k closest environmental features is the distance from the selected feature to the environmental feature furthest from the selected feature, where k and i are natural numbers and i ≤ k.

[0146] In some embodiments, the selected features and environmental features are obtained through a trained feature transformation model. The feature transformation model is trained using the following steps: performing feature transformation on the environmental states adjacent to the selected state in the sequence of selected and environmental states to obtain selected features and adjacent features; sending the environmental states adjacent to the selected state to the agent to obtain the behavioral state output by the agent; obtaining the predicted value of the selected features based on the behavioral state and adjacent features; obtaining the predicted value of the behavioral state based on the adjacent features and selected features; and adjusting the parameters of the feature transformation network based on the predicted value of the selected features, the predicted value of the behavioral state, and the behavioral state to obtain the feature transformation model.

[0147] In some embodiments, the intelligent agent includes an autonomous vehicle, and the environmental state includes the operating state of the autonomous vehicle; the device is further configured to determine the behavioral state of the autonomous vehicle based on the selected state and intrinsic reward.

[0148] The data processing apparatus provided in the embodiments of this disclosure first selects the environmental state of the agent's current environment as the selected state by the selection unit 501; second, the determination unit 502 determines a predetermined number of environmental states sequentially before determining the selected state; third, the dimensional representation unit 503 determines a representation formula for estimating the local intrinsic dimension of the selected state based on the selected state and the predetermined number of environmental states, where the local intrinsic dimension is a mathematical quantity that measures the dimension of the state space; finally, the calculation unit 504 calculates the intrinsic reward guiding the agent's actions based on the representation formula. Thus, when the agent interacts with the environment, it estimates the dimension of its current state space, determines the agent's intrinsic reward based on the estimated dimension of the state space, thereby encouraging the agent to explore higher-dimensional state spaces as much as possible, helping the agent to better explore the environment, and improving the efficiency of reinforcement learning.

[0149] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device 600 suitable for implementing embodiments of the present disclosure.

[0150] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0151] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0152] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by a processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0153] It should be noted that the computer-readable medium in the embodiments of this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0154] The aforementioned computer-readable medium may be included in the aforementioned server; or it may exist independently and not assembled into the server. The aforementioned computer-readable medium carries one or more programs that, when executed by the server, cause the server to: take the current environmental state of the agent's environment as the selected state; determine a predetermined number of environmental states sequentially prior to the selected state; based on the selected state and the predetermined number of environmental states, determine a representation for estimating the local intrinsic dimension of the selected state, where the local intrinsic dimension is a mathematical quantity that measures the dimension of the state space; and calculate, based on the representation, the intrinsic reward guiding the agent's actions.

[0155] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a selection unit, a determination unit, a dimension representation unit, and a computation unit. The names of these units do not necessarily limit the specific unit itself; for example, a selection unit may be described as a unit "configured to take the current environmental state of the agent's environment as the selection state."

[0158] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A data processing method, the method comprising: The current environmental state of the environment in which the intelligent agent is located is used as the selected state, wherein the intelligent agent includes an autonomous vehicle, and the environmental state includes the operating state of the autonomous vehicle. A predetermined number of environmental states are sequentially selected before the selected state is determined; Based on the selected state and a set number of environmental states, a representation for estimating the local intrinsic dimension of the selected state is determined. The local intrinsic dimension is a mathematical quantity that measures the dimension of the state space, and the representation of the local intrinsic dimension is related to the dimension of the state space. Based on the representation, the intrinsic reward guiding the agent's actions is calculated, including: performing feature transformation on the selected state and a set number of environmental states, calculating the representation based on the transformed features to obtain the local intrinsic dimension, and using the local intrinsic dimension as the intrinsic reward for the agent's actions.

2. The method according to claim 1, wherein calculating the intrinsic reward guiding the agent's actions based on the representation includes: The number of times the agent accesses the selected state is estimated to obtain a pseudo-count value for the selected state; Based on the pseudo-count value and the representation, calculate the local intrinsic dimension estimate of the selected state; The estimated local intrinsic dimension is used as an intrinsic reward to guide the agent's actions.

3. The method according to claim 2, wherein, The step of estimating the number of times the agent accesses the selected state to obtain a pseudo-count value for the selected state includes: The probability of the selected state occurring in the set time period is estimated using the trained first density model to obtain a first density value. The first density model is trained based on the set number of environmental states. The probability of the selected state occurring during the set time period is estimated using the trained second density model to obtain the second density value. The second density model is trained based on the selected state and the set number of environmental states. Based on the first density value and the second density value, the pseudo-count value of the selected state is obtained.

4. The method according to claim 3, wherein, The process of obtaining the pseudo-count value of the selected state based on the first density value and the second density value includes: Calculate the information gain of the first density value and the second density value; Based on the information gain, the pseudo-count value of the selected state is calculated.

5. The method according to claim 2, wherein, The step of calculating the local intrinsic dimension estimate of the selected state based on the pseudo-count value and the representation includes: The environmental state corresponding to the representation is split into the pseudo-count value of the selected state and the difference value of the environmental state, where the difference is the difference between the set quantity and the pseudo-count value; The representation is decomposed into an expression including a first representative expression and a second representative expression, wherein the first representative expression corresponds to the pseudo-count value of the selected state, and the second representative expression corresponds to the difference value of the environmental state; Based on the difference in environmental states, the second representative expression is calculated to obtain the local intrinsic dimension estimate of the selected state.

6. The method according to any one of claims 1-5, wherein, When the state space is a high-dimensional state space, the representation is: in, To select a state, This refers to the sequence of k sequential environmental states preceding the selected state. Represents the sequence of environmental states In the middle, away from the selected state From the i-th environment state among the k most recent environment states to the selected state The distance; Represents the sequence of environmental states In the middle, away from the selected state Among the most recent k environmental states and the selected state From the furthest environmental state to the selected state The distance between and , where k and i are natural numbers, and i ≤ k.

7. The method according to any one of claims 1-5, wherein, When the state space is a high-dimensional state space, the representation is: in, To select a state, This refers to the sequence of k sequential environmental states preceding the selected state. The selection features for selecting states. Represents the sequence of environmental states In the context of the k closest environmental features to the selected feature, the distance from the environmental feature furthest from the selected feature is the distance from the selected feature to the selected feature, where the environmental feature is a feature of the environmental state. Represents the sequence of environmental states In the equation, the distance from the selected feature to the k closest environmental features is the distance from the selected feature to the environmental feature furthest from the selected feature, where k and i are natural numbers and i ≤ k.

8. The method according to claim 7, wherein, The selected features and the environmental features are obtained through a trained feature transformation model, which is trained using the following steps: Perform feature transformation on the selected state and the environmental states adjacent to the selected state in the environmental state sequence to obtain the selected feature and the adjacent feature; Send the environmental state adjacent to the selected state to the agent to obtain the behavioral state output by the agent. Based on the behavioral state and the adjacent features, the predicted value of the selected feature is obtained; Based on the adjacent features and the selected features, the predicted value of the behavioral state is obtained; Based on the predicted values ​​of the selected features, the predicted values ​​of the behavioral states, and the behavioral states, the parameters of the feature transformation network are adjusted to obtain the feature transformation model.

9. The method according to claim 1, wherein, The method further includes: determining the behavioral state of the autonomous vehicle based on the selected state and the intrinsic reward.

10. A data processing apparatus, the apparatus comprising: The selection unit is configured to take the environmental state of the environment in which the agent is located at the current moment as the selection state, wherein the agent includes an autonomous vehicle and the environmental state includes the operating state of the autonomous vehicle. The determining unit is configured to determine a predetermined number of environmental states sequentially prior to the selection state; The dimension representation unit is configured to determine a representation of the local intrinsic dimension of the selected state based on the selected state and a set number of environmental states. The local intrinsic dimension is a mathematical quantity that measures the dimension of the state space, and the representation of the local intrinsic dimension is related to the dimension of the state space. The computing unit is configured to calculate the intrinsic reward guiding the agent's actions based on the representation, including: performing feature transformation on the selected state and a set number of environmental states, calculating the representation based on the transformed features to obtain the local intrinsic dimension, and using the local intrinsic dimension as the intrinsic reward for the agent's actions.

11. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images