Potential energy-based reward automaton method and device in partially observable reinforcement learning

Abstract

The invention relates to the technical field of computer science and artificial intelligence, and provides a potential energy-based reward automaton method and device in partially observable reinforcement learning, and the method comprises the steps: constructing a standard POMDP; generating a track from a part of observable environment, compressing the track, removing continuous and repeated advanced observation, and retaining key change points; constructing an optimal reward automaton through the trajectory, learning the reward automaton by using genetic local search, and solving the optimal reward automaton; expanding a reward automaton to a partially observable Markov decision process, and defining a potential energy-based reward automaton in partially observable reinforcement learning; and calculating the potential energy value of the potential energy-based reward automaton in the partial observable reinforcement learning by using a value iteration strategy. According to the invention, through the potential energy-based reward automaton in partial observable reinforcement learning, a plurality of problems in the prior art facing partial observable environments are solved, so that the learning and decision-making efficiency of the intelligent agent is improved.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the fields of computer science and artificial intelligence technology, and more particularly to a method and apparatus for a potential-based reward automaton in partially observable reinforcement learning. Background Art

[0002] In the field of reinforcement learning, traditional reinforcement learning methods are mostly based on Markov decision processes (MDPs) and have achieved certain results in fully observable environments. However, in practical applications, such as robotics and autonomous driving scenarios, they often face problems such as sensor noise, occlusion, and incomplete information, resulting in partially observable environments. In partially observable environments, it is difficult for intelligent agents to obtain complete state information, which poses new challenges to reinforcement learning. For example, in autonomous driving, severe weather or physical obstacles can limit the perception capabilities of vehicle sensors, resulting in incomplete data acquisition. The vehicle must make navigation decisions based on limited sensor input and must dynamically adjust its route when encountering road congestion or diversions.

[0003] The design of the reward function is crucial when addressing reinforcement learning problems in partially observable environments. Due to a lack of comprehensive state information, the agent may misassociate observations with rewards, leading to policy deviations. In complex environments, rewards for the agent's actions may be delayed or sparse, making it difficult to learn effective policies directly from experience. Furthermore, the high variability and randomness of real-world scenarios, as well as the presence of multiple objectives and constraints, further complicate reward function design.

[0004] To address the challenge of partial observability, Icarte et al. introduced reward automata (RAs) and developed the QRM method, applying them to partially observable environments. There have also been innovations in learning RAs directly from empirical data. However, traditional methods still suffer from high computational complexity, such as the PSPACE-complete problem of computing optimal policies and the NP-hard problem of optimizing memoryless policies. Existing algorithms also have limitations in practical applications. Therefore, a more efficient method is needed to optimize potential reward automata to enhance the learning and decision-making capabilities of intelligent agents in partially observable environments. Summary of the Invention

[0005] To address the above problems, the present invention solves many difficulties faced by existing technologies when facing partially observable environments through a potential-based reward automaton in partially observable reinforcement learning, thereby improving the learning and decision-making efficiency of the intelligent agent.

[0006] According to an embodiment of the present invention, a potential-based reward automaton method and apparatus in partially observable reinforcement learning are provided.

[0007] In a first aspect of the present invention, a method for potential-based reward automata in partially observable reinforcement learning is provided. The method comprises:

[0008] Step S01: construct a standard POMDP;

[0009] Step S02: Generate trajectories from a partially observable environment and compress the trajectories, remove consecutive and repeated high-level observations, and retain key change points;

[0010] Step S03: constructing an optimal reward automaton through the trajectory, using genetic local search to learn the reward automaton and solve the optimal reward automaton;

[0011] Step S04: Extend the reward automaton to the partially observable Markov decision process and define the potential-based reward automaton in partially observable reinforcement learning;

[0012] Step S05: Use the value iteration strategy to calculate the potential energy value of the potential energy-based reward automaton in partially observable reinforcement learning.

[0013] Furthermore, the specific content of step S01 is:

[0014] Define a standard partially observable Markov decision process as a tuple:

[0015] P o =(S,O,A,r,p,ω,γ,μ)

[0016] Where: S is a finite set of states, where each state s∈S represents a possible situation in the environment; O is a finite set of observations, where each observation o∈O represents a possible outcome perceived by the agent in a given state; A is a finite set of actions, where each action a∈A represents the available choice of the agent in any state s∈S; r is the reward function; p is the state transition probability distribution; ω is the observation probability distribution; γ is the discount factor; μ is the initial state distribution.

[0017] Furthermore, the trajectory described in step S02 is a sequence representing the interaction of the agent with its environment, capturing the states visited, actions taken, and rewards received, and is defined as:

[0018] T={(s0,a0,r0),(s1,a1,r1),…,(s t ,a t ,r t )}

[0019] Among them, s t represents the state of the agent at time step t, a t represents the action taken by the agent at time step t, tt represents the reward the agent receives from the environment at time step t.

[0020] Furthermore, in partially observable environments, the agent relies on observations to make decisions, and the trajectory is defined as:

[0021] T={(o0,a0,r0),(o1,a1,r1),…,(o t ,a t ,r t )}

[0022] Among them: t represents the observation value of the agent at time step t.

[0023] Furthermore, the specific steps of step S03 are:

[0024] Step S031: Initialization: receiving observation set Σ, training trajectory T, maximum reward automaton state number u max and the time limit or maximum number of steps t max Termination criteria, receiving the randomly generated initial reward automaton R and the number of iterations n, the initial conditions are: the current time t is set to 0; the optimal cost c* is initialized to infinity ∞; the optimal reward automaton R* is initialized to "None"; the initial population G is set to empty;

[0025] Step S032: Optimizing the reward automaton;

[0026] Step S033: Genetic operation: transform the population G using the mapping function * To adapt to the next genetic operation;

[0027] Step S034: Neighborhood search: Get the neighborhood A of the current reward automaton R, and search for each reward automaton R in the neighborhood. n ∈A is evaluated, and the cost c of each domain individual is obtained through the evaluation function n , when the maximum number of steps t is reached max Or the maximum number of iterations, return the optimal reward automaton R * .

[0028] Furthermore, the specific steps of step S032 are:

[0029] Step S0321: Evaluate the cost of the current reward automaton: In each iteration, first evaluate the cost c of the current reward automaton R on the given trajectory T, as shown below:

[0030]

[0031] Among them, E represents the state set, which is the state transition sequence in the trajectory T, |f (o,i)| represents f (o,i) length;

[0032] Step S0322: Update the optimal solution: If the current cost c is less than the optimal cost c * , then update the optimal solution to the current solution, and update the current reward automaton R to the optimal reward automaton R * , and calculate the current population and fitness. The fitness of individuals in the current population is as follows:

[0033] f(G * )=(f1(g1),f2(g2),...,f n (g n ))

[0034] Among them, g i are individuals in a population;

[0035] If normalization is required, the fitness function is improved as follows:

[0036]

[0037] Here, Δf represents the range of f, i.e., max(f)-min(f).

[0038] Furthermore, the specific steps of step S04 are:

[0039] Given a partially observable Markov decision process P0 and an optimal reward automaton R P , where P0 and R P Sharing the labeling function L, we define a partially observable Markov decision process with a potential-based reward automaton as:

[0040]

[0041] in:

[0042] The combined state space of the original partially observable Markov decision states and reward automaton states; The initial state consists of the initial state of the partially observable Markov decision process and the initial state of the reward automaton; is the potential energy function defined for the reward structure; Γ:O×A→Σ is the labeling function that maps partially observable Markov decision process transformations to automaton symbols; is the reward function; is the state transition probability.

[0043] In a second aspect of the present invention, a device for a potential-based reward automaton in partially observable reinforcement learning is provided. The device comprises:

[0044] POMDP construction module: used to construct standard POMDP;

[0045] Trajectory generation module: used to generate trajectories from partially observable environments and compress them, removing repeated high-level observations and retaining key change points;

[0046] Solving module: used to construct the optimal reward automaton through trajectory, learn the reward automaton using genetic local search and solve the optimal reward automaton;

[0047] Definition module: used to extend the reward automaton to partially observable Markov decision processes and define the potential-based reward automaton in partially observable reinforcement learning;

[0048] Calculation module: used to calculate the potential value of the potential-based reward automaton in partially observable reinforcement learning using the value iteration strategy.

[0049] In a third aspect of the present invention, an electronic device is provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method according to the first aspect of the present invention is implemented.

[0050] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present invention is implemented.

[0051] Beneficial effects of the present invention:

[0052] 1. This paper combines genetic local search (GLS) with potential reward automata (RA) to construct a novel framework that integrates historical information compression and Markov recovery. Trajectory compression technology removes consecutive repeated observations, retains key change points, and converts the original trajectory into a high-level symbol sequence. This not only reduces computational costs but also encodes historical information through the RA state transition function, allowing future observation predictions in partially observable environments to rely solely on the current state. This solves the non-Markov problem of traditional POMDP and restores the Markov property of policy learning.

[0053] 2. The genetic local search algorithm proposed in this invention explores the global space of RA structure through genetic operations such as crossover and mutation, and combines neighborhood search for local optimization, thus balancing global exploration and local development.

[0054] 3. This paper uses a value iteration strategy to calculate the potential energy function of the RA state and integrates it into the reward function to form a potential energy shaping mechanism. This mechanism dynamically guides the agent to move to high-value states through potential energy differences, solving the problem of learning stagnation caused by sparse rewards.

[0055] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings, in which:

[0057] Figure 1 A flowchart of a method for potential-based reward automata in partially observable reinforcement learning according to an embodiment of the present invention is shown;

[0058] Figure 2 A flow chart of a method for solving an optimal reward automaton according to an embodiment of the present invention is shown;

[0059] Figure 3 shows the performance of different reinforcement learning methods according to embodiments of the present invention in six cases of partially observable environments;

[0060] Figure 4 FIG2 shows a block diagram of a device for a potential-based reward automaton in partially observable reinforcement learning according to an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of a device for a potential-based reward automaton in partially observable reinforcement learning according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0063] According to an embodiment of the present invention, a potential-based reward automaton method and device in partially observable reinforcement learning are proposed. Through the potential-based reward automaton in partially observable reinforcement learning, many difficulties faced by existing technologies when facing partially observable environments are solved, thereby improving the learning and decision-making efficiency of intelligent agents.

[0064] The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0065] Figure 11 is a flow chart of a method for a potential-based reward automaton in partially observable reinforcement learning according to an embodiment of the present invention. The method includes:

[0066] Step S01: construct a standard POMDP;

[0067] Step S02: Generate trajectories from a partially observable environment and compress the trajectories, remove consecutive and repeated high-level observations, and retain key change points;

[0068] Step S03: constructing an optimal reward automaton through the trajectory, using genetic local search to learn the reward automaton and solve the optimal reward automaton;

[0069] Step S04: Extend the reward automaton to the partially observable Markov decision process and define the potential-based reward automaton in partially observable reinforcement learning;

[0070] Step S05: Use the value iteration strategy to calculate the potential energy value of the potential energy-based reward automaton in partially observable reinforcement learning.

[0071] It should be noted that although the operations of the method of the present invention are described in a specific order in the above embodiments and drawings, this does not require or imply that these operations must be performed in this specific order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0072] In order to more clearly explain the potential-based reward automaton method in the above-mentioned partially observable reinforcement learning, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0073] The following example further illustrates the potential-based reward automaton method in partially observable reinforcement learning:

[0074] Step S01: Construct a standard POMDP.

[0075] Specifically, the standard partially observable Markov decision process (POMDP) is defined as a tuple:

[0076] P o =(S,O,A,r,p,ω,γ,μ)

[0077] Where: S is a finite set of states, where each state s∈S represents a possible situation in the environment; O is a finite set of observations, where each observation o∈O represents a possible outcome perceived by the agent in a given state; A is a finite set of actions, where each action a∈A represents the available choice of the agent in any state s∈S; r is the reward function, S×A→r, where r(S,a) represents the reward obtained by the agent for taking action a in state s; p is the state transition probability distribution, S×A×S→[0,1], defined as:

[0078]

[0079] Where p(s t+1 =s'|s t =s,a t =a) is the conditional probability of transitioning from state s to state s' when action a is taken;

[0080] ω is the observation probability distribution, O×S→[0,1], defined as:

[0081]

[0082] Where ω(o t =o|s t =s) is the conditional probability of observing o∈O given a state s∈S;

[0083] γ is a discount factor, γ∈[0,1), which discounts the value of future rewards. A smaller γ indicates a preference for short-term rewards, while a larger γ indicates a preference for long-term rewards.

[0084] μ is the initial state distribution, μ∈Δ(S), where μ(s0)=p(s0=s) is the probability that the agent starts from state s0∈s.

[0085] The environment starts in the initial state s0~μ. At time step t, the agent is in state s t ∈S, execute action a t ∈A, receives an immediate reward r t+1 (s t ,a t ,s t+1 ), and according to the transition probability p(s t+1 |s t ,a t )Transfer to state s t+1 However, the agent does not directly observe s t The agent's observation o t ∈O, where ω(o t |s t ) is the observed o tGiven state s t probability.

[0086] Step S02: Generate trajectories from a partially observable environment and compress the trajectories, remove continuously repeated high-level observations, and retain key change points.

[0087] A trajectory is a sequence that represents the interaction of an agent with its environment, capturing the states visited, actions taken, and rewards received. It is formally defined as:

[0088] T={(s0,a0,r0),(s1,a1,r1),…,(s t ,a t ,r t )}

[0089] Among them, s t represents the state of the agent at time step t, a t represents the action taken by the agent at time step t, r t represents the reward the agent receives from the environment at time step t.

[0090] In partially observable environments, the agent cannot fully observe the entire state of the environment. Instead, it relies on observations to make decisions. In this case, the trajectory is defined as:

[0091] T={(o0,a0,r0),(o1,a1,r1),…,(o t ,a t ,r t )}

[0092] Among them: t represents the observation value of the agent at time step t.

[0093] A predefined set of Boolean propositions P is defined by a labeling function L(e):e→2 P Each trajectory T i Converted into a sequence of truth value assignments to the set P, as follows:

[0094]

[0095] Among them, σ i,t is the labeling function L(e t ) gives the high-level observation, e t Represents the experience at time t. This transformation represents the interaction between the agent and the environment from a higher level, providing a more abstract data form for subsequent processing.

[0096] In the abstract space defined by L and P, the same high-level observations often appear multiple times consecutively. Compression removes these repeated high-level observations while retaining key change points. When computing resources are limited, this can significantly improve the quality of the reward automata found by the model, reduce the computational cost of traversal trajectories, and help the model find high-quality reward automata more quickly.

[0097] Step S03: construct an optimal reward automaton through the trajectory, use genetic local search to learn the reward automaton and solve the optimal reward automaton.

[0098] In partially observable Markov decision processes with incomplete observations, traditional Markov properties and convergence guarantees are no longer feasible. To better learn policies, historical data must be incorporated. By embedding memory in partially observable environments using reward automata (RAs), this paper employs RAs, originally designed as structured representations of reward functions and capable of learning from experience, as memory aids for reinforcement learning (RL) algorithms.

[0099] The optimal reward automaton is defined as the tuple:

[0100] R P = <U,φ,e0,δ u ,δ r ,x i,t ,N u,σ >

[0101] Where: U is a finite set of states, e0∈U is the initial state; φ is the value of the potential energy function; δ u ∈U×2 ∑ →U is the state transition function; δ r ∈U×2 ∑ →R is the reward function; auxiliary variable x i,t ∈U represents the reward automaton in the observation trajectory T i until the state after time t, and x i,t+1 =δ u (x i,t ,L(e i,t+1 )); is the set of all next high-level observations σ that appear in the reward automaton state u and the high-level observations σ in the trajectory T, which satisfies

[0102] For P o (o0,a0,...,o t ,a t ), any trajectory generated by any strategy on it is:

[0103] Pr(o t+1 , r t |Ti )=Pr(o t+1 ,r t |o t ,x t ,a t )

[0104] Where x0=u0 and x t =δ u (x t-1 ,L(o t-1 ,a t-1 , o t )).

[0105] The goal is to determine the current RA state x i，t , find a i,t ) predict L(e i,t+1 )'s reward automaton R P , where e i,t+1 The experience represented is (o i,t ,a i,t ,o i,t+1 ), which is a triple consisting of the observation at time t, the action taken, and the observation at time t+1. i,0 Defined as (Φ, Φ,o i,0 ), where the first two Φ can be understood as some information that does not exist at the beginning, o i,0 is the initial observation model parameter. The parameters of the model include the trajectory set T, which stores a series of trajectories generated by the interaction between the agent and the environment. These trajectories are used to train the reward automaton R. P Crucially, R P It is necessary to learn the patterns and laws of the environment from these trajectories, and then predict future observation labels based on the current state and existing experience. By learning the trajectory set T, R P Able to continuously adjust its own parameters and state transfer mechanism to improve the prediction L(e i+1,t ) accuracy, thereby better guiding the agent's decision-making and actions in partially observable environments and helping the agent learn better strategies.

[0106] The objective function of the target reward automaton is as follows:

[0107]

[0108] It is defined as maximizing the log-likelihood of predicting future observations and minimizing the complexity of the reward automaton. This ensures that the learned reward automaton effectively represents the environment and can be used as memory in the reinforcement learning algorithm. The objective function uses the state transition probability δ uTo predict the likelihood of the next observation, we use the current state and transfer function of the reward automaton given the current state and action. It also considers the number of states and transitions in the reward automaton to ensure simplicity and efficiency.

[0109] A key feature of this formulation is that as the number of training trajectories approaches infinity, any optimal reward automaton is an optimal solution to the target reward automaton of the objective function.

[0110] Heuristic methods often produce the most effective results when solving large-scale problems. Heuristics provide polynomial-time approximate solutions to NP-hard problems, typically focusing on finding good solutions rather than guaranteeing strong optimality. This paper proposes a genetic local search method that treats each set of trajectories as a population, evaluates the fitness of each individual to assess its performance within the problem domain, and performs a local search on the individuals in the population after each generation of genetic manipulation. This method balances the trade-off between global and local search in heuristic algorithms, further optimizing their solutions.

[0111] The following demonstrates the process of learning a reward automaton using genetic local search:

[0112]

[0113]

[0114] The neighborhood of a reward automaton is defined as those reward automata that differ only by one transition path, i.e. by removing, adding or modifying a transition. These automata are evaluated using the objective function of the target reward automaton. The algorithm treats the trajectories as a population and formalizes the process of learning reward automata using genetic operations on the population. Each iteration involves genetic operations on the reward automaton, including selection, crossover and mutation, based on the evaluated advantages and disadvantages, which correspond to fitness. Figure 2 As shown, the specific steps are:

[0115] Step S031: Initialization: receiving observation set Σ, training trajectory T, maximum reward automaton state number u max and the time limit or maximum number of steps t max It also receives a randomly generated initial reward automaton R and the number of iterations n. Auxiliary variables are defined to help learn the reward automaton. The initial conditions are as follows:

[0116] The current time t is set to 0; the optimal cost c* is initialized to infinity ∞; the optimal reward automaton R* is initialized to "None"; and the initial population G is set to empty.

[0117] Step S032: Optimizing the reward automaton, including:

[0118] Step S0321: Evaluate the cost of the current reward automaton: In each iteration, first evaluate the cost c of the current reward automaton R on the given trajectory T, as shown below:

[0119]

[0120] Among them, E represents the state set, which is the state transition sequence in the trajectory T, |f (o,i) | represents f (o,i) length.

[0121] Step S0322: Update the optimal solution: If the current cost c is less than the optimal cost c * , then update the optimal solution to the current solution. Update the current reward automaton R to the optimal reward automaton R * , and calculate the current population and fitness. The fitness of individuals in the current population is as follows:

[0122] f(G * )=(f1(g1),f2(g2),...,f n (g n ))

[0123] Among them, g i is an individual in the population. If normalization is required, the fitness function is improved as follows:

[0124]

[0125] Here, Δf represents the range of f, i.e., max(f)-min(f).

[0126] Step S033: Genetic operation: transform the population G using the mapping function * To adapt to the next genetic operation, return the transformed population and fitness function f. Amdahl's law limits the achievable speedup S by considering the proportion of tasks that remain serial, as follows:

[0127]

[0128] Where n is the number of processing units and α is the fraction of tasks that remain serial. The selection probability is as follows:

[0129]

[0130] Among them, f o is the fitness value of individual o, It is the sum of all individual fitness values in the game.

[0131] Step S034: Neighborhood search: Get the neighborhood A of the current reward automaton R. The neighborhood consists of reward automata that differ from the current reward automaton by a single transformation (e.g., adding, deleting, or modifying a transformation). For each reward automaton R in the neighborhood n ∈A is evaluated, and the cost c of each domain individual is obtained through the evaluation function n When the maximum number of steps t is reached max Or the maximum number of iterations, return the optimal reward automaton R * .

[0132] Step S04: Extend the reward automaton to a partially observable Markov decision process to form a potential-based reward automaton in partially observable reinforcement learning (GLS-PRA): Given a partially observable Markov decision process P0 and an optimal reward automaton R P , where P0 and R P Shared labeling function L, then the potential-based reward automaton in partially observable reinforcement learning is defined as:

[0133]

[0134] in:

[0135] The combined state space of the original partially observable Markov decision states and reward automaton states;

[0136] The initial state consists of the initial state of the partially observable Markov decision process and the initial state of the reward automaton;

[0137] is the potential energy function defined for the reward structure;

[0138] Γ:O×A→∑ is a labeling function that maps partially observable Markov decision process transformations to automaton symbols;

[0139] The current state of the reward automaton. a∈A represents the action performed by the agent in the current state. (s',e') is the combination of the next state, where s'∈S represents the next state of the environment and e'∈U represents the next state of the reward automaton.

[0140] is the state transition probability.

[0141] In a partially observable Markov decision process, the agent follows the labeling function L:S×A×S→2 ∑ Execute action a to transition from state s to state s'. If<e,L(s,a,s')> ∈dom(δ e), then R P Transition from state e to Otherwise, the reward automaton remains in its original state.

[0142] Assume that the state space S and action space A are finite in a partially observable Markov decision process. For all states s and actions a, the immediate reward r(s,a) has a constant R max , so that |r(s,a)|≤R max The policy π(a|s,θ) is a parameter and is continuously differentiable with respect to the parameter θ. The discount factor γ satisfies 0<γ<1.

[0143] Step S05: Use a value iteration strategy to calculate the potential value of the potential-based reward automaton for partially observable reinforcement learning: The core idea is to view the reward automaton as a partially observable Markov decision process that approximates the expected discounted reward for each reward automaton state. By assigning a potential value to each reward automaton state, a reward function is defined to incentivize the agent to complete the task. This approach is only applicable to simple reward automata because it does not contain information about the environment state S and the action a∈A.

[0144] Intuitively, this can be viewed as a partially observable Markov decision process, where each state transition in the reward automaton corresponds to a deterministic action. Under the optimal strategy, numerical iteration and other methods can be used to calculate the state v j (u) = maxq j The value of (u,σ). This calculation is given U,F,P,δ u ,δ r and γ are performed as follows:

[0145]

[0146]

[0147] A value iteration strategy is used to calculate the potential energy function of the reward automaton (RA) state, providing a dynamic guidance signal for reward shaping. The RA state is considered the state space of a Markov decision process. The optimal value function of each state is iteratively solved using the Bellman optimality equation and then converted into a potential energy function. This ensures that reward shaping does not change the optimal policy but accelerates learning.

[0148] Step S051: Initialization: state list U, state transition function δ u , state reward value U r , terminal state term u , the reward forms a discount factor γ. The initial conditions are as follows:

[0149] Initialize the value function V(u)=0(u∈U) of all states.

[0150] Step S052: Value iteration update: traverse non-terminal state u∈U-term u Update value function:

[0151]

[0152] Where Σ is the observation symbol set, δ u (u,σ) is the state transition result. Calculation error:

[0153] ∈←max(∈,|V(u)-V old (u)|)

[0154] Step S053: Convergence judgment: Repeat iteration until ∈<∈ min , ensuring that the value function converges to the optimal solution V * .

[0155] Step S054: Potential energy function conversion: convert the optimal value function into a potential energy function: φ(u)←-V * {u}.

[0156] In this embodiment, six scenarios are used to specifically illustrate the test results of the present invention, and the specific contents are as follows.

[0157] Case 1:

[0158] Case Background: This case study is set in a smart factory production workshop, divided into three distinct work areas connected by walkways. In one area, a control terminal allows workers to initiate a material allocation program, which randomly generates special parts for collection in the other two areas. Various equipment and shelves within the workshop can obstruct workers' vision, making the search more difficult.

[0159] After starting the program, workers must quickly search for and collect special parts in the other two work areas. Each successfully collected part earns a +1 reward. The goal is to collect as many parts as possible within the allotted time, while also strategically planning the search path to minimize unnecessary movement and maximize efficiency.

[0160] First, a large amount of surveillance camera image data within the workshop is preprocessed and annotated to identify key elements such as different work areas, aisles, control terminals, and special parts. A deep learning algorithm is then used to train an object recognition model that accurately identifies these elements, thereby constructing a state space. The dimensionality of the state space is determined by the complexity of the workshop and other factors to be considered. For example, it may include the location coordinates of workers, the presence and quantity of parts in each area, and the operating status of equipment.

[0161] The action space is defined as operations such as moving forward, moving backward, turning left, turning right, entering the work area, and collecting parts. For each action, corresponding execution rules and conditions are set. For example, when moving forward, the presence of obstacles ahead must be considered, and when entering the work area, access control conditions must be met.

[0162] Using the GLS-PRA method, workers on the shop floor select actions based on an initial strategy formed from monitoring information and their own experience. At each decision moment, the agent selects an action based on the current state information and strategy. For example, if a worker is in an aisle and the object recognition model determines that a certain work area may contain parts, the worker may choose to enter that work area.

[0163] When a worker performs an action, the state of the environment changes accordingly, and the reward automaton evaluates the new state and action. If the worker successfully finds and collects a part, the reward automaton provides positive feedback, such as a reward of +1, and adjusts the potential reward value based on the reward shaping mechanism to reinforce this behavior pattern. If a worker wastes too much time in an area without parts, the reward automaton gives a lower reward, such as -0.5, prompting the worker to adjust their search strategy. The learning process of the reward automaton is continuously optimized through iteration. It adjusts its state transition function and reward function based on a large amount of historical action and reward data to better guide the behavior of the agent.

[0164] During training, the genetic algorithm component of the GLS-PRA method performs crossover and mutation operations on different strategy combinations to generate a new strategy population. It then evaluates the performance of these new strategies in the environment (i.e., cumulative rewards and task completion efficiency) to select the optimal strategy for the next generation of evolution. Simultaneously, a local search algorithm searches within the neighborhood of the currently optimal strategy to further fine-tune the strategy and improve its performance. Through multiple task executions and continuous learning iterations, workers are able to gradually optimize path planning, improve parts collection efficiency, and achieve higher cumulative rewards.

[0165] Case 2:

[0166] Case Background: This case study is set in a large data center consisting of three computer rooms. The doors between the rooms utilize an electronic access control system, requiring specific authorization key cards to open. The first computer room houses a management console where data center operations personnel can access task information and system prompts. The final computer room houses critical data backup equipment, requiring regular maintenance access by operations personnel. Authorization key cards are randomly placed in specific cabinets or drawers within the first two computer rooms.

[0167] The O&M personnel's task is to search the computer room for the authorization key card, open the corresponding door, and ultimately enter the room housing the data backup equipment to complete the maintenance operation. Completing the task earns a +1 reward. During this task, they must avoid the risk of equipment failure or data loss due to misoperation, while also ensuring efficient retrieval of the key card and planning the door opening path.

[0168] Leveraging data from sensors within the computer room (such as temperature sensors, humidity sensors, door magnetic sensors, and equipment status sensors) and surveillance video, the data is cleaned and annotated to identify key elements such as the computer room, doors, key card storage locations, and the location of data backup devices. This annotated data is then used to train an environmental perception model, for example, using a deep learning object detection algorithm, to construct a state space. In addition to the location of maintenance personnel, the state space also includes information such as the operating status of equipment within each computer room, the open and closed status of doors, and the probability of key card locations. This information is updated in real time as maintenance personnel perform their actions and the environment changes.

[0169] The action space includes operations such as moving within the computer room, opening cabinets or drawers, picking up key cards, and using key cards to open doors. For each action, detailed operation procedures and conditional judgments are set. For example, opening a cabinet or drawer requires meeting specific permissions or operation sequences, and using a key card to open a door requires ensuring that the key card matches the door.

[0170] Using the GLS-PRA method, operators execute actions based on sensor feedback and current policies during the exploration process. In the initial phase, operators develop a preliminary search strategy based on task information provided by the management console and their own experience. For example, they might first inspect key areas within the computer room or prioritize certain cabinets based on their importance.

[0171] When the operator performs an action, the state of the environment changes, which is evaluated by the reward automaton. If the operator successfully picks up the correct key fob and opens the door, the reward automaton provides a positive reward, such as +1, and adjusts the potential reward value based on the reward shaping mechanism to encourage continued progress. If the operator attempts to open the door with the wrong key fob or searches aimlessly within a computer room for a long time, the reward automaton provides negative feedback, such as -0.5, to guide the operator in improving their strategy. During the learning process, the reward automaton continuously analyzes the operator's action sequences and the corresponding rewards, optimizing its parameters through a combination of a genetic algorithm and a local search algorithm. The genetic algorithm performs crossover and mutation operations on different individual strategies to generate a new population of strategies. The performance of these new strategies is then evaluated through simulations in the real environment, and the best strategies are selected for evolution in the next generation. The local search algorithm fine-tunes the currently best strategy to find even better ones. After repeated training, the operator can improve their success rate in finding key fobs and opening doors, completing tasks more efficiently.

[0172] Case 3

[0173] Case Background: This case study is set in a power dispatch center, consisting of various monitoring rooms and operation rooms. Display screens in some monitoring rooms display specific power system status symbols, representing different equipment operating states or fault warnings. In specific operation rooms, a main control console displays target symbols and operational instructions for equipment requiring special attention. Dispatchers are required to navigate between these rooms, observing the symbols and making appropriate operational decisions.

[0174] The dispatcher's task is to find the correct target device in the operation room based on the symbols observed in other rooms and perform power dispatch operations according to the instructions. If the correct choice is made and the dispatch operation is successfully completed, the dispatcher receives a +1 reward. If the incorrect choice may cause power system instability, the dispatcher receives a -1 reward. Other meaningless operations are not rewarded. The key to the task is to accurately identify the symbols and make the correct power dispatch decisions, while moving quickly between rooms to ensure the safe and stable operation of the power system.

[0175] Utilizing the vast amount of surveillance video and power system operation data within the power dispatch center, the system processes and annotates the data, identifying key elements such as symbols in different rooms and display screens, equipment locations, and operational instructions. Deep learning image recognition and classification algorithms are used to train a symbol recognition and equipment location model, thereby constructing a state space. This state space includes information such as the dispatcher's current location, the display status of symbols in each room, equipment operating parameters, and the overall load of the power system. This information is continuously updated over time and based on the dispatcher's actions.

[0176] The action space is defined as actions such as moving between rooms, observing display symbols, and selecting target devices for operation. For each action, detailed execution rules and conditional restrictions are established. For example, moving between rooms requires compliance with security passages and access control regulations, and selecting target devices requires ensuring the correct operation permissions and sequence.

[0177] Using the GLS-PRA method, the dispatcher selects actions at each time step based on current symbolic observations and a strategy formed from their own experience. Initially, the dispatcher develops a basic operational strategy based on the power system's normal operating conditions and past experience. For example, they observe and operate key equipment in order of priority.

[0178] When the dispatcher performs an action, the environment state changes, which the reward automaton evaluates. If the dispatcher successfully recognizes the symbol and makes the correct equipment operation, the reward automaton grants a reward, such as +1, and adjusts the potential reward value based on the reward shaping mechanism to reinforce this cognitive and decision-making process. If the dispatcher makes an incorrect choice, the reward automaton imposes a penalty, such as -1, to guide the dispatcher in learning the correct symbol association and equipment operation strategy. During the learning process, the reward automaton evolves different strategy combinations using a genetic algorithm to generate a new strategy population. The reward automaton then evaluates the performance of these new strategies in a simulated power dispatching environment and selects the optimal strategy for the next generation of evolution. Simultaneously, a local search algorithm searches within the neighborhood of the currently optimal strategy to further optimize the strategy. After repeated training and practical operation, the dispatcher is able to improve the accuracy of symbol recognition and the correctness of equipment operation decisions, thereby ensuring the normal operation of the power system and obtaining higher cumulative rewards.

[0179] Case 4

[0180] Case Background: This case study is set in a research laboratory with four experimental areas for different research projects. In the first experimental area, researchers are tasked with matching symbols associated with experimental data or samples. For example, in a biological experiment, different combinations of symbols for gene sequences might be displayed. Based on the symbol knowledge learned in the first experimental area, researchers need to find matching symbols on the experimental samples or data in the subsequent three experimental areas.

[0181] After selecting a symbol task in the first experimental area, researchers will proceed to the other three experimental areas in sequence, searching for experimental samples or data corresponding to the symbol that matches the initial symbol in each experimental area and conducting corresponding experimental analysis or recording. Each correct match and completion of the relevant experimental task will earn a +1 reward, while an incorrect match will earn a -1 reward. This task requires researchers to possess excellent memory and symbol recognition skills, as well as the ability to efficiently plan movement between experimental areas. This will ensure that researchers complete as many correct matches as possible within the specified timeframe and advance the research project.

[0182] Based on a database of scientific research projects and a large amount of laboratory environment image data, the data is annotated and organized to identify key elements such as symbols on different experimental areas, experimental samples or data, as well as related experimental equipment and operating procedures. A symbol recognition and classification model is trained using deep learning image recognition and sequence analysis algorithms. This model then constructs a state space based on the researcher's location information and task progress. The dimensions of the state space may include information such as the researcher's experimental area, the number of completed tasks, the number and type of symbols to be matched within each experimental area, and the availability of experimental equipment. This information is continuously updated as the researcher's actions and experimental progress progress.

[0183] The action space includes operations such as moving between experimental areas, selecting experimental samples or data, and confirming symbol matching. For each action, clear operating specifications and conditional judgments are set. For example, moving between experimental areas must comply with laboratory safety regulations and channel layout, and selecting experimental samples or data requires meeting specific experimental requirements and operating permissions.

[0184] Using the GLS-PRA method, researchers make decisions based on their current knowledge and strategies during the task execution process. In the initial phase, researchers develop a preliminary experimental plan and action strategy based on their understanding of the task and previous research experience. For example, they search and match experimental areas in a logical order.

[0185] When researchers perform actions, the state of the environment changes, which is then evaluated by the reward automaton. If researchers successfully match symbols in subsequent experimental areas and complete the task, the reward automaton grants a positive reward, such as +1, and adjusts the potential reward value based on the reward shaping mechanism to encourage researchers to continue their research methods. If a mismatch occurs, the reward automaton grants a negative reward, such as -1, to encourage researchers to review their knowledge and adjust their strategies. During the learning process, the reward automaton uses a genetic algorithm to perform crossover and mutation operations on different individual strategies to generate a new population of strategies. The reward automaton then evaluates the performance of these new strategies in a simulated scientific experimental environment, selecting the best strategies for the next generation of evolution. Simultaneously, a local search algorithm fine-tunes the current best strategy to find even better ones. Through continuous research and practice, researchers can improve the accuracy of symbol matching and optimize their movement paths between experimental areas, thereby better completing their research tasks and earning higher rewards for their research achievements.

[0186] Case 5:

[0187] Case Background: This case study is set in a modern logistics park with four cargo sorting areas. At the park's management office, couriers can access a task terminal and, by clicking a specific button, randomly generate sorting tasks for high-value packages in one of the four cargo sorting areas. The numerous shelves, transport vehicles, and other staff within the logistics park present obstacles to couriers' movements.

[0188] After receiving the task terminal and triggering the high-value package sorting task, the courier's task is to quickly locate the corresponding cargo sorting area and accurately sort the high-value package. Each high-value package successfully sorted is considered a +1 reward. The difficulty of the task lies in quickly locating the target area in the complex logistics park environment, while also rationally planning the route to avoid collisions and delays.

[0189] The state space is constructed using electronic map data of the logistics park and courier location information, combined with data such as the type and time of cargo sorting tasks. This includes the courier's current location coordinates, the task status of each cargo sorting area (whether there are high-value parcel sorting tasks), traffic conditions within the park (the location and direction of transport vehicles), shelf layout and occupancy, and other information. This information is updated in real time through the park's management system.

[0190] The action space includes operations such as walking through campus corridors, driving transport vehicles, querying campus guide information, and heading to the cargo sorting area. For each action, corresponding execution rules and conditional restrictions are set. For example, walking through campus corridors requires compliance with traffic rules and safety regulations, while driving transport vehicles requires corresponding driving qualifications and operating procedures.

[0191] Using the GLS-PRA method, couriers navigate the campus, choosing their actions based on navigation prompts from their handheld devices and strategies developed from their own work experience. Initially, couriers develop a preliminary search strategy based on the campus' layout and previous sorting experience. For example, they prioritize sorting areas close to the management office or with a historically high frequency of high-value packages.

[0192] When a courier performs an action, the state of the environment changes, which the reward automaton evaluates. If the courier successfully reaches the area with a high-value package sorting task and completes the sorting, the reward automaton grants a positive reward, such as +1, and adjusts the potential reward value based on the reward shaping mechanism to reinforce this behavior. If the courier takes too long to find the target area or deviates from the path due to other factors, the reward automaton grants a lower reward, such as -0.5, to guide the courier to adjust their work strategy. During the learning process, the reward automaton uses a genetic algorithm to evolve different strategy combinations to generate a new strategy population. The reward automaton then evaluates the performance of these new strategies in a simulated logistics park environment and selects the best strategy for the next generation of evolution. Simultaneously, a local search algorithm searches within the neighborhood of the currently best strategy to further optimize the strategy. By participating in such tasks repeatedly, couriers become more proficient in sorting high-value packages in the logistics park, improving their work efficiency and income.

[0193] Case 6

[0194] Case Background: This case study is set in a multi-story parking garage, divided into multiple floors and zones. Vehicles can only view information about the floor and zone they are currently in. Multiple verification checkpoints are set up at the exit of the parking garage, including license plate recognition, electronic payment verification, and vehicle safety checks. Only by passing these checks in sequence can vehicles enter the next zone and ultimately exit the parking garage.

[0195] The vehicle's mission is to navigate the parking lot, unlocking checkpoints by activating the corresponding verification steps, and ultimately navigate to the exit. Successfully exiting the parking lot will earn you a reward. During the mission, you must avoid getting lost or reaching dead ends in the parking lot, and accurately complete each verification step.

[0196] Using data from parking lot surveillance cameras and the vehicle's positioning system, a map and state space of the parking lot are constructed. This state space includes the vehicle's current location coordinates, floor and area information, the status of each verification checkpoint (passed or failed), and the parking situation of surrounding vehicles. This information is updated in real time as the vehicle moves and verification progresses.

[0197] The action space is defined as operations such as driving forward, driving backward, turning left, turning right, and initiating a verification phase. For each action, clear execution conditions and restrictions are set. For example, driving forward requires ensuring that the road ahead is clear, and initiating a verification phase requires the vehicle to be in the appropriate position and state.

[0198] Using the GLS-PRA method, a vehicle navigates the parking lot, executing actions based on sensor feedback and its current strategy. Initially, the vehicle develops a preliminary navigation strategy based on the parking lot's entrance location and the general direction of the exit. For example, it might choose to navigate the main exit corridor and prioritize routes with less traffic.

[0199] When the vehicle performs an action, the state of the environment changes, which the reward automaton evaluates. If the vehicle successfully passes verification and enters the next area, the reward automaton provides a positive reward, such as +1, and adjusts the potential reward value based on the reward shaping mechanism to encourage the vehicle to continue moving forward. If the vehicle fails verification for a long time in an area or goes in the wrong direction, the reward automaton provides negative feedback, such as -0.5, to guide the vehicle to adjust its driving strategy. During the learning process, the reward automaton uses a genetic algorithm to perform crossover and mutation operations on different strategy combinations to generate a new strategy population. The reward automaton then evaluates the performance of these new strategies in a simulated parking environment and selects the optimal strategy for the next generation of evolution. Simultaneously, a local search algorithm searches within the neighborhood of the current optimal strategy to further optimize the strategy. After repeated training and optimization, the vehicle gradually becomes familiar with the parking lot layout and verification process, navigating to the exit more efficiently.

[0200] exist Figure 3The performance of different reinforcement learning methods in the six partially observable environments described above is demonstrated in this paper. GLS-PRA, a method proposed in this paper, is based on genetic local search and potential reward automata. It optimizes samples through compressed trajectories, defines an optimization objective function, and uses genetic local search to find the optimal reward automata. It also employs a value iteration strategy to calculate potential energy, aiming to maximize the log-likelihood of future observations and minimize the complexity of the reward automata to guide the agent to learn the optimal policy. LRM+DDQN uses dual deep Q-networks to learn policies, combined with tabu search or local search to solve the target reward automata problem, and learns the reward automata through experience. LRM+DQRM uses an improved deep Q-learning and reward automata approach to learn policies, also combining tabu search or local search to solve the target reward automata problem, and learning the reward automata based on experience. Across all test environments, GLS-PRA performs exceptionally well in terms of cumulative rewards, demonstrating strong learning capabilities and efficient policy optimization. It can guide the agent to converge to a better policy more quickly, demonstrating its effectiveness in solving partially observable reinforcement learning problems. LRM+DDQN's reward values are relatively low across all domains, particularly in complex environments, where it struggles to learn the optimal policy, resulting in significantly reduced rewards. This suggests limitations in handling complex tasks and uncertainty. LRM+DQRM outperforms LRM+DDQN, showing some improvement across all domains, but overall still falls short of GLS-PRA, indicating room for improvement in policy learning and optimization.

[0201] Based on the same inventive concept, the present invention also proposes a device for a potential-based reward automaton in partially observable reinforcement learning. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. Figure 4 As shown, the device 100 includes:

[0202] POMDP construction module 101: used to construct a standard POMDP;

[0203] Trajectory generation module 102: used to generate trajectories from a partially observable environment and compress the trajectories, remove consecutive and repeated high-level observations, and retain key change points;

[0204] Solving module 103: used to construct an optimal reward automaton through trajectories, learn the reward automaton using genetic local search, and solve the optimal reward automaton;

[0205] Definition module 104: used to extend the reward automaton to the partially observable Markov decision process and define the potential-based reward automaton in partially observable reinforcement learning;

[0206] Calculation module 105: used to calculate the potential energy value of the potential energy-based reward automaton in partially observable reinforcement learning using a value iteration strategy.

[0207] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0208] like Figure 5 As shown, the device includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0209] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0210] The processing unit performs the various methods and processes described above, such as method steps S01 to S05. For example, in some embodiments, method steps S01 to S05 can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more of the method steps S01 to S05 described above can be executed. Alternatively, in other embodiments, the CPU can be configured to execute method steps S01 to S05 by any other appropriate means (for example, by means of firmware).

[0211] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0212] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0213] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0214] In addition, although adopting specific order to describe each operation, this should be understood as requiring such operation to be carried out in the specific order shown or in sequential order, or requiring all illustrated operations to be carried out to obtain desired result.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be construed as limiting the scope of the present invention.Some features described in the context of independent embodiment can also be realized in single realization in combination.On the contrary, the various features described in the context of independent realization also can be realized in multiple realizations individually or in the mode of any suitable subcombination.

[0215] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A potential-based reward automaton method for partially observable reinforcement learning, characterized in that: The method includes: Step S01: construct a standard POMDP; Step S02: Generate trajectories from a partially observable environment and compress the trajectories, remove consecutive and repeated high-level observations, and retain key change points; Step S03: constructing an optimal reward automaton through the trajectory, using genetic local search to learn the reward automaton and solve the optimal reward automaton; Step S04: Extend the reward automaton to the partially observable Markov decision process and define the potential-based reward automaton in partially observable reinforcement learning; Step S05: Use the value iteration strategy to calculate the potential energy value of the potential energy-based reward automaton in partially observable reinforcement learning.

2. The method of potential-based reward automata in partially observable reinforcement learning according to claim 1, characterized in that The specific content of step S01 is: Define a standard partially observable Markov decision process as a tuple: P o =(S,O,A,r,p,ω,γ,μ) Where: S is a finite set of states, where each state s∈S represents a possible situation in the environment; O is a finite set of observations, where each observation o∈O represents a possible outcome perceived by the agent in a given state; A is a finite set of actions, where each action a∈A represents the available choice of the agent in any state s∈S; r is the reward function; p is the state transition probability distribution; ω is the observation probability distribution; γ is the discount factor; μ is the initial state distribution.

3. The method of potential-based reward automata in partially observable reinforcement learning according to claim 1, characterized in that The trajectory described in step S02 is a sequence representing the interaction of the agent with its environment, capturing the states visited, actions taken, and rewards received, defined as: T={(s0,a0,r0),(s1,a1,r1),…,(s t ,a t ,r t )} Among them, s t represents the state of the agent at time step t, a t represents the action taken by the agent at time step t, t t represents the reward the agent receives from the environment at time step t.

4. The method of potential-based reward automata in partially observable reinforcement learning according to claim 3, characterized in that: In partially observable environments, the agent relies on observations to make decisions, and the trajectory is defined as: T={(o0,a0,r0),(o1,a1,r1),…,(o t ,a t ,r t )} Among them: t represents the observation value of the agent at time step t.

5. The method of potential-based reward automaton in partially observable reinforcement learning according to claim 1, characterized in that The specific steps of step S03 are: Step S031: Initialization: receiving observation set Σ, training trajectory T, maximum reward automaton state number u max and the time limit or maximum number of steps t max The termination criteria receives the randomly generated initial reward automaton R and the number of iterations n. The initial conditions are: the current time t is set to 0; the optimal cost c* is initialized to infinity ∞; the optimal reward automaton R* is initialized to "None"; the initial population G is set to empty; Step S032: Optimizing the reward automaton; Step S033: Genetic operation: transform the population G using the mapping function * To adapt to the next genetic operation; Step S034: Neighborhood search: Get the neighborhood A of the current reward automaton R, and search for each reward automaton R in the neighborhood. n ∈A is evaluated, and the cost c of each domain individual is obtained through the evaluation function n , when the maximum number of steps t is reached max Or the maximum number of iterations, return the optimal reward automaton R * .

6. The method of potential-based reward automaton in partially observable reinforcement learning according to claim 5, characterized in that The specific steps of step S032 are: Step S0321: Evaluate the cost of the current reward automaton: In each iteration, first evaluate the cost c of the current reward automaton R on the given trajectory T, as shown below: Among them, E represents the state set, which is the state transition sequence in the trajectory T, |f (o,i) | represents f (o,i) length; Step S0322: Update the optimal solution: If the current cost c is less than the optimal cost c * , then update the optimal solution to the current solution, and update the current reward automaton R to the optimal reward automaton R * , and calculate the current population and fitness. The fitness of individuals in the current population is as follows: f(G * )=(f1(g1),f2(g2),...,f n (g n )) Among them, g i are individuals in a population; If normalization is required, the fitness function is improved as follows: Here, Δf represents the range of f, i.e., max(f)-min(f).

7. The method of potential-based reward automaton in partially observable reinforcement learning according to claim 2 or 6, characterized in that: The specific steps of step S04 are: Given a partially observable Markov decision process P0 and an optimal reward automaton R P , where P0 and R P Sharing the labeling function L, we define a partially observable Markov decision process with a potential-based reward automaton as: in: The combined state space of the original partially observable Markov decision states and reward automaton states; The initial state consists of the initial state of the partially observable Markov decision process and the initial state of the reward automaton; Ψ: is the potential energy function defined for the reward structure; Γ:O×A→Σ is the labeling function that maps partially observable Markov decision process transformations to automaton symbols; is the reward function; is the state transition probability.

8. A device for a potential-based reward automaton in partially observable reinforcement learning, characterized in that The device implements the method according to any one of claims 1 to 7, comprising: POMDP construction module: used to construct standard POMDP; Trajectory generation module: used to generate trajectories from partially observable environments and compress them, removing repeated high-level observations and retaining key change points; Solving module: used to construct the optimal reward automaton through trajectory, learn the reward automaton using genetic local search and solve the optimal reward automaton; Definition module: used to extend the reward automaton to partially observable Markov decision processes and define the potential-based reward automaton in partially observable reinforcement learning; Calculation module: used to calculate the potential value of the potential-based reward automaton in partially observable reinforcement learning using the value iteration strategy.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images