Intelligent agent target searching method based on time relation context

Abstract

The invention relates to an agent target searching method based on a time relation context. According to the method, firstly, a context matrix of an object is extracted through an RGB image obtained by an intelligent agent under a central view angle, and target information in a current frame is represented; then, an object scene graph is constructed using a graph convolutional network to capture relationships between objects in the environment. Then, the context matrix of the current frame is stored in a memory pool so as to focus on the most relevant frame information, thereby improving the navigation performance. In order to enhance the generalization ability of the target, the weight of the strategy network is dynamically adjusted by integrating semantic information of the object and the frame. Specifically, through an adaptive weight strategy module, a navigation strategy can be used for navigation according to different targets. Finally, the target navigation of the intelligent agent in an unknown environment is realized. The technical scheme provided by the invention can significantly improve the target search capability of the intelligent agent, and is suitable for the fields of intelligent robots, automatic navigation, augmented reality and the like.

Description

Technical Field

[0001] The present invention relates to an intelligent agent target search method based on temporal relationship context, which is particularly applicable to navigation and target positioning in three-dimensional environments. This technical field covers multiple cutting-edge fields such as intelligent navigation systems, computer vision and image processing, deep learning and reinforcement learning. This method effectively solves the limitations that the intelligent agent can only search within a specific target range and cannot understand and reason about long-term targets. By introducing two core modules, the present invention successfully copes with the target positioning challenges faced by intelligent agents in complex environments. First, the target semantic trajectory module uses the memory-stored context matrix and self-attention mechanism to model spatial relationships, and captures the relationships between objects through a graph convolutional network. Secondly, this method can dynamically adjust the weight parameters according to the target semantics, thereby adapting to the changes of the target in space and significantly improving the navigation performance. Background Art

[0002] Visual navigation refers to guiding an agent from its current position to a predetermined target through visual observation. One of the traditional solutions is the simultaneous localization and mapping method, which relies on the construction of a geometric map to simplify path planning. However, due to the lack of previous scene data, SLAM is very complex when adapting to new environments and requires map reconstruction in unexplored environments, which increases uncertainty and initial instability. In addition, the sensitivity of the sensor and the accumulation of noise in SLAM data processing can also reduce the robustness of the method.

[0003] With the rise of deep learning, a paradigm shift has occurred in the field of visual navigation. Reinforcement learning-based algorithms can simultaneously capture two key aspects of the task in an end-to-end framework: semantic scene understanding and path planning. This approach no longer relies on explicit maps, and the agent can rely on visual information to identify target objects in unknown environments and plan paths to find the target. This approach reduces the need to build maps and improves operational efficiency, however, there are still some challenges, especially in dynamic environments, where the role of memory is crucial because a single time frame often cannot accurately estimate motion.

[0004] In indoor environments, agents face a complex task: not only identifying a target, but also taking appropriate actions based on observed information to reach the target. This task is further complicated because agents need to adapt to the dynamic changes of the surrounding environment in real-time processing. When performing long-term tasks in partially observable environments, agents must build a holistic understanding of the surrounding environment based on historical observations, however, existing work fails to build an understanding of the environment based on historical frames. Initially, the strategy to solve time-related problems was to superimpose past observations to form a historical context, thereby preserving memory. However, in recent years, agents have been endowed with recurrent neural networks that combine past hidden states with current frame features into a hidden state vector. Although RNN-based variants perform well on a variety of problems, they have difficulty learning long sequences, and long-term memory is crucial for making wrong long-term decisions, which can lead to catastrophic failures. To address the long-term memory problem, strategies have been proposed in recent years to incorporate external memory into the environment representation to enhance learning capabilities.

[0005] In this context, many memory neural networks have emerged, which can be divided into general memory and map-type memory. The storage capacity of map-type memory is closely related to the navigation context, so its scalability is somewhat limited. In contrast, general memory performs better in terms of reliability and persistence of stored values. However, when agents face similar situations, they are prone to fall into cyclic trajectories because they cannot effectively distinguish the current situation from historical observations and extract goal-related information from them. In addition, most existing zero-shot learning object navigation models rely on semantic embeddings (such as Word2Vec) as the main semantic auxiliary source. These methods have promoted the development of zero-shot learning by leveraging the interaction between policies to map the spatial relationship between goals and objects in the scene. However, they often directly regard semantic embeddings as a replacement for traditional single-hot encoding labels, while ignoring the need to design a dedicated network architecture for semantic embeddings. Although Zhao et al. proposed a method to transfer unknown categories to the policy network by combining the target semantic trajectory offset weight, this method failed to effectively distinguish the offset relationship between the target and the current object, resulting in only marginal improvements. Therefore, these methods still have difficulties in deeply exploring and effectively utilizing semantic embedding relationships, limiting the transferability of policies in zero-shot settings. Summary of the invention

[0006] The present invention aims to solve the stability and adaptability problems of existing intelligent agent navigation methods when facing unknown environments, especially how to effectively handle long-term memory in partially visible scenes. The present invention aims to provide an intelligent agent target search method based on temporal relationship context to solve this problem.

[0007] The intelligent agent target search method based on time relationship context includes the following steps:

[0008] 1. Goal and agent state space construction: First, according to the task requirements, determine the state space input and output of the agent in reinforcement learning. The target is specified as the name of the target to be searched, and the input of the agent is the RGB image obtained through the camera as a representation of the environment state; the output of the agent is the action generated by the model, which is used to guide the agent to perform corresponding behaviors in the environment.

[0009] 2. Build a target search model: Build a target search model based on the temporal relationship context. The model diagram is as follows: Figure 2 As shown in Figure 2, the model combines time series information, uses the agent's historical state and current environment information to achieve intelligent search and positioning of the target;

[0010] 3. State space preprocessing: The agent interacts with the environment. By obtaining the agent's first-person perspective RGB image and preprocessing the image, the current state space representation of the environment is obtained. The state space representation reflects the relationship between the objects and the target in the current environment.

[0011] 4. Strategy generation and reward calculation: The current state space is input into the target search model, and the model generates the agent's current search strategy (i.e. the selected action) based on the state. At the same time, the agent interacts with the environment based on the selected action and calculates the reward value;

[0012] 5. Strategy update: Evaluate the current strategy through the Critic network in the A3C algorithm, calculate the advantage function based on the reward obtained by the agent and the value function of the Critic network. Use the advantage function to update the strategy of the Actor network.

[0013] The step 1 is specifically as follows:

[0014] In the autonomous target search task, the agent cannot directly obtain the global environment state and can only rely on the RGB images of the first-person camera as input, which provide local environment information. The text information of the specified target defines the specific requirements of the task and indicates the target category that the agent needs to search. By combining the visual input and the target text information, the agent infers the target location and executes the corresponding search strategy.

[0015] The step 2 is specifically as follows:

[0016] The overall architecture of the model consists of two main sub-modules: the object semantic trajectory module (OST) and the adaptive weight policy module (AWP). The construction process of the model is divided into the construction of two modules:

[0017] (1) Constructing the Object Semantic Trajectory Module (OST)

[0018] Initialize the memory bank: implement a circular buffer to store the historical semantic features of objects in the scene. The memory length is set to T to save the spatiotemporal relationship and semantic information of the objects. Design the memory attention mechanism (MemoryAttention): extract the historical features of all frames from the memory bank. Define the projection matrix W q , W k , W v , map the feature data into Query, Key, and Value. Use the dot product to calculate the similarity between the query and the key:

[0019]

[0020] The Softmax normalization weights are used to perform weighted calculations on the values ​​to generate global spatiotemporal features. The extracted spatiotemporal features are used as the core representation of the semantic trajectory and passed to the strategy module.

[0021] (2) Constructing the Adaptive Weight Policy Module (AWP)

[0022] The adaptive weight strategy module aims to adapt to the needs of different target navigation tasks by dynamically adjusting the weights of the policy network. This module is based on the Actor network method, combined with target embedding features and scene semantic information, and is implemented in two parts: first, the target embedding features are mapped to the policy space and directly integrated into the input of the policy network through a linear layer; second, a semantic weight bias matrix is ​​generated using a hypernet, which consists of two fully connected layers and uses a ReLU activation function to project the target embedding features into the policy weight space. The weights of the final policy network are obtained by combining the initial weights with the semantic weight bias, thereby dynamically adjusting the policy to adapt to different targets or scenarios. For unknown target categories, the module migrates the policies of known categories through an offset weight mechanism.

[0023] The step 3 is specifically as follows:

[0024] Define the object matrix as the state space s of the model input t : Different from other visual navigation methods, we abandon the use of visual features or knowledge graphs and only rely on the detection box value matrix and the cosine distance of the target embedding as input. We choose not to use visual data as state representation to reduce the variation of visual information caused by different environments and objects. Our state representation structure is a 20-row 5-column matrix, where each row is represented by C i =[t,x c ,y c , area, sim]. In this format, t indicates whether the object is a target, xc, y c represents the coordinates of the center point of the object, and area represents the size of the object. Then, sim is sorted in descending order according to the similarity value (sim), and the first 20 rows are selected. If there are less than 20 objects detected in a frame, the remaining rows are filled with zeros to ensure that the matrix size is consistent. The sim in this matrix combines the similarity measures of the target and the detected object. The embedded representations of the target and the object are denoted as t g and t o , mapped into a 300-dimensional word embedding vector through the word2vec algorithm, e g and e o ∈R 300 By calculating e g and e o The cosine similarity (CS) between them is calculated as follows:

[0025]

[0026] The step 4 is specifically as follows:

[0027] In the A3C algorithm, the agent first sets the current state s t Input the target search model, and the model generates the agent's action strategy based on the state. Specifically, the policy network outputs the probability distribution of an action π(a t |s t θ π ), from which the current action a is sampled t , while the value network estimates the state s t The value function V(s t θ v ). The agent interacts with the environment according to the selected action. After executing the action, the environment returns to the new state s t+1 and reward r t , which reflects the effectiveness of the agent’s behavior in the current state.

[0028] The step 5 is specifically as follows:

[0029] The agent calculates the advantage function A t =R t -V(s t θ v) is used to evaluate the effect of taking an action in the current state, where Rt is the cumulative reward starting from time t. The advantage function reflects the performance of the current strategy relative to the baseline value, which can help the agent learn more effectively during training. On this basis, the agent optimizes the parameters of the policy network and the value network through gradient updates. Specifically, the policy network is updated by maximizing the expected reward, thereby increasing the probability that the agent chooses the optimal action; while the value network adjusts its parameters by minimizing the mean square error between the predicted value and the actual cumulative reward. In order to speed up training and improve the generalization ability of the strategy, multiple agent copies can be executed in parallel, and by asynchronously updating the global network parameters, the delay problem caused by long-term training of a single agent is avoided, thereby improving training efficiency and model performance.

[0030] The beneficial effects of the present invention are as follows: the intelligent agent target search method based on time relationship context proposed in the present invention can efficiently perform target search in an unknown environment by combining the first-person perspective RGB image of the intelligent agent with the target text information, thereby significantly improving the efficiency of autonomous search. Using the reinforcement learning model of the A3C algorithm, the intelligent agent can optimize the search strategy in real time, enhancing its adaptability and generalization ability in a dynamic environment. By modeling the time relationship context, the intelligent agent can memorize the historical state and capture the timing dependency information, thereby improving the accuracy and reliability of target positioning. In addition, the present invention does not rely on prior environmental knowledge and does not require pre-constructed environmental maps, so that the intelligent agent can work autonomously in a completely new or changing environment. Through the adaptive target search model, the need for manual intervention is reduced and the level of automation of task execution is improved. In short, the present invention has significant advantages in improving target search efficiency, enhancing the adaptability of intelligent agent targets, and simplifying manual intervention. It has broad application prospects and is particularly suitable for robots, autonomous driving and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the abstract flow chart of the present invention

[0033] Figure 2 It is a model diagram of the intelligent agent target search based on the time relationship context of the present invention.

[0034] Figure 3 It is a flow chart of a specific implementation method of the present invention.

[0035] Figure 4 This is a table of search success rates of the present invention and different models. DETAILED DESCRIPTION

[0036] The specific flow chart is as follows Figure 3As shown. The autonomous target search method proposed in the present invention constructs an efficient model architecture by combining deep reinforcement learning with the specific requirements of the target search task. First, the agent combines the RGB image of the first perspective with the text information of the task target to infer the target position and execute the search strategy. Then, the model improves the decision-making ability and adaptability of the agent through three core modules: object semantic trajectory module, memory library and adaptive weight strategy module. The OST module extracts spatiotemporal features through memory and attention mechanism, and the AWP module responds to different targets and environments by dynamically adjusting the weights of the policy network. In the input stage, the state of the agent is represented as an object matrix containing object features, which reduces the fluctuation of visual information. Then, the agent generates a strategy through the A3C algorithm and interacts with the environment, obtains rewards based on the selected actions, and optimizes the parameters of the policy network and the value network by calculating the advantage function. The whole process uses parallel execution and asynchronous update strategies to accelerate training, improve the generalization ability of the strategy, and finally achieve efficient and robust target search capabilities. The specific implementation process of the technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0037] 1. Goal and Agent State Space Construction

[0038] In the autonomous target search task, the agent cannot directly obtain the complete state information of the global environment, but takes the RGB images captured by the first-person camera as input, which can only provide local environment information. At the same time, the text description information of the task target clearly specifies the target category that the agent needs to search. The agent infers the location of the target in the environment by combining the visual information obtained from the camera and the text information of the target, and executes the corresponding search strategy based on this information.

[0039] 2. Build a target search model

[0040] The model architecture proposed in this paper consists of two main modules: object semantic trajectory module (OST) and adaptive weight strategy module (AWP). The construction process of the model is as follows:

[0041] (1) Constructing the Object Semantic Trajectory Module (OST)

[0042] The object semantic trajectory module stores the historical semantic feature information of objects in the scene by building a memory bank. The memory bank is designed as a circular buffer with a length of T, which is used to store the spatiotemporal relationship and semantic information of the most recent T frames. The spatiotemporal features of each object are stored in the memory bank and extracted through the designed memory attention mechanism (Memory Attention). In the specific implementation, all features in the memory bank are converted into Query, Key, and Value through the projection matrix to calculate the similarity between objects. The similarity between the query and the key is calculated using the dot product, and then the Value is weighted by the weight generated by Softmax normalization to generate semantic features with global spatiotemporal information. These semantic features not only describe the object itself, but also reflect the spatiotemporal relationship of the object in the scene. These spatiotemporal features will be passed as input to the subsequent strategy module for the agent to use to generate a target search strategy.

[0043] (2) Constructing the Adaptive Weight Policy Module (AWP)

[0044] The core goal of the adaptive weight policy module is to dynamically adjust the weights of the policy network to adapt to the needs of different goals and scenarios. This module combines the Actor-Critic framework to generate adaptive policies through target embedding features and scene semantic information. In specific implementation, the target embedding features are mapped to the policy space through a linear layer and used as the input of the policy network. At the same time, a semantic weight bias matrix is ​​generated through the hypernet, which consists of two layers of fully connected layers and uses the ReLU activation function for nonlinear transformation. By projecting the target embedding features into the policy weight space, the adjusted policy weights are finally generated. The policy network weights of the agent are a combination of the initial weights and the semantic weight bias generated by the hypernet, allowing the agent to dynamically adjust the strategy according to the semantic features of the target to adapt to different navigation tasks and environments.

[0045] 3. State space preprocessing

[0046] In order to avoid the difference in visual information caused by environmental differences and object diversity, an object matrix is ​​designed as the representation of the agent state. This matrix only relies on the detection box value matrix and the cosine similarity measurement of the target embedding, and abandons the visual features and knowledge graphs in traditional methods. The state representation we designed is a 20-row 5-column matrix, where each row represents the features of an object, including:

[0047] t: whether the object is a target (target object tag)

[0048] xc, yc: coordinates of the center point of the object

[0049] area: the size of the object

[0050] sim: the similarity between the object and the target

[0051] These rows are sorted in descending order based on the similarity sim between the object and the target, and the top 20 most similar objects are selected. If fewer than 20 objects are detected in a frame, the remaining rows are filled with zeros to ensure that the dimensions of the matrix are consistent. The embedding representation of the target and the object is converted into a 300-dimensional word embedding vector through the word2vec algorithm, and the cosine similarity between the target and the object is calculated to help the agent accurately locate the target.

[0052] 4. Strategy Generation and Reward Calculation

[0053] In the A3C algorithm, the agent first inputs the current state into the target search model. Based on this state, the model generates the current action strategy. Specifically, the policy network outputs a probability distribution of an action based on the input state, from which the agent samples the current action, and the value network is used to estimate the value function of the state. The agent interacts with the environment based on the selected action. After executing the action, the environment returns a new state and reward. The reward value reflects the effect of the action taken by the agent in the current state.

[0054] 5. Strategy Update

[0055] The agent calculates the advantage function A t To evaluate the effect of choosing an action in the current state. The calculation formula of the advantage function is:

[0056] A t =R t -V(s t θ v )

[0057] Among them, R t represents the cumulative reward starting from time t, V(s t θ v ) is the value function used to estimate the state s t The advantage function reflects the performance of the current strategy relative to the benchmark value (i.e., the prediction of the value function), helping the agent to more effectively evaluate the benefits of each action during training. On this basis, the agent optimizes the parameters of the policy network and the value network through gradient updates. The policy network is updated by maximizing the expected reward, thereby increasing the probability that the agent chooses the optimal action; while the value network adjusts its parameters by minimizing the mean square error between the predicted value and the actual accumulated reward. In order to improve training efficiency and accelerate convergence, multiple agent replicas can perform tasks in parallel and update global network parameters asynchronously. This asynchronous update strategy can avoid the delay problem caused by long-term training of a single agent, thereby accelerating the training process of the model and improving the final policy performance.

[0058] Experimental setup:

[0059] (1) Dataset: This dataset uses the AI2-THOR environment and contains 120 scenes divided into four different types of rooms. Each room contains various objects for the agent to observe and interact with. These scenes are randomly divided into three different experimental partitions. In each room type, 80 rooms are allocated for training and 40 rooms are allocated for testing. In addition, the 22 target classes (common objects) in the dataset are reorganized into two partitions: the first partition: 18 seen objects and 4 unseen objects; the second partition: 14 seen objects and 8 unseen objects. This structure enables the study to simultaneously evaluate the performance of the agent in seen and unseen environments, thereby improving the generalization ability of the agent's learning and navigation capabilities.

[0060] (2) Experimental details:

[0061] We implemented our models in PyTorch. The models were trained extensively, including 2.2 million episodes, using offline data from the AI2-THOR environment. This training was performed on a 24 GB RTX 3090 GPU. We used the Adam optimizer with a learning rate of 0.0001. During evaluation, we executed approximately 250 episodes per scene, ultimately executing a total of 1000 episodes across all scenes.

[0062] (3) Experimental results:

[0063] We evaluated the effectiveness of the model. Figure 4 A comparative analysis with related methods is presented. Our method significantly outperforms the baseline methods in terms of SR and SPL metrics. Our model is compared with the CIRN model: in the 18 / 4 unknown setting, our CIRN achieves SR of 74.2%, while in the seen setting, the comparison algorithm CIRN improves by 8.0 percentage points. Overall, this shows that our method is more effective in generalizing navigation policies, especially in the zero-shot setting where the model encounters unknown objects and environments. There are significant improvements in SR (ranging from 4.0% to 9.0%) and SPL (ranging from 0.4% to 6.0%) compared to the CIRN method.

[0064] It should be noted that the above is only an embodiment of the present invention, which is only to explain the present invention and does not limit the patent scope of the present invention. Any obvious changes that belong to the technical concept of the present invention are also within the protection scope of the present invention.

Claims

1. The present invention mainly relates to an agent target search method based on time relationship context, comprising the following steps: (1) Construction of target and agent state space: First, according to the task requirements, determine the state space input and output of the agent in reinforcement learning. The target is specified as the name of the target to be searched, and the input of the agent is the RGB image obtained by the camera as a representation of the environment state; the output of the agent is the action generated by the model, which is used to guide the agent to perform corresponding behaviors in the environment. (2) Constructing a target search model: Construct a target search model based on the temporal relationship context, as shown in FIG2 . The model combines time series information, uses the agent's historical state and current environment information to achieve intelligent search and positioning of the target; (3) State space preprocessing: The agent interacts with the environment. By obtaining the agent's first-person perspective RGB image and preprocessing the image, the current state space representation of the environment is obtained. The state space representation reflects the relationship between the objects and the target in the current environment. (4) Strategy generation and reward calculation: The current state space is input into the target search model, and the model generates the agent's current search strategy (i.e., the selected action) based on the state. At the same time, the agent interacts with the environment based on the selected action and calculates the reward value; (5) Strategy update: The current strategy is evaluated through the Critic network in the A3C algorithm, and the advantage function is calculated based on the reward obtained by the agent and the value function of the Critic network. The advantage function is used to update the strategy of the Actor network.

2. The method for searching an intelligent target based on a temporal context as claimed in claim 1, characterized in that: According to step (1), in the autonomous target search task, the agent cannot directly obtain the global environment state and can only rely on the RGB images of the first-person camera as input, which provide local environment information. The text information of the specified target defines the specific requirements of the task and indicates the target category that the agent needs to search. The agent infers the target location and executes the corresponding search strategy by combining the visual input and the target text information.

3. The method for searching a target based on a time-relationship context according to claim 1, characterized in that: According to step (2), the overall architecture of the model includes two main sub-modules: object semantic trajectory module (OST) and adaptive weight strategy module (AWP). The construction process of the model is divided into the construction of two modules: (1) Constructing the Object Semantic Trajectory Module (OST) Initialize the memory bank: implement a circular buffer to store the historical semantic features of objects in the scene. The memory length is set to T to save the spatiotemporal relationship and semantic information of the objects. Design the memory attention mechanism: extract the historical features of all frames from the memory bank. Define the projection matrix W q , W k , W v , map the feature data into Query, Key, and Value. Use the dot product to calculate the similarity between the query and the key: The Softmax normalization weights are used to perform weighted calculations on the values ​​to generate global spatiotemporal features. The extracted spatiotemporal features are used as the core representation of the semantic trajectory and passed to the strategy module. (2) Constructing the Adaptive Weight Policy Module (AWP) The adaptive weight strategy module aims to adapt to the needs of different target navigation tasks by dynamically adjusting the weights of the policy network. This module is based on the Actor network method, combined with target embedding features and scene semantic information, and is implemented in two parts: first, the target embedding features are mapped to the policy space and directly integrated into the input of the policy network through a linear layer; second, a semantic weight bias matrix is ​​generated using a hypernet, which consists of two fully connected layers and uses a ReLU activation function to project the target embedding features into the policy weight space. The weights of the final policy network are obtained by combining the initial weights with the semantic weight bias, thereby dynamically adjusting the policy to adapt to different targets or scenarios. For unknown target categories, the module migrates the policies of known categories through an offset weight mechanism.

4. The method for searching an intelligent target based on a temporal context according to claim 1, characterized in that: In step (3) we define the object matrix as the state space s of the model input t : Different from other visual navigation methods, we abandon the use of visual features or knowledge graphs and only rely on the detection box value matrix and the cosine distance of the target embedding as input. We choose not to use visual data as state representation to reduce the variation of visual information caused by different environments and objects. Our state representation structure is a 20-row 5-column matrix, where each row is represented by C i =[t,x c ,y c , area, sim]. In this format, t indicates whether the object is a target, x c ,y c represents the coordinates of the center point of the object, and area represents the size of the object. Then, sim is sorted in descending order according to the similarity value (sim), and the first 20 rows are selected. If there are less than 20 objects detected in a frame, the remaining rows are filled with zeros to ensure that the matrix size is consistent. The sim in this matrix combines the similarity measures of the target and the detected object. The embedded representations of the target and the object are denoted as t g and t o , mapped into a 300-dimensional word embedding vector through the word2vec algorithm, e g and e o ∈R 300 By calculating e g and e o The cosine similarity (CS) between them is calculated as follows:

5. The method for searching an intelligent target based on a temporal context according to claim 1, characterized in that: In step (4), in the A3C algorithm, the agent first sets the current state s t Input the target search model, and the model generates the agent's action strategy based on the state. Specifically, the policy network outputs the probability distribution of an action π(a t |s t θ π ), from which the current action a is sampled t , while the value network estimates the state s t The value function V(s t θ v ). The agent interacts with the environment according to the selected action. After executing the action, the environment returns to a new state S t+1 and reward r t , which reflects the effectiveness of the agent’s behavior in the current state.

6. The method for searching an intelligent agent target based on a temporal context as claimed in claim 1, characterized in that: In step (5), the agent calculates the advantage function A t =R t -V(s t θ v ) is used to evaluate the effect of taking an action in the current state, where Rt is the cumulative reward starting from time t. The advantage function reflects the performance of the current strategy relative to the baseline value, which can help the agent learn more effectively during training. On this basis, the agent optimizes the parameters of the policy network and the value network through gradient updates. Specifically, the policy network is updated by maximizing the expected reward, thereby increasing the probability that the agent chooses the optimal action; while the value network adjusts its parameters by minimizing the mean square error between the predicted value and the actual cumulative reward. In order to speed up training and improve the generalization ability of the strategy, multiple agent copies can be executed in parallel, and by asynchronously updating the global network parameters, the delay problem caused by long-term training of a single agent is avoided, thereby improving training efficiency and model performance.

Images