A model training method, a path tracking method, a device, and an electronic device

Abstract

The application provides a model training method, a path tracking method, a device and an electronic device. The method comprises: obtaining trajectory slice information composed of a plurality of trajectory point information; obtaining an actor hidden state value and a critic hidden state value corresponding to each trajectory point information in the trajectory slice information; performing advantage estimation based on the trajectory slice information, the actor hidden state value and the critic hidden state value to obtain a Q value sequence; determining an actor network gradient and a critic network gradient based on the Q value sequence; updating parameters of an actor network based on the actor network gradient and updating parameters of a critic network based on the critic network gradient to obtain a path tracking model. The calculation of the Q value sequence for the actor network and the critic network in the model is performed by using the historical trajectory slice information and the hidden state information of the network, and the path tracking precision of the path tracking model obtained by training can be improved by performing the operation on the actor network and the critic network in the model through the Q value sequence.

Description

Technical Field

[0001] This application relates to the field of path tracking technology, and in particular to a model training method, path tracking method, apparatus, and electronic device. Background Technology

[0002] The path tracking problem is a classic control task. The goal of path tracking is to control a vehicle to follow a reference path without time constraints.

[0003] Many control methods have been developed in related technologies, such as backtracking, model predictive control, optimal control, and Lyapunov-based control. These methods are typically designed based on the relative position between the vehicle and a reference path (or a virtual point on the path), such as tracking error or relative distance. In practical control tasks, accurate position information requires complex state estimation systems or environmental instruments. Existing visual path tracking work employs a perception-control scenario for training. This two-stage approach estimates the explicit relative position from the original image through hand-designed feature detection or 3D reconstruction, and then trains the path tracking model in a non-visual context.

[0004] However, the performance of the path tracking model trained by the above method is highly dependent on the accuracy of environmental perception. The positioning error generated during the model usage phase will reduce the control precision, resulting in a large tracking error and low path tracking accuracy. Summary of the Invention

[0005] This application provides a model training method, a path tracking method, an apparatus, and an electronic device to solve the problem of poor accuracy when performing path tracking using a path tracking model in the prior art.

[0006] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0007] Acquire trajectory slice information composed of multiple trajectory point information; wherein, the trajectory point information includes first observation information, second observation information, action information and reward information, and the first observation information and the second observation information are obtained by the agent observing the environment at adjacent time points;

[0008] Obtain the actor hidden state value and commenter hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein, the actor hidden state value is generated by the actor network in the model to be trained, and the commenter hidden state value is generated by the commenter network in the model to be trained;

[0009] Based on the trajectory slice information, the actor's hidden state value, and the commentator's hidden state value, advantage estimation is performed to obtain the Q-value sequence;

[0010] The actor network gradient and the commentator network gradient are determined based on the Q-value sequence.

[0011] The parameters of the actor network are updated based on the gradient of the actor network, and the parameters of the commentator network are updated based on the gradient of the commentator network to obtain a path tracking model; wherein, the path tracking model is used for path tracking.

[0012] Secondly, embodiments of this application provide a path tracking method applied to an intelligent agent, wherein the intelligent agent is equipped with the path tracking model described in the first aspect, and the method includes:

[0013] Acquire observational information from the surrounding environment using a monocular camera;

[0014] The observation information is input into the path tracking model to obtain the action information output by the path tracking model;

[0015] The agent's movement is controlled based on the motion information to perform path tracking.

[0016] Thirdly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0017] The first acquisition module is used to acquire trajectory slice information composed of multiple trajectory point information; wherein, the trajectory point information includes first observation information, second observation information, action information and reward information, and the first observation information and the second observation information are obtained by the agent observing the environment at adjacent time points;

[0018] The second acquisition module is used to acquire the actor hidden state value and commenter hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein, the actor hidden state value is generated by the actor network in the model to be trained, and the commenter hidden state value is generated by the commenter network in the model to be trained;

[0019] The sequence module is used to perform advantage estimation based on the trajectory slice information, the actor's hidden state value, and the commentator's hidden state value to obtain a Q-value sequence;

[0020] The gradient module is used to determine the gradient of the actor network and the gradient of the commentator network based on the Q-value sequence;

[0021] The training module is used to update the parameters of the actor network based on the gradient of the actor network and the parameters of the commentator network based on the gradient of the commentator network to obtain a path tracking model; wherein, the path tracking model is used for path tracking.

[0022] Fourthly, embodiments of this application provide a path tracking device applied to an intelligent agent, the intelligent agent being equipped with the path tracking model as described in the first aspect, the device comprising:

[0023] The data acquisition module is used to acquire observational information from the surrounding environment using a monocular camera.

[0024] The prediction module is used to input the observation information into the path tracking model to obtain the action information output by the path tracking model;

[0025] An execution module is used to control the movement of the intelligent agent based on the action information to perform path tracking.

[0026] Fifthly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the model training method or path tracking method provided in this application.

[0027] Sixthly, embodiments of this application also provide a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the model training method or path tracking method provided in this application.

[0028] In this embodiment, the method includes: acquiring trajectory slice information composed of multiple trajectory point information; wherein the trajectory point information includes first observation information, second observation information, action information, and reward information, and the first and second observation information are obtained by the agent observing the environment at adjacent time points; acquiring the actor hidden state value and the commentator hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein the actor hidden state value is generated by the actor network in the model to be trained, and the commentator hidden state value is generated by the commentator network in the model to be trained; performing advantage estimation based on the trajectory slice information, the actor hidden state value, and the commentator hidden state value to obtain a Q-value sequence; determining the actor network gradient and the commentator network gradient based on the Q-value sequence; updating the parameters of the actor network based on the actor network gradient, updating the parameters of the commentator network based on the commentator network gradient, and obtaining a path tracking model; wherein the path tracking model is used for path tracking. It can combine historical trajectory slice information and hidden state information of the network to calculate the Q-value sequence of the actor network and commentator network in the model. By performing calculation on the actor network and commentator network in the model through the Q-value sequence, the path tracking accuracy of the trained path tracking model can be improved. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the steps of a model training method provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of a Markov decision process provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of trajectory slice information acquisition provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of a carrot tracking strategy provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of a nonlinear guiding law reward provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of a gaze reward provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of a training model structure provided in an embodiment of this application;

[0036] Figure 8 This is a flowchart illustrating the steps of a path tracing method provided in an embodiment of this application;

[0037] Figure 9 This is a block diagram of a model training device provided in an embodiment of this application;

[0038] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0040] In this embodiment, Deep Reinforcement Learning (DRL) is applied to the monocular visual path following (MVPF) problem, considering two issues. First, the weak supervision of reinforcement learning limits the complexity of the model and the depth of the neural network, thus narrowing the receptive field for feature extraction. Second, due to the lack of depth of field and limited viewpoint, monocular camera images can only partially observe the relative position between the vehicle and the path. A single frame cannot provide sufficient information for continuous tracking tasks. The controller needs memory capabilities to infer integrated positional information from multiple consecutive frames. To overcome these two problems, this embodiment proposes a recurrent DRL framework based on deterministic policy gradient theory.

[0041] Figure 1 This is a flowchart illustrating the steps of a model training method provided in an embodiment of this application. This method is applied to an electronic device, such as... Figure 1 As shown, the method may include:

[0042] Step 101: Obtain trajectory slice information composed of multiple trajectory point information; wherein, the trajectory point information includes first observation information, second observation information, action information, reward information, actor hidden state value and commenter hidden state value, and the first observation information and the second observation information are obtained by the agent observing the environment at adjacent time moments.

[0043] The goal of path tracing is to control a vehicle to travel along a reference path (such as lane lines, pipe sidewall lines, etc.) without time constraints. For example, the reference path can be represented as p. r =[x r (λ),y r (λ),z r (λ)] T The trajectory of the vehicle can be represented as p = [x(t), y(t), z(t)]. T The symbol λ is a dummy variable used to generate the path, following a time law λ(t). The path tracking controller needs to satisfy the following condition: the tracking error converges to zero, i.e. The second condition is to prevent the vehicle from stopping on the path, and λ(t) is defined as the distance between the vehicle and the nearest point on the reference path. In this embodiment, a Bezier curve can be used to represent the path. The Bezier curve is determined by the start point, the end point, and several control points. The control points control the curvature of the curve. The Bezier curve can be expressed as the following equation (1):

[0044] B(λ)=P0(1-λ) 3 +3P1λ(1-λ)2 +3P2λ 2 (1-λ)+P3λ 3 (1)

[0045] Where P0 and P3 represent the starting point and ending point, respectively, and P1 and P2 are control points. The dummy variable λ∈[0,1] corresponds one-to-one with the points on the curve.

[0046] Because the environmental information acquired in a monocular scene lacks depth and has a limited field of view, monocular vision can only observe partial information about the actual location. This incomplete information prevents the visual signal conversion from satisfying the Markov property. Therefore, in this embodiment, the monocular vision path tracking (MVPF) problem is modeled as a partially observable Markov decision process (POMDP).

[0047] In this embodiment, trajectory slice information composed of multiple trajectory point information can be obtained; wherein, the trajectory point information includes first observation information, second observation information, action information, and reward information, and the first observation information and the second observation information are obtained by an intelligent agent (e.g., a vehicle, a ship, etc.) observing the environment at adjacent time points. The first observation information at time t can be referred to as o. t Action information can be called a t The second observation information can be called o t +1.

[0048] The aforementioned intelligent agent can be an agent proxy, which can simulate the behavior of the intelligent agent during model training so as to train the model in a virtual scene.

[0049] Reference Figure 2 , Figure 2 This illustration shows a schematic diagram of a Markov decision process provided in an embodiment of this application, such as... Figure 2 As shown, at a certain time step t, the agent obtains the real state s from the environment. t First observation information o t Agent execution action information a t Then transition to the subsequent real state s t +1, simultaneously obtain reward information rt and second observation information o t +1.

[0050] In this embodiment of the application, a monocular image can be regarded as observation information. t POMDP states t This represents the relative position where a vehicle cannot be accessed along a path. This can be achieved by defining the observation history h. t And true history H tTo formulate strategies and optimize the problem. t h represents the sequence of observation and action information prior to time t. t = <o0,a0,o1,a1,…,a t-1 ,o t >. H t It is the real history, which adds the real state sequence to the observation history, and can be represented as H. t = <s0,o0,a0,s1,…,s t ,o t Then, the embodiments of this application are based on the observation history h. t Define policy distribution a t ~π(·∣h t Through real history H t Given policy π, MVPF can be formulated as an optimal control problem, maximizing the expected discount total reward as shown in equation (2):

[0051]

[0052] Where Ω represents the observation space and T represents the time limit of the partially observable Markov decision process (POMDP). Based on equation (2), the value function (3) and the action-value function (4) can be further defined:

[0053]

[0054]

[0055] Unlike fully observable Markov decision processes (MDPs), the value function of a partially observable Markov decision process (POMDP) ​​is a time-varying function and is determined by the simulated trajectory.

[0056] The optimal control problem represented by equation (2) above can be solved by the policy gradient method. Policy gradient theory assumes a parameterized policy function π. θ And update the parameter θ along the gradient ascent direction of J using equation (5):

[0057]

[0058] In this application embodiment, a recurrent deep deterministic policy gradient (RDPG) for solving the MVPF POMDP is proposed to estimate the above.

[0059] The Recurrent Deep Deterministic Policy Gradient (RDPG) algorithm addresses the partial observability of the Proof-of-Concept (POMDP) ​​problem by introducing recurrent neural networks (RNNs) into the policy and value networks. RNNs can capture long-term dependencies in time series, thus improving performance in partially observable environments. By applying RDPG to the MVPF POMDP problem, the algorithm can learn an effective control policy, enabling stable control of agents operating in unstable environments such as marine environments.

[0060] Furthermore, the RDPG algorithm can be combined with other control strategies. For example, RDPG can be combined with model predictive control (MPC) to achieve more accurate and stable control of the MVPF. Additionally, during training, environmental noise and disturbances can be modeled to account for uncertainties and environmental changes, thereby improving the algorithm's robustness.

[0061] In this embodiment, training can be performed using an experience replay strategy adapted to both the actor network and the commentator network. The replay buffer stores complete trajectory point information during the agent's interaction with the environment. When training by sampling, a batch of trajectory point information can be extracted from random positions on the randomly selected trajectory in the replay buffer to obtain trajectory slices. The purpose of truncating the trajectory is to avoid excessively long sequences, thereby alleviating the gradient vanishing problem in RNN training. For each trajectory slice, this embodiment randomizes its position and the index of the source trajectory, indirectly expanding the sample pool.

[0062] Reference Figure 3 , Figure 3 This illustration shows a schematic diagram of trajectory slice information acquisition provided in an embodiment of this application, such as... Figure 3 As shown, Trajectory slices can be obtained by extracting and combining a continuous segment of trajectory point information from various information in the playback buffer.

[0063] To train a model based on the RDPG algorithm, a replay buffer can be used to store policy-related trajectory point information. During each iteration, a batch of trajectory points is randomly selected from the replay buffer to form trajectory slices. These trajectory slices are then used to update the network, thus avoiding correlations between sequence data during gradient updates and improving the stability and convergence speed of the training process.

[0064] Optionally, trajectory point information can be generated through the following sub-steps 1011 to 1014:

[0065] Sub-step 1011: Obtain the state information of the agent at the first moment, and the first observation information collected by the agent at the first moment.

[0066] Sub-step 1012: Calculate the reward information based on the state information.

[0067] In reinforcement learning, rewards serve as feedback signals to aid the learning strategy. It's important to note that reward information is only used during model training; the actor network does not need to consider rewards during the process of generating action information.

[0068] Optionally, in this embodiment, three reward functions can be designed using real location information: carrot chasing reward, nonlinear guidance law (NLGL)-based reward, and line of sight (LOS) reward. The specific process of carrot chasing reward is described below through sub-steps A1 to A6.

[0069] Sub-step A1: Determine the closest point on the reference path that is closest to the agent.

[0070] State information can include the agent's position coordinates P(x,y). T ψ is the heading direction, u is the pitch speed, v is the yaw speed, and r is the yaw speed.

[0071] The carrot-and-stick reward model assumes a uniformly moving virtual point on the reference path. The agent can converge to the path by tracking the virtual point. Figure 4 , Figure 4 This illustration shows a carrot tracking strategy provided in an embodiment of this application. Figure 4 As shown, assume the vehicle is located at point P with coordinates (x, y). T The heading angle is ψ. The point on the path closest to the vehicle is denoted as A, with coordinates (x(λ), y(λ)). T , where λ is a dummy Bessel variable. The carrot-tracking strategy defines the dummy target point as one of two adjacent points B and C obtained by adding a small constant step size ±δ to λ.

[0072] Sub-step A2: Determine the dummy variable corresponding to the nearest point based on the Newton-Raphson algorithm.

[0073] In this embodiment, the dummy variable corresponding to the nearest point can be calculated using the following equation (6):

[0074]

[0075] Sub-step A3: Determine the first candidate coordinates and the second candidate coordinates based on the virtual variables.

[0076] In the embodiments of this application, the coordinates of the first candidate point B can be represented as B(λ-δ), and the coordinates of the second candidate point C can be represented as B(λ+δ).

[0077] Sub-step A4: Determine the target virtual point from the first candidate coordinates and the second candidate coordinates based on the position coordinates and the heading direction.

[0078] In this embodiment of the application, the heading direction can be determined. By comparison and The smaller of the two candidate points is selected as the target virtual point.

[0079] Sub-step A5: Obtain the vector direction of the vector formed by the target virtual point and the position coordinates.

[0080] In the embodiments of this application, such as Figure 4 As shown, assuming the target virtual point is the first candidate point B mentioned above, the target virtual point and its position coordinates can form a vector. The direction of this vector can be defined as ψ. d .

[0081] Sub-step A6: Calculate the reward information based on the heading direction, the vector direction, the pitch speed, the yaw speed, and the yaw speed.

[0082] In this embodiment of the application, the reward information r can be calculated using the following equation (7). CC Implement the carrot-tracking reward strategy:

[0083] r CC =ucos(ψ-ψ d )-ρ1v 2 -ρ2r 2 (7)

[0084] Among them, ρ1 and ρ2 are constant parameters that can be flexibly set according to actual needs. The specific values ​​of ρ1 and ρ2 are not limited in the embodiments of this application.

[0085] Optionally, the specific process of nonlinear guided law reward is explained below through sub-steps B1 to B4.

[0086] Sub-step B1: Determine the shortest distance between the agent and the reference path based on the Newton-Raphson algorithm.

[0087] Reference Figure 5 , Figure 5Shows a schematic diagram of the reward of a non - linear guidance law provided by an embodiment of the present application, as Figure 5 shown, based on the position coordinates of the agent (e.g., P(x, y) T ) and the expression of the reference path (e.g., Bezier curve B(λ)=[x r (λ), y r (λ)] T ), and determine the closest distance l by solving the following Newton - Raphson equation (8):

[0088]

[0089] Sub - step B2, determine the first candidate coordinate and the second candidate coordinate according to the position coordinate and the preset radius.

[0090] As Figure 5 shown, when the above - mentioned closest distance l≥c (preset radius), the first candidate coordinate and the second candidate coordinate can be determined according to the position coordinate and the preset radius. Specifically, λ1 and λ2 can be determined by solving the following equation (9), and then the first candidate coordinate B is defined as B(λ1), and the second candidate coordinate C is defined as B(λ2).

[0091] f(λ)=(x r (λ)-x) 2 +(y r (λ)-y) 2 -c 2 = 0 (9)

[0092] Sub - step B3, determine the target virtual point from the first candidate coordinate and the second candidate coordinate based on the position coordinate and the heading direction.

[0093] In the embodiment of the present application, the heading direction can be determined by comparing <00003​​​​​​​​​​​​​​​​.

[0097] Sub-step B5: Calculate the reward information based on the heading direction, the vector direction, the pitch speed, the yaw speed, and the yaw speed.

[0098] In this embodiment of the application, the reward information r can be calculated using the following equation (10). NLGL To implement a nonlinear guided law reward strategy:

[0099] r NLGL =ucos(ψ-ψ d )-ρ1v 2 -ρ2r 2 (10)

[0100] Among them, ρ1 and ρ2 are constant parameters that can be flexibly set according to actual needs. The specific values ​​of ρ1 and ρ2 are not limited in the embodiments of this application.

[0101] The main advantage of the NLGL strategy is that it can adaptively allocate control force by adjusting the virtual point based on the distance *l* between the agent and the reference path. When *l* is large, more control force will be allocated to make the agent move closer to the reference path; otherwise, more control force will drive the agent to travel along the reference path. If *l* is greater than a preset radius *c*, the nearest point A will become the virtual point, and the agent will move perpendicularly towards the reference path.

[0102] Optionally, the specific process of visual reward is explained below through sub-steps C1 to C2.

[0103] Sub-step C1: Determine the line-of-sight point closest to the agent on the reference path, and determine the line-of-sight direction based on the line-of-sight point and the position coordinates.

[0104] Reference Figure 6 , Figure 6 This illustration shows a visual reward diagram provided in an embodiment of this application, such as... Figure 6 As shown, a virtual point B is determined along the tangent direction of the nearest point A. The constant distance Δ between the two points is called the look-ahead distance, and the virtual point B is called the line-of-sight (LOS) point. The vector... This is called the line-of-sight (LOS) direction.

[0105] Sub-step C2: Calculate the reward information based on the heading direction, the line-of-sight direction, the pitch speed, the yaw speed, and the yaw speed.

[0106] In this embodiment of the application, the gaze reward r can be calculated using the following equation (11). NLGL Implement a gaze reward strategy:

[0107] r LOS=u·cos(ψ-ψ d )-ρ0d 2 -ρ1v 2 -ρ2r 2 (11)

[0108] Among them, ψ d It is the tangent angle at the nearest point A, and d represents the lateral tracking error. Coefficients ρ0 to ρ2 are used to balance several terms representing different targets. The first term is u·cos(ψ-ψ). d The projection of the pitch velocity u onto the path tangent can be considered as the unit displacement of the agent along the reference path at each time step. The second term ρ0d 2 This is a penalty for lateral tracking error. The third term ρ1v 2 and the fourth term ρ2r 2 These are constraints on yaw rate and lateral speed, respectively. The coefficients ρ0 to ρ2 are constant parameters and can be flexibly set according to actual needs. This application embodiment does not limit the specific values ​​of the coefficients ρ0 to ρ2.

[0109] Sub-step 1013: Input the first observation information collected by the intelligent agent into the actor network to obtain the action information output by the actor network.

[0110] In this embodiment of the application, the actor network can predict the motion actions that the agent needs to take based on the observation information and output the corresponding action information.

[0111] Sub-step 1014: Obtain the second observation information collected by the agent at the second time point to obtain a trajectory point information composed of the state information, the first observation information, the reward information and the second observation information; wherein, the second time point is the time after the agent moves based on the motion information.

[0112] Step 102: Obtain the actor hidden state value and commenter hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein, the actor hidden state value is generated by the actor network in the model to be trained, and the commenter hidden state value is generated by the commenter network in the model to be trained.

[0113] In this embodiment, a recurrent neural network (RNN) is introduced to represent the action-value and policy functions. An RNN is a network that processes variable-length sequences. The RNN introduces an internal hidden state to convey temporal information, as shown in equation (12) below:

[0114] y t ,χ t+1 =f θ (x t ,χ t (12)

[0115] By expanding equation (12) over time and ignoring the output hidden state, we can obtain equation (13):

[0116] y t ,χ t+1 =f θ (x t ,f θ (x t-1 ,χ t-1 ))=…=F θ (x t ,x t-1 ,…,x0,χ0) (13)

[0117] Therefore, an RNN can be viewed as a sequence (x) t ,x t-1 The function F of (x0, χ0) θ It can solve the problem of variable-length inputs for action-value and policy functions in POMDP.

[0118] In this embodiment of the application, critic networks and actor networks can be constructed based on the above action-value and policy functions:

[0119]

[0120]

[0121] Where, χ Q It is the commenter's hidden state value output by the commenter network, and χ μ This is the hidden state value of the actor output by the actor network. Due to variables... The variable encodes a sequence of observation and action information prior to time t, therefore the variable It can be the real state s t The internal representation of .

[0122] Step 103: Based on the trajectory slice information, the actor's hidden state value, and the commentator's hidden state value, perform advantage estimation to obtain the Q-value sequence.

[0123] Calculate the Q function There are several methods, specifically, the temporal difference (TD) method shown in equation (16):

[0124]

[0125] The computational bias can also be reduced by combining the rewards of more steps using equation (17):

[0126]

[0127] Using the above method, the Q-value corresponding to each trajectory point in the trajectory slice information set can be calculated, thus obtaining a Q-value sequence formed by these Q-values. It should be noted that those skilled in the art can use other methods to calculate the Q-values, and this application embodiment does not specifically limit the methods used.

[0128] Sub-step 1031: Reconstruct the hidden state value of the actor to obtain the reconstructed hidden state value of the actor; and reconstruct the hidden state value of the commenter based on the reconstructed hidden state value of the actor to obtain the reconstructed hidden state value of the commenter.

[0129] The calculation method provided by equation (16) above can be called TD(1), and the calculation method provided by equation (17) above can be called TD(n). Since the second term in TD(1) is the estimated subsequent Q value, It will be affected by high bias. Since there is noise in each reward information, the accumulation of reward information in TD(n) will bring cumulative noise, resulting in high variance.

[0130] In this application embodiment, a recurrent generalized advantage estimation (RGAE) method for trajectory slice training is proposed to calculate the Q-value. The RGAE method can include forward and backward stages. The forward stage infers the recurrent network along the trajectory slice and obtains the prediction of the Q-function. The backward stage starts from the tail of the trajectory slice, calculates and accumulates the TD error with a scaling factor γλ to obtain the advantage function and the Q-value.

[0131] Specifically, the cyclic generalized dominance estimation method requires trajectory slicing. <o t ,a r ,r t ,…,o t+K ,a t+K ,r t+K Initial hidden state and Commentator Network Q ω And the actor network μ θ Hyperparameters λ and γ. The following operations (forward phase) can be performed for i = 1 to K+1:

[0132] The hidden state value of the actor is reconstructed based on the following equation (18), resulting in the reconstructed hidden state value of the actor.

[0133]

[0134] The hidden state value of the commenter is reconstructed based on the following equation (19), resulting in the reconstructed hidden state value of the commenter.

[0135]

[0136] Sub-step 1032: Calculate the time difference error value based on the reconstructed commenter hidden state value and the reward information in the slice information subset.

[0137] In this embodiment of the application, it can be set And perform the following operations (backward phase) for i = K to 1:

[0138] The time difference error value (TD) δ is calculated using the following equation (20). t+i-1 :

[0139]

[0140] Sub-step 1033: Determine the Q-value sequence based on the time difference error value.

[0141] In this embodiment of the application, after obtaining the time difference error value, the dominance function value can be estimated based on the time difference error value to obtain the dominance function value, and the Q value can be calculated based on the dominance function value.

[0142] Specifically, the dominant function value can be calculated using the following equation (21).

[0143]

[0144] The Q value is calculated using the following equation (22).

[0145]

[0146] In the embodiments of this application, it can be As a sequence of Q values.

[0147] Step 104: Determine the actor network gradient and the commentator network gradient based on the Q-value sequence.

[0148] Through a recurrent commentator-actor network, embodiments of this application propose a recurrent DPG form (23) to approximate equation (24):

[0149]

[0150]

[0151] Among them, the actual action-value function Q μ By the loop commentator network Q ω Estimate. Equation (24) is a traditional DPG form, expressed as along the time axis. The summation. The parameter ω is optimized to make Q... ω Minimize the mean square error between the true Q value and the actual Q value.

[0152] Finally, the embodiments of this application provide the basic form of the network gradient of the recurrent commentator and actor, thus yielding equations (25) and (26):

[0153]

[0154]

[0155] Assume N trajectories are sampled. in Then we can obtain the simplified equations (27) and (28):

[0156]

[0157]

[0158] in, Approximately equal to 1 / N multiplied by the sum of N trajectories, where each trajectory is weighted at each time step using a discount factor γ, and the gradient depends on Q. ω and μ θ Estimate. Temporary difference loss. The gradient is approximately equal to 1 / N multiplied by the sum of the N trajectories. Q is calculated for each trajectory at each time step. ω and The gradient of the square of the difference.

[0159] Step 105: Update the parameters of the actor network based on the gradient of the actor network, and update the parameters of the commentator network based on the gradient of the commentator network to obtain the path tracking model; wherein, the path tracking model is used for path tracking.

[0160] The overall scheme of this application can be summarized as follows: each iteration of the algorithm includes a sampling and training phase. Sampling phase (steps 3-10): Forward inference of the following commentator and actor networks, control of agent interaction with the environment, and then collection and storage of trajectory information. The actions performed include prediction and exploration of noise from the actor network, which comes from a Gaussian distribution. Training phase (steps 11-23): Sampling of a batch of trajectory slice information of length K, and then using the retracing strategy proposed in equations (18) and (19) to reconstruct the hidden state for each trajectory slice. The Q-value of each trajectory point in each trajectory slice is calculated by the RGAE method proposed in step 103 above. It should be noted that the retracing strategy consumes the first k elements of a trajectory slice; therefore, gradient accumulation starts from index t+k. The process of determining the gradient and updating the network parameters based on the gradient can be summarized as the following pseudocode steps:

[0161] 1: Initialize the commentator network Qω, the actor network μθ, and the replay buffer R.

[0162] 2: Enter the loop and perform the following operations for iter = 1 to M:

[0163] 3: Initialize the observation information o0, the commenter's hidden state χQ0, and the actor's hidden state χμ0.

[0164] 4: Enter the loop and perform the following operations for t=0 to T:

[0165] 5: The actor network outputs action information based on the first observation and adds exploration noise:

[0166]

[0167] 6: The commentator network outputs action information based on the output of the actor network:

[0168]

[0169] 7: Execution action information a t Obtain second observation information. t+1 and reward r t Collect trajectory point information

[0170] 8: End the loop.

[0171] 9: Transform the entire trajectory {e0,e1,…,e T Push it into the playback buffer R.

[0172] 10: Perform the following operations for c = 0 to C:

[0173] 11: Extract N trajectory slice information from R.

[0174] 12: The hidden state is reconstructed from the trajectory point information in each trajectory slice information by equations (18) and (19).

[0175] 13: Calculate the Q-value sequence in the manner shown in step 103.

[0176] 14: Initialize the commenter network gradient Δω = 0 and the actor network gradient Δθ = 0.

[0177] 15: Enter the loop and perform the following operations for i = 1 to N:

[0178] 16: Enter the loop and perform the following operations for j = t + k to t + k - 2:

[0179] 17: Calculate the commenter network gradient Δω:

[0180]

[0181]

[0182] 18: Calculate the gradient Δθ of the actor network:

[0183]

[0184]

[0185] 19: End the loop

[0186] 20: End the loop

[0187] 21: Update the parameters ω of the actor network θ and the commentator network:

[0188]

[0189]

[0190] 22: End the loop

[0191] 23: End the loop

[0192] Optionally, in the embodiments of this application, the actor network and the commentator network can be composed of a multilayer perceptron, a long short-term memory network, or a spatially sensitive convolutional neural network.

[0193] Reference Figure 7 , Figure 7 This application provides a schematic diagram of a trainable model structure, as shown in the embodiment of the present application. Figure 7As shown, the model can include an actor network and a commentator network, both of which can be constructed from recurrent networks, fully connected networks, and spatially sensitive convolutional neural networks. The input to the actor network can include not only observation information (e.g., visual images) but also physical vector information such as the agent's velocity. The input to the commentator network can include observation information (a...). t The agent's speed information v t In addition to physical vector information, the information can also include the agent's action information. In the commentator network and actor network described above, visual images can be processed into visual feature maps, which are then extracted and flattened into one-dimensional feature vectors using a Space-sensitive Convolutional Neural Network (CNN). These vectors are then concatenated with the physical features extracted by a Multilayer Perceptron (MLP) to obtain merged features. These merged features can be passed to a Long Short-Term Memory (LSTM) network. Finally, the memory encoding is processed by the output layer: the commentator network outputs the Q-value, and the actor network outputs the action information. It should be noted that the actor network can use the Tanh function to constrain the output action information under the constraints of the agent's action actuator, while the commentator network can use the Linear function to constrain the output value. The Long Short-Term Memory network is a recurrent network that can utilize logic gates to filter memory information in a time series.

[0194] It should be noted that after the model training is completed, the commentator network can be removed from the model, while the trained actor network is retained. The agent can acquire observation information from the surrounding environment through a monocular camera; input the observation information into the path tracking model to obtain the action information output by the path tracking model; and control the agent's actuator actions based on the action information to achieve the purpose of accurate path tracking.

[0195] In summary, the model training method provided in this application includes: acquiring trajectory slice information composed of multiple trajectory point information; wherein the trajectory point information includes first observation information, second observation information, action information, and reward information, and the first and second observation information are obtained by the agent observing the environment at adjacent time points; acquiring the actor hidden state value and the commentator hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein the actor hidden state value is generated by the actor network in the model to be trained, and the commentator hidden state value is generated by the commentator network in the model to be trained; performing advantage estimation based on the trajectory slice information, the actor hidden state value, and the commentator hidden state value to obtain a Q-value sequence; determining the actor network gradient and the commentator network gradient based on the Q-value sequence; updating the parameters of the actor network based on the actor network gradient, updating the parameters of the commentator network based on the commentator network gradient, and obtaining a path tracking model; wherein the path tracking model is used for path tracking. It can combine historical trajectory slice information and hidden state information of the network to calculate the Q-value sequence of the actor network and commentator network in the model. By performing calculation on the actor network and commentator network in the model through the Q-value sequence, the path tracking accuracy of the trained path tracking model can be improved.

[0196] In this embodiment of the application, the path tracking model trained in steps 101 to 105 can be mounted on the agent so that the agent can perform path tracking based on the path tracking model.

[0197] Figure 8 This is a flowchart illustrating the steps of a path tracking method provided in an embodiment of this application. This method is applied to an intelligent agent, such as... Figure 8 As shown, the method may include:

[0198] Step 201: Obtain observation information from the surrounding environment using a monocular camera.

[0199] Step 202: Input the observation information into the path tracking model to obtain the action information output by the path tracking model.

[0200] Step 203: Control the movement of the intelligent agent based on the action information to perform path tracking.

[0201] In summary, the path tracking method provided in this application can combine historical trajectory slice information and hidden state information of the network to calculate the Q-value sequence of the actor network and commentator network in the model. Calculating the Q-value sequence for the actor network and commentator network in the model can improve the path tracking accuracy of the trained path tracking model. Mounting the path tracking model on an agent for path tracking can improve the accuracy of path tracking performed by the agent.

[0202] Figure 9 This is a block diagram of a model training device provided in an embodiment of this application, such as... Figure 9 As shown, the model training device includes:

[0203] The first acquisition module 901 is used to acquire trajectory slice information composed of multiple trajectory point information; wherein, the trajectory point information includes first observation information, second observation information, action information and reward information, and the first observation information and the second observation information are obtained by the agent observing the environment at adjacent time points;

[0204] The second acquisition module 902 is used to acquire the actor hidden state value and the commenter hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein, the actor hidden state value is generated by the actor network in the model to be trained, and the commenter hidden state value is generated by the commenter network in the model to be trained;

[0205] Sequence module 903 is used to perform advantage estimation based on the trajectory slice information, the actor's hidden state value and the commentator's hidden state value to obtain a Q-value sequence;

[0206] Gradient module 904 is used to determine the actor network gradient and the commentator network gradient based on the Q-value sequence;

[0207] Training module 905 is used to update the parameters of the actor network based on the gradient of the actor network and the parameters of the commentator network based on the gradient of the commentator network to obtain a path tracking model; wherein, the path tracking model is used for path tracking.

[0208] Optionally, the device further includes:

[0209] The first information acquisition module is used to acquire the state information of the agent at a first moment, and the first observation information collected by the agent at the first moment.

[0210] The reward information module is used to calculate the reward information based on the status information;

[0211] The action information module is used to input the first observation information collected by the intelligent agent into the actor network to obtain the action information output by the actor network;

[0212] The second information acquisition module is used to acquire the second observation information collected by the agent at the second moment, and obtain a trajectory point information composed of the state information, the first observation information, the reward information and the second observation information; wherein, the second moment is the moment after the agent moves based on the motion information.

[0213] Optionally, the state information includes the agent's position coordinates, heading direction, pitch speed, yaw speed, and yaw rate; the reward information module includes:

[0214] The nearest point module is used to determine the nearest point on the reference path that is closest to the agent;

[0215] The dummy variable submodule is used to determine the dummy variable corresponding to the nearest point based on the Newton-Raphson algorithm;

[0216] The first candidate coordinate submodule is used to determine the first candidate coordinate and the second candidate coordinate based on the virtual variable;

[0217] The first virtual point submodule is used to determine a target virtual point from the first candidate coordinates and the second candidate coordinates based on the position coordinates and the heading direction;

[0218] The first vector direction submodule is used to obtain the vector direction of the vector formed by the target virtual point and the position coordinates;

[0219] The first reward submodule is used to calculate the reward information based on the heading direction, the vector direction, the pitch speed, the yaw speed, and the yaw speed.

[0220] Optionally, the state information includes the agent's position coordinates, heading direction, pitch speed, yaw speed, and yaw rate; the reward information module includes:

[0221] The nearest distance submodule is used to determine the nearest distance between the agent and the reference path based on the Newton-Raphson algorithm;

[0222] The second candidate coordinate submodule is used to determine the first candidate coordinate and the second candidate coordinate based on the position coordinate and the preset radius;

[0223] The second virtual point submodule is used to determine a target virtual point from the first candidate coordinates and the second candidate coordinates based on the position coordinates and the heading direction;

[0224] The second vector direction submodule is used to obtain the vector direction of the vector formed by the target virtual point and the position coordinates;

[0225] The second reward submodule is used to calculate the reward information based on the heading direction, the vector direction, the pitch speed, the yaw speed, and the yaw speed.

[0226] Optionally, the state information includes the agent's position coordinates, heading direction, pitch speed, yaw speed, and yaw rate; the reward information module includes:

[0227] The gaze direction submodule is used to determine the gaze point closest to the agent on the reference path, and to determine the gaze direction based on the gaze point and the position coordinates.

[0228] The third reward submodule is used to calculate the reward information based on the heading direction, the line of sight direction, the pitch speed, the yaw speed, and the yaw speed.

[0229] Optionally, the sequence module includes:

[0230] The reconstruction submodule is used to reconstruct the hidden state value of the actor to obtain the reconstructed hidden state value of the actor, and to reconstruct the hidden state value of the commenter based on the reconstructed hidden state value of the actor to obtain the reconstructed hidden state value of the commenter.

[0231] The error value submodule is used to calculate the time difference error value based on the reconstructed commenter hidden state value and the reward information in the slice information subset;

[0232] The Q-value sequence submodule is used to determine the Q-value sequence based on the time difference error value.

[0233] In summary, the apparatus provided in this application includes: a first acquisition module for acquiring trajectory slice information composed of multiple trajectory point information; wherein the trajectory point information includes first observation information, second observation information, action information, and reward information, and the first and second observation information are obtained by an agent observing the environment at adjacent time points; a second acquisition module for acquiring the actor hidden state value and the commentator hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein the actor hidden state value is generated by the actor network in the model to be trained, and the commentator hidden state value is generated by the commentator network in the model to be trained; a sequence module for performing advantage estimation based on the trajectory slice information, the actor hidden state value, and the commentator hidden state value to obtain a Q-value sequence; a gradient module for determining the actor network gradient and the commentator network gradient based on the Q-value sequence; and a training module for updating the parameters of the actor network based on the actor network gradient and updating the parameters of the commentator network based on the commentator network gradient to obtain a path tracking model; wherein the path tracking model is used for path tracking. This method can combine historical trajectory slice information and hidden state information of the network to calculate the Q-value sequence of the actor network and commentator network in the model. Using the Q-value sequence to operate on the actor network and commentator network can improve the path tracking accuracy of the trained path tracking model. Loading the path tracking model onto an agent for path tracking can improve the agent's path tracking accuracy.

[0234] This application also provides an electronic device, as described in the embodiments. Figure 10 , Figure 10 A schematic diagram of the hardware structure of an electronic device provided in this application is shown.

[0235] The electronic device 500 includes, but is not limited to, components such as: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510, and a power supply 511. Those skilled in the art will understand that... Figure 10 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In the embodiments of this application, the electronic device includes, but is not limited to, mobile phones, tablets, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.

[0236] Processor 510 is used to perform the following procedures:

[0237] Acquire a first sound spectrum of the external environment; determine the external scene category corresponding to the first sound spectrum; determine a first gain parameter of the audio output signal based on the external scene category; compensate the audio output signal based on the first gain parameter, and play the compensated audio output signal to reduce the masking effect of the environmental noise of the external environment on the audio output signal.

[0238] In this embodiment, a first sound spectrum of the external environment is acquired; an external scene category corresponding to the first sound spectrum is determined; a first gain parameter of the audio output signal is determined based on the external scene category; the audio output signal is compensated based on the first gain parameter, and the compensated audio output signal is played to reduce the masking effect of environmental noise on the audio output signal. This application can identify the user's current scene category based on the ambient sound and determine the first gain parameter based on the scene category. By compensating the sound played by the speaker using the first gain parameter, the impact of environmental noise on sound quality can be reduced, improving the user's listening experience.

[0239] It should be understood that, in this embodiment, the radio frequency unit 501 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 510; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 501 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 501 can also communicate with networks and other devices through a wireless communication system.

[0240] The electronic device provides users with wireless broadband internet access through the network module 502, such as helping users send and receive emails, browse web pages, and access streaming media.

[0241] The audio output unit 503 can convert audio data received by the radio frequency unit 501 or the network module 502 or stored in the memory 509 into audio signals and output them as sound. Furthermore, the audio output unit 503 can also provide audio output related to specific functions performed by the electronic device 500 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 503 includes a speaker, a buzzer, and a receiver, etc.

[0242] Input unit 504 is used to receive audio or video signals. Input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on display unit 506. The image frames processed by GPU 5041 can be stored in memory 509 (or other storage medium) or transmitted via radio frequency unit 501 or network module 502. Microphone 5042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 501 in telephone call mode.

[0243] The electronic device 500 also includes at least one sensor 505, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 5061 according to the ambient light level, and the proximity sensor can turn off the display panel 5061 and / or backlight when the electronic device 500 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 505 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.

[0244] The display unit 506 is used to display information input by the user or information provided to the user. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0245] User input unit 507 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 507 includes a touch panel 5071 and other input devices 5072. Touch panel 5071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 5071). Touch panel 5071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 510, which receives and executes commands from the processor 510. In addition, touch panel 5071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides touch panel 5071, user input unit 507 may also include other input devices 5072. Specifically, other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0246] Furthermore, the touch panel 5071 can cover the display panel 5061. When the touch panel 5071 detects a touch operation on or near it, it transmits the information to the processor 510 to determine the type of touch event. Subsequently, the processor 510 provides corresponding visual output on the display panel 5061 based on the type of touch event. Although in Figure 10 In this embodiment, the touch panel 5071 and the display panel 5061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 5071 and the display panel 5061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.

[0247] Interface unit 508 serves as an interface for connecting external devices to electronic device 500. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 508 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 500, or it can be used to transmit data between electronic device 500 and external devices.

[0248] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 509 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0249] The processor 510 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 509, and by calling data stored in the memory 509, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 510 may include one or more processing units; preferably, the processor 510 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 510.

[0250] The electronic device 500 may also include a power supply 511 (such as a battery) for supplying power to various components. Preferably, the power supply 511 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0251] In addition, the electronic device 500 includes some functional modules not shown, which will not be described in detail here.

[0252] Preferably, this application embodiment also provides a mobile terminal, including a processor 510, a memory 509, and a computer program stored in the memory 509 and executable on the processor 510. When the computer program is executed by the processor 510, it implements the various processes of the above-described model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0253] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described model training method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0254] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0255] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0256] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method models the monocular vision path tracking MVPF problem as a partially observable Markov decision process (POMDP), and applies the recurrent deep deterministic policy gradient algorithm (RDPG). The RDPG addresses the partial observability of the POMDP by introducing recurrent neural networks (RNNs) into the policy network and value network. The method includes: Acquire trajectory slice information composed of multiple trajectory point information; wherein, the trajectory point information includes first observation information, second observation information, action information and reward information, and the first observation information and the second observation information are obtained by the agent observing the environment at adjacent time points; Obtain the actor hidden state value and commenter hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein, the actor hidden state value is generated by the actor network in the model to be trained, and the commenter hidden state value is generated by the commenter network in the model to be trained; Based on the trajectory slice information, the actor's hidden state value, and the commentator's hidden state value, advantage estimation is performed to obtain the Q-value sequence; The actor network gradient and the commentator network gradient are determined based on the Q-value sequence. The parameters of the actor network are updated based on the gradient of the actor network, and the parameters of the commentator network are updated based on the gradient of the commentator network to obtain a path tracing model; wherein, the path tracing model is used for path tracing. The acquisition of trajectory slice information composed of multiple trajectory point information includes: The trajectory slice information is obtained by extracting multiple trajectory point information from random positions on the trajectory randomly selected from the playback buffer; The advantage estimation based on the trajectory slice information, the actor's hidden state value, and the commentator's hidden state value yields a Q-value sequence, including: The hidden state value of the actor is reconstructed to obtain the reconstructed hidden state value of the actor. Based on the reconstructed hidden state value of the actor, the hidden state value of the commenter is reconstructed to obtain the reconstructed hidden state value of the commenter. The time difference error value is calculated based on the reconstructed commenter hidden state value and the reward information in the slice information subset; The Q-value sequence is determined based on the time difference error value.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the state information of the agent at the first moment, and the first observation information collected by the agent at the first moment; The reward information is calculated based on the status information; The first observation information collected by the intelligent agent is input into the actor network to obtain the action information output by the actor network; The agent acquires second observation information collected at a second time point to obtain trajectory point information composed of the state information, the first observation information, the action information, the reward information, and the second observation information; wherein, the second time point is the time after the agent moves based on the action information.

3. The method according to claim 2, characterized in that, The state information includes the agent's position coordinates, heading direction, pitch speed, yaw speed, and yaw speed. The calculation of the reward information based on the state information includes: Determine the closest point to the agent on the reference path; The dummy variable corresponding to the nearest point is determined based on the Newton-Raphson algorithm; The first and second candidate coordinates are determined based on the virtual variables; The target virtual point is determined from the first candidate coordinates and the second candidate coordinates based on the position coordinates and the heading direction; Obtain the vector direction of the vector formed by the target virtual point and the position coordinates; The reward information is calculated based on the heading direction, the vector direction, the pitch speed, the yaw speed, and the yaw speed.

4. The method according to claim 2, characterized in that, The state information includes the agent's position coordinates, heading direction, pitch speed, yaw speed, and yaw speed. The calculation of the reward information based on the state information includes: The shortest distance between the agent and the reference path is determined based on the Newton-Raphson algorithm; Determine the first candidate coordinates and the second candidate coordinates based on the position coordinates and the preset radius; The target virtual point is determined from the first candidate coordinates and the second candidate coordinates based on the position coordinates and the heading direction; Obtain the vector direction of the vector formed by the target virtual point and the position coordinates; The reward information is calculated based on the heading direction, the vector direction, the pitch speed, the yaw speed, and the yaw speed.

5. The method according to claim 2, characterized in that, The state information includes the agent's position coordinates, heading direction, pitch speed, yaw speed, and yaw speed. The calculation of the reward information based on the state information includes: Determine the line-of-sight point closest to the agent on the reference path, and determine the line-of-sight direction based on the line-of-sight point and the position coordinates; The reward information is calculated based on the heading direction, the line of sight direction, the pitch speed, the yaw speed, and the yaw speed.

6. The method according to claim 1, characterized in that, The actor network and the commentator network are constructed based on multilayer perceptron, long short-term memory network, and spatially sensitive convolutional neural network.

7. A path tracing method, characterized in that, Applied to an intelligent agent, the intelligent agent being equipped with a path tracking model obtained by the model training method of claim 1, the method comprising: Acquire observational information from the surrounding environment using a monocular camera; The observation information is input into the path tracking model to obtain the action information output by the path tracking model; The agent's movement is controlled based on the motion information to perform path tracking.

8. A model training device, characterized in that, The device models the monocular vision path tracking MVPF problem as a partially observable Markov decision process (POMDP), and applies the recurrent deep deterministic policy gradient algorithm (RDPG). The RDPG addresses the partial observability of the POMDP by introducing recurrent neural networks (RNNs) into the policy network and value network. The device includes: The first acquisition module is used to acquire trajectory slice information composed of multiple trajectory point information; wherein, the trajectory point information includes first observation information, second observation information, action information and reward information, and the first observation information and the second observation information are obtained by the agent observing the environment at adjacent time points; The step of obtaining trajectory slice information composed of multiple trajectory point information includes: The trajectory slice information is obtained by extracting multiple trajectory point information from random positions on the trajectory randomly selected from the playback buffer; The second acquisition module is used to acquire the actor hidden state value and commenter hidden state value corresponding to each trajectory point information in the trajectory slice information; wherein, the actor hidden state value is generated by the actor network in the model to be trained, and the commenter hidden state value is generated by the commenter network in the model to be trained; The sequence module is used to perform advantage estimation based on the trajectory slice information, the actor's hidden state value, and the commentator's hidden state value to obtain a Q-value sequence; The Q-value sequence is obtained by: The hidden state value of the actor is reconstructed to obtain the reconstructed hidden state value of the actor. Based on the reconstructed hidden state value of the actor, the hidden state value of the commenter is reconstructed to obtain the reconstructed hidden state value of the commenter. The time difference error value is calculated based on the reconstructed commenter hidden state value and the reward information in the slice information subset; The Q-value sequence is determined based on the time difference error value; The gradient module is used to determine the gradient of the actor network and the gradient of the commentator network based on the Q-value sequence; The training module is used to update the parameters of the actor network based on the gradient of the actor network and the parameters of the commentator network based on the gradient of the commentator network to obtain a path tracking model; wherein, the path tracking model is used for path tracking.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as described in any one of claims 1 to 7.

Images