Target navigation system and method based on deep reinforcement learning in map-free environment

Abstract

The invention discloses a target navigation system and method based on deep reinforcement learning in a map-free environment. The method comprises the following steps: acquiring and preprocessing an image; predicting navigation actions; and executing and feeding back navigation actions. According to the method, map-free active target search is realized based on a deep reinforcement learning algorithm, the navigation performance is improved by implementing a staged dynamic reward mechanism combining distance and visual features, and the initiative and practicability of the robot are enhanced by adding an active stop mechanism.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a target navigation system and method based on deep reinforcement learning in a map-free environment. Background Art

[0002] Goal-based navigation is a key research area in embodied AI. It aims to guide robots through visual perception, guiding them to locate and reach designated targets while simultaneously outputting task-terminating actions. Unlike traditional navigation methods that rely on high-precision maps, map-free navigation uses end-to-end learning to predict actions directly from pixels. This uses reinforcement learning algorithms to implement an active navigation system, giving robots greater adaptability.

[0003] In recent years, navigation technology based on reinforcement learning (RL) has attracted much attention. Existing technologies have proposed a variety of methods to improve the navigation performance of robots in different environments, such as "Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning" (DOI: 10.1109 / ICRA.2017.7989381), which is based on the asynchronous advantage actor-critic (A3C) visual navigation model. By processing first-person visual input to output navigation actions, it realizes navigation without a map; and "Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization" (DOI: 10.1109 / LRA.2020.3048668 ) simulates action-observation dynamics by learning a variational generative model. Based on this model, the robot generates the next observation from the current observation and navigation goal, thereby enhancing the agent's ability to model action-observation dynamics and navigation performance. However, the sparse and fixed reward settings used in existing technologies restrict the learning of optimal policies, resulting in ineffective improvements in navigation performance. Furthermore, the lack of an active stopping mechanism in actual navigation tasks limits practical application effectiveness. Summary of the Invention

[0004] The purpose of the present invention is to provide a target navigation system and method based on deep reinforcement learning in a map-free environment to solve the problems raised in the above background technology.

[0005] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0006] A target navigation system based on deep reinforcement learning in a map-free environment, comprising an image input module (1), an image processing module (2), a variational reasoning module (3), a generation module (4), an action prediction module (5), a reward shaping module (6), an active stopping module (7) and an action execution module (8);

[0007] Image input module (1) obtains the robot's current observation image O in real time t With the target image I Goal , the image processing module (2) extracts O through convolutional neural network (CNN) t and I Goal The high-dimensional feature vectors are output to the variational reasoning module (3), the reward shaping module (6) and the active stopping module (7) respectively; the reward shaping module (6) calculates the reward value according to the new state of the robot and the staged dynamic reward function, and outputs the reward value to the variational reasoning module (3), the generation module (4) and the action prediction module (5); the variational reasoning module (3) calculates the reward value according to the staged dynamic reward function according to the new state of the robot t and I Goal The high-dimensional feature vector of , outputs the latent variable z to the generation module (4); the generation module (4) generates the next observation image O according to the latent variable z t+1 , output to the action prediction module (5); the action prediction module (5) performs the navigation task: obtains the next observation image O t+1 The last layer of features, the current observation image O t The features and action a at time t-1 t-1 features, jointly predict the navigation action a at time t t And output to the action execution module (8); the action execution module (8) receives the navigation action a output by the action prediction module (5) t , control the robot to perform corresponding actions in the simulation environment, so that the robot moves in the simulation environment; the active stop module (7) according to O t and I Goal The high-dimensional feature vector of real-time monitoring of the target achievement status, when O t and I Goal When the feature distance of the high-dimensional feature vector is higher than the threshold or the number of steps exceeds the limit, a termination signal is triggered, which causes the action execution module to control the robot to stop working, otherwise it continues to perform the navigation task.

[0008] Furthermore, the image input module (1) includes a current observation image input unit (11) and a target image input unit (12);

[0009] The current observation image input unit (11) is used to obtain the robot's observation image O of the environment at time t. t The target image input unit (12) is used to input the target image I Goal, the current observation image O t With the target image I Goal The state space of the extended Markov decision is used to represent the current state of the robot, where the extended Markov decision MDP g The expression is as follows:

[0010] MDP g = <S,A,P sa ,G,R>

[0011] Among them, S is the state space, A is the action space, and P sa is the state transition function, G is the target space, and R is the reward function.

[0012] Furthermore, the expression of the state space S is as follows:

[0013] s t =(O t ,I Goal )

[0014] S={s0,s1,……s t-1 ,s t ,s t+1 ……s n}

[0015] Among them, s t is the state of the robot at time t. When t = 0, the robot is at the initial moment; O t is the current observed image, I Goal is the target image; s0 represents the state of the robot at the initial moment, and n represents the maximum time step of the state sequence;

[0016] The expression of action space A is as follows:

[0017] a t ={MoveAhead,MoveBack,MoveLeft,MoveRight,RotateLeft,RotateRight,Stop}

[0018] A={a0,a1,……a t-1 ,a t ,a t+1 ……a n-1}

[0019] Among them, MoveAhead, MoveBack, MoveLeft, MoveRight, RotateLeft, RotateRight, and Stop represent moving forward, moving backward, moving left, moving right, turning left, turning right, and stopping respectively; the state is s tThe robot takes navigation action a at time t t After that, it enters the state s at time t+1 t+1 ; a0 represents the action taken by the robot at the initial moment. a0 is the first action in the action sequence, corresponding to the process of the robot transitioning from the initial state s0 to the next state s1;

[0020] State transition function P sa The expression is as follows:

[0021] P sa =p(s t+1 |s t ,a t )

[0022] Among them, P sa Represents the robot in state s t Execute navigation action a t Then transfer to state s t+1 probability;

[0023] The expression of the target space G is as follows:

[0024] g i =I Goal i (i=1,2,3,……,m)

[0025] G={g1,g2,……,g m}

[0026] Among them, g i Represents the target, which is an image containing the target object I Goal Specify; G is the target space, m is the total number of different targets in the target space; g i Represents the i-th target in the target space G, represented by the corresponding I Goali Indicates; I Goali Represents a given navigation target image in a task.

[0027] Stage-by-stage dynamic reward function R t The expression is as follows:

[0028] R t =f(s t ,a t )

[0029] Among them, R t Represents a robot performing a navigation action a at time t t The rewards obtained from the environment.

[0030] Furthermore, the current observation image O tis the observation image obtained by the robot at time t, O t+1 is the observation image obtained at time t+1, and the navigation action a t Indicates that O t With O t+1 The predicted action associated with two observed images, a t-1 is the action predicted at time t-1, a t gt is the correct action obtained from the simulation environment at time t, O t+1 gt Assume that the correct action a is taken at time t t gt The next correct observation image after the current observation information O t , navigation action a t and the next observation information O t+1 There is mutual information between them.

[0031] Furthermore, the A3C algorithm based on information theory regularization is adopted in the variational reasoning module (3), generation module (4) and action prediction module (5);

[0032] The variational reasoning module (3) calculates and outputs the KL divergence loss E3; in the generation module (4), the correct action a is assumed to be taken at time t obtained from the simulation environment t gt The next correct observation image O t+1 gt and the next observed image O generated t+1 The reconstruction loss E2 is formed between them; in the action prediction module (5), the navigation action a t and the correct action a obtained from the simulation environment at time t t gt Constitute the predictive control item E1.

[0033] Furthermore, the prediction control item E1 is as follows:

[0034]

[0035] The reconstruction loss E2 is as follows:

[0036] E2=‖O t+1 gt -O t+1 ‖ 2

[0037] In the actual navigation process, the correct action is a t gt A priori unknown, correct action a t gt Related to the target g, use the distribution q(z|Ot ,g) to approximate This process is constrained by KL divergence, as follows:

[0038] E3=KL[q(z|O t ,g)]‖p(z|a t gt ,O t )]

[0039] The objective function of the target navigation task is used to constrain the action prediction module (5) to predict the navigation action a t , as follows:

[0040] J(O t ,g)=αE1-βE2-γE3

[0041] Among them, α is the weight parameter of E1, β is the weight parameter of E2, and γ is the weight parameter of E3.

[0042] Furthermore, the reward shaping module (6) is based on the new state of the robot and the stage-by-stage dynamic reward function R t Calculate the reward value and output the reward value as follows:

[0043]

[0044] Among them, a t represents the navigation action at time t predicted by the action prediction module (5), r time Set the time step penalty for each time step; x t (x1, y1) is the position of the robot at time t, g(x gi ,y gi ) is the target g i The location of Geo(x t ,g) is the geodesic distance between the robot and the target position. The geodesic distance is calculated as the straight-line distance between the robot's current position and the target position. The geodesic distance calculation formula is:

[0045]

[0046] D is the set distance threshold, when Geo(x t ,g)>D, it means that the robot is far away from the target position, and the robot is considered to be in the exploration stage; Geo(x t ,g)≤D, it means that the robot position is close to the target position, and the robot is considered to be in the approaching stage; distance is the geodesic distance reward, which represents the difference between the robot's geodesic distance at the previous moment and the current moment t, λ far is the weight parameter of the geodesic distance reward in the exploration phase, λnear is the weight parameter of the geodesic distance reward in the approach phase; r visual is the visual feature reward, which is obtained by observing the robot's image O t and target image I Goal The cosine similarity of the extracted feature vectors is used to shape; β far is the weight parameter of the visual feature reward in the exploration phase, β near is the weight parameter of the visual feature reward in the approach phase;

[0047] r distance The expression is as follows:

[0048] r distance =Geo(x t-1 ,g)-Geo(x t ,g)

[0049] When Geo(x t-1 ,g)>Geo(x t ,g), indicating that the robot has completed the current action a at time t t After that, it is moving towards the target direction, and the distance is getting smaller and smaller. At this time, r distance It is a positive reward;

[0050] r visual The expression is as follows:

[0051]

[0052] in, is the current observation image O t The eigenvector of is the target image I Goal If the eigenvector of and F g The more similar, the visual The closer the value of is to 1, the closer the current visual information is to the target visual information. and F g If the directions are not similar, then r visual The closer the value is to -1, the greater the reward value r visual It will be lower.

[0053] Furthermore, the active stopping module (7) calculates the current observation image O extracted by the image processing module (2). t The eigenvector of and target image I Goal The eigenvector of Cosine similarity between Compare with the set threshold τ to determine whether to issue a stop command. If the robot reaches the target position, it will issue a stop command and end the navigation task; if the threshold is not reached, it will continue with the next round of environment perception, action decision-making and navigation execution process.

[0054] Furthermore, the expression of cosine similarity is as follows:

[0055]

[0056] in, is the current observation image O t The eigenvector of is the target image I Goal The eigenvector of .

[0057] The target navigation method based on deep reinforcement learning in a map-free environment includes the following steps:

[0058] S1. Image acquisition and preprocessing: Real-time acquisition of the robot's current observation image O t With the target image I Goal , extract O through convolutional neural network (CNN) t and I Goal High-dimensional feature vector of ;

[0059] S2, navigation action prediction: According to O t and I Goal The high-dimensional feature vector of the output latent variable z; according to the latent variable z, the next observation image O is generated t+1 ; According to the next observation image O t+1 The last layer of features, the current observation image O t The features and action a at time t-1 t-1 features, jointly predict the navigation action a at time t t ;

[0060] S3, Navigation action execution and feedback: Control the robot to perform navigation action a at time t in the simulation environment t ; According to the navigation action a performed by the robot at time t t The reward value is calculated based on the staged dynamic reward function: if the robot reaches the target position and stops, it is given a positive reward; if it collides, exceeds the drivable area, or stops before reaching the target, it is given a negative reward. During normal movement, the reward is given according to the distance to the target and the similarity of visual features.

[0061] The robot performs navigation action a t Afterwards, according to O t and I Goal The high-dimensional feature vector of real-time monitoring of the target achievement status, when O t and IGoal When the feature distance of the high-dimensional feature vector is higher than the threshold or the number of steps exceeds the limit, a termination signal is triggered, and the termination signal causes the action execution module to control the robot to stop working. If it is lower than the threshold, it returns to step S1.

[0062] Compared with the existing technology, the beneficial effects of the present invention are: the present invention realizes active target search without a map based on a deep reinforcement learning algorithm, improves navigation performance by implementing a phased dynamic reward mechanism that combines distance and visual features, and enhances the initiative and practicality of the robot by adding an active stopping mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a structural block diagram of a target navigation system based on deep reinforcement learning in a map-free environment in an embodiment of the present invention.

[0064] Figure 2 This is a structural block diagram of an algorithm for predicting navigation actions by an action prediction module in an embodiment of the present invention.

[0065] Figure 3 This is a flowchart of a target navigation method based on deep reinforcement learning in a map-free environment in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0067] In one embodiment, a target navigation system based on deep reinforcement learning in a map-free environment, such as Figure 1 As shown, it includes an image input module 1, an image processing module 2, a variational reasoning module 3, a generation module 4, an action prediction module 5, a reward shaping module 6, an active stopping module 7 and an action execution module 8;

[0068] Image input module 1 obtains the robot's current observation image O in real time t With the target image I Goal , the image processing module 2 extracts O through the convolutional neural network CNN t and I Goal The high-dimensional feature vectors are output to the variational reasoning module 3, the reward shaping module 6 and the active stopping module 7 respectively; the reward shaping module 6 calculates the reward value according to the staged dynamic reward function based on the new state of the robot, and outputs the reward value to the variational reasoning module 3, the generation module 4 and the action prediction module 5; the variational reasoning module 3 calculates the reward value according to the staged dynamic reward function based on the new state of the robot, and outputs the reward value to the variational reasoning module 3, the generation module 4 and the action prediction module 5; t and I Goal The high-dimensional feature vector of the latent variable z is output to the generation module 4; the generation module 4 generates the next observation image O according to the latent variable z t+1, output to the action prediction module 5; the action prediction module 5 performs the navigation task: obtain the next observation image O t+1 The last layer of features, the current observation image O t The features and action a at time t-1 t-1 features, jointly predict the navigation action a at time t t And output to the action execution module 8; the action execution module 8 receives the navigation action a output by the action prediction module 5 t , control the robot to perform corresponding actions in the simulation environment, so that the robot moves in the simulation environment; active stop module 7 according to O t and I Goal The high-dimensional feature vector of real-time monitoring of the target achievement status, when O t and I Goal When the feature distance of the high-dimensional feature vector is higher than the threshold or the number of steps exceeds the limit, a termination signal is triggered, which causes the action execution module to control the robot to stop working, otherwise it continues to perform the navigation task.

[0069] Furthermore, the image input module 1 includes a current observation image input unit 11 and a target image input unit 12;

[0070] The current observation image input unit 11 is used to obtain the robot's observation image O of the environment at time t. t The target image input unit 12 is used to input the target image I Goal , the current observation image O t With the target image I Goal The state space of the extended Markov decision is used to represent the current state of the robot, where the extended Markov decision MDP g The expression is as follows:

[0071] MDP g = <S,A,P sa ,G,R>

[0072] Among them, S is the state space, A is the action space, and P sa is the state transition function, G is the target space, and R is the reward function.

[0073] Furthermore, the expression of the state space S is as follows:

[0074] s t =(O t ,I Goal )

[0075] S={s0,s1,……s t-1 ,s t ,s t+1 ……s n}

[0076] Among them, s t is the state of the robot at time t. When t = 0, the robot is at the initial moment; O t is the current observed image, I Goal is the target image; s0 represents the state of the robot at the initial moment, and n represents the maximum time step of the state sequence;

[0077] The expression of action space A is as follows:

[0078] a t ={MoveAhead,MoveBack,MoveLeft,MoveRight,RotateLeft,RotateRight,Stop}

[0079] A={a0,a1,……a t-1 ,a t ,a t+1 ……a n-1}

[0080] Among them, MoveAhead, MoveBack, MoveLeft, MoveRight, RotateLeft, RotateRight, and Stop represent moving forward, moving backward, moving left, moving right, turning left, turning right, and stopping respectively; the state is s t The robot takes navigation action a at time t t After that, it enters the state s at time t+1 t+1 ; a0 represents the action taken by the robot at the initial moment. a0 is the first action in the action sequence, corresponding to the process of the robot transitioning from the initial state s0 to the next state s1;

[0081] State transition function P sa The expression is as follows:

[0082] P sa =p(s t+1 |s t ,a t )

[0083] Among them, P sa Represents the robot in state s t Execute navigation action a t Then transfer to state s t+1 probability;

[0084] The expression of the target space G is as follows:

[0085] g i =I Goali (i=1,2,3,……,m)

[0086] G={g1,g2,……,g m}

[0087] Among them, g i Represents the target, which is an image containing the target object I Goal Specify; G is the target space, m is the total number of different targets in the target space; g i Represents the i-th target in the target space G, represented by the corresponding I Goali Indicates; I Goali Represents a given navigation target image in a task.

[0088] Stage-by-stage dynamic reward function R t The expression is as follows:

[0089] R t =f(s t ,a t )

[0090] Among them, R t Represents a robot performing a navigation action a at time t t The rewards obtained from the environment.

[0091] like Figure 2 As shown, the current observation image O t is the observation image obtained by the robot at time t, O t+1 is the observation image obtained at time t+1, and the navigation action a t Indicates that O t With O t+1 The predicted action associated with two observed images, a t-1 is the action predicted at time t-1, a t gt is the correct action obtained from the simulation environment at time t, O t+1 gt Assume that the correct action a is taken at time t t gt The next correct observation image after the current observation information O t , navigation action a t and the next observation information O t+1 There is mutual information between them.

[0092] Furthermore, the A3C algorithm based on information-theoretic regularization (Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization (DOI: 10.1109 / LRA.2020.3048668)) is used in the variational reasoning module 3, the generation module 4, and the action prediction module 5.

[0093] In the variational reasoning module 3, the KL divergence loss E3 is calculated and output; in the generation module 4, the correct action a is assumed to be taken at time t obtained from the simulation environment t gt The next correct observation image O t+1 gt and the next observed image O generated t+1 The reconstruction loss E2 is formed between them; in the action prediction module 5, the navigation action a t and the correct action a obtained from the simulation environment at time t t gt Constitute the predictive control item E1.

[0094] Variational inference module 3 converts a t gt With O t Construct the prior distribution p(z|a t gt ,O t ) and the output distribution q(z|O t ,g), calculate the KL divergence loss E3, the generation module 4 generates the next observation image O from the latent variable z t+1 , and calculate the Ground-Truth next observation image O t+1 gt and the next observation image O generated by the model t+1 The reconstruction loss E2 between them is constructed using the MSE loss. The action prediction module 5 will generate the next observation image O of the module 4. t+1 The last layer of features, the current observation image O t The features of the previous action are fused and the navigation action a is predicted based on the fused features. t , through the predicted navigation action a t With Ground-Truth action a t gt Calculate the predictive control term E1, which is constructed using cross entropy loss;

[0095] Furthermore, the prediction control item E1 is as follows:

[0096]

[0097] The reconstruction loss E2 is as follows:

[0098] E2=‖O t+1 gt -O t+1 ‖ 2

[0099] In the actual navigation process, the correct action is a t gt A priori unknown, correct action a t gt Related to the target g, use the distribution q(z|O t ,g) to approximate This process is constrained by KL divergence, as follows:

[0100] E3=KL[q(z|O t ,g)]‖p(z|a t gt ,O t )]

[0101] The objective function of the target navigation task is used to constrain the action prediction module (5) to predict the navigation action a t , as follows:

[0102] J(O t ,g)=αE1-βE2-γE3

[0103] Wherein, α is the weight parameter of E1, β is the weight parameter of E2, and γ is the weight parameter of E3. In one embodiment, α=1, β=0.1, and γ=0.0001.

[0104] Furthermore, the reward shaping module 6 is based on the new state of the robot and the stage-by-stage dynamic reward function R t Calculate the reward value and output the reward value as follows:

[0105]

[0106] Among them, a t represents the navigation action at time t predicted by the action prediction module 5, r time Set a time step penalty for each time step, in one embodiment, set to -0.01; x t (x1, y1) is the position of the robot at time t, g(x gi ,y gi ) is the target g i The location of Geo(x t,g) is the geodesic distance between the robot and the target position. The geodesic distance is calculated as the straight-line distance between the robot's current position and the target position. The geodesic distance calculation formula is:

[0107]

[0108] D is the set distance threshold, when Geo(x t ,g)>D, it means that the robot is far away from the target position, and the robot is considered to be in the exploration stage; Geo(x t ,g)≤D, it means that the robot position is close to the target position, and the robot is considered to be in the approaching stage; distance is the geodesic distance reward, which represents the difference between the robot's geodesic distance at the previous moment and the current moment t, λ far is the weight parameter of the geodesic distance reward in the exploration phase, λ near is the weight parameter of the geodesic distance reward in the approach phase; r visual is the visual feature reward, which is obtained by observing the robot's image O t and target image I Goal The cosine similarity of the extracted feature vectors is used to shape; β far is the weight parameter of the visual feature reward in the exploration phase, β near is the weight parameter of the visual feature reward in the approach phase;

[0109] r distance The expression is as follows:

[0110] r distance =Geo(x t-1 ,g)-Geo(x t ,g)

[0111] When Geo(x t-1 ,g)>Geo(x t ,g), indicating that the robot has completed the current action a at time t t After that, it is moving towards the target direction, and the distance is getting smaller and smaller. At this time, r distance It is a positive reward;

[0112] r visual The expression is as follows:

[0113]

[0114] in, is the current observation image O t The eigenvector of The target image I Goal If the eigenvector of and F g The more similar, thevisual The closer the value of is to 1, the closer the current visual information is to the target visual information. and F g If the directions are not similar, then r visual The closer the value is to -1, the greater the reward value r visual It will be lower.

[0115] Furthermore, the active stopping module 7 calculates the current observation image O extracted by the image processing module 2. t The eigenvector of and target image I Goal The eigenvector of Cosine similarity between Compare with the set threshold τ to determine whether to issue a stop command. If the robot reaches the target position, it will issue a stop command and end the navigation task; if the threshold is not reached, it will continue with the next round of environment perception, action decision-making and navigation execution process.

[0116] Furthermore, the expression of cosine similarity is as follows:

[0117]

[0118] in, is the current observation image O t The eigenvector of The target image I Goal The eigenvector of .

[0119] Target navigation method based on deep reinforcement learning in map-free environment, such as Figure 3 As shown, the following steps are included:

[0120] S1. Image acquisition and preprocessing: Real-time acquisition of the robot's current observation image O t With the target image I Goal , extract O through convolutional neural network (CNN) t and I Goal High-dimensional feature vector of ;

[0121] S2, navigation action prediction: According to O t and I Goal The high-dimensional feature vector of the output latent variable z; according to the latent variable z, the next observation image O is generated t+1 ; According to the next observation image O t+1 The last layer of features, the current observation image O t The features and action a at time t-1 t-1 features, jointly predict the navigation action a at time t t ;

[0122] S3, Navigation action execution and feedback: Control the robot to perform navigation action a at time t in the simulation environment t ; According to the navigation action a performed by the robot at time t t The reward value is calculated based on the staged dynamic reward function: if the robot reaches the target position and stops, it is given a positive reward; if it collides, exceeds the drivable area, or stops before reaching the target, it is given a negative reward. During normal movement, the reward is given according to the distance to the target and the similarity of visual features.

[0123] The robot performs navigation action a t Afterwards, according to O t and I Goal The high-dimensional feature vector of real-time monitoring of the target achievement status, when O t and I Goal When the feature distance of the high-dimensional feature vector is higher than the threshold or the number of steps exceeds the limit, a termination signal is triggered, and the termination signal causes the action execution module to control the robot to stop working. If it is lower than the threshold, it returns to step S1.

[0124] In one embodiment, in order to verify the effectiveness of the present invention in the target navigation task, the following experiment is conducted:

[0125] 1. Training settings:

[0126] The operating system is Ubuntu 20.04.6 LTS, and the environment configuration versions are Python 3.6.6 and PyTorch 1.0.1. Six independent running environment instances were built, with the discount factor set to 0.99, the entropy coefficient set to 0.01, the gradient clipping threshold set to 0.5, the sampling step number set to 10, and the RMSprop optimizer used to update the network parameters. The total number of training frames was 4 million, and the learning rate was fixed at 1e-4.

[0127] 2. Experimental setup:

[0128] The simulation environment used in this experiment is AI2-THOR. In AI2-THOR, the robot obtains observation images and target position images taken from a first-person perspective through an API. At each time step t, the API can be called to directly determine whether the robot's action has collided, whether the current observation includes the navigation target, and the distance to the target. The optional actions of the simulated robot include moving forward, moving backward, moving left, moving right, rotating left, rotating right, and stopping. Among them, the stop action indicates the end of the task. When the stop action is executed, the robot remains in place. The robot moves a distance of 0.25 meters each time and rotates 90 degrees. The settings for successful task completion are as follows:

[0129] Time step limit: The robot completes the task within a limited number of steps and outputs a stop action;

[0130] Goal achievement requirements: the target object is visible in the target image, and the distance between the robot and the target object is less than the preset threshold;

[0131] 3. Evaluation index setting:

[0132] Success rate SR: When the robot issues a "stop" action, it means that a task is completed. A total of N task paths are tested. When the robot meets the requirements for task success for the i-th time, S i Recorded as 1, otherwise S i It is recorded as 0 and defined as the following formula:

[0133]

[0134] Success weighted path length SPL:

[0135] The success-weighted path length takes into account both the mission success rate and the optimality of the robot’s navigation path and is defined as follows:

[0136]

[0137] where l i is the shortest path length in the actual navigation path of the robot, p i is the actual path length of the robot during the i-th task;

[0138] 4. Algorithm overall performance experiment:

[0139] The robot starts from a random position and navigates to the target position using the motion prediction method of the present invention. The experiment involves four scenarios, with 250 experiments in each scenario, for a total of 1000 experiments. To evaluate navigation performance, tasks are divided into two categories, L≥1 and L≥5, based on the optimal path length L. The success rates and successful weighted path lengths of the two categories of tasks are statistically analyzed, and the results are shown in Tables 1 and 2. It can be seen that the overall performance of the present invention in the target navigation task is superior to that of other benchmark methods.

[0140] Table 1 Results of L≥1

[0141]

[0142] Table 2 Results of L>=5

[0143]

[0144] Based on the above, the advantages of the present invention are that the present invention realizes active target search without a map based on a deep reinforcement learning algorithm, improves navigation performance by implementing a phased dynamic reward mechanism that combines distance and visual features, and enhances the initiative and practicality of the robot by adding an active stopping mechanism.

[0145] The preferred embodiments of the present application disclosed above are intended only to facilitate understanding of the present invention and its core concepts. Those skilled in the art will appreciate that specific application scenarios and implementations may vary based on the principles of the present invention, and this description should not be construed as limiting the present invention. The present invention is intended to be limited only by the claims and their full scope and equivalents.

Claims

1. A target navigation system based on deep reinforcement learning in a map-free environment, characterized by: It includes an image input module (1), an image processing module (2), a variational reasoning module (3), a generation module (4), an action prediction module (5), a reward shaping module (6), an active stopping module (7) and an action execution module (8); Image input module (1) obtains the robot's current observation image O in real time t With the target image I Goal , the image processing module (2) extracts O through convolutional neural network (CNN) t and I Goal The high-dimensional feature vectors are output to the variational reasoning module (3), the reward shaping module (6) and the active stopping module (7) respectively; the reward shaping module (6) calculates the reward value according to the new state of the robot and the staged dynamic reward function, and outputs the reward value to the variational reasoning module (3), the generation module (4) and the action prediction module (5); the variational reasoning module (3) calculates the reward value according to the staged dynamic reward function according to the new state of the robot t and I Goal The high-dimensional feature vector of , outputs the latent variable z to the generation module (4); the generation module (4) generates the next observation image O according to the latent variable z t+1 , output to the action prediction module (5); the action prediction module (5) performs the navigation task: obtains the next observation image O t+1 The last layer of features, the current observation image O t The features and action a at time t-1 t-1 features, jointly predict the navigation action a at time t t And output to the action execution module (8); the action execution module (8) receives the navigation action a output by the action prediction module (5) t , control the robot to perform corresponding actions in the simulation environment, so that the robot moves in the simulation environment; the active stop module (7) according to O t and I Goal The high-dimensional feature vector of real-time monitoring of the target achievement status, when O t and I Goal When the feature distance of the high-dimensional feature vector is higher than the threshold or the number of steps exceeds the limit, a termination signal is triggered, which causes the action execution module to control the robot to stop working, otherwise it continues to perform the navigation task.

2. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 1, characterized in that: The image input module (1) includes a current observation image input unit (11) and a target image input unit (12); The current observation image input unit (11) is used to obtain the robot's observation image O of the environment at time t. t The target image input unit (12) is used to input the target image I Goal , the current observation image O t With the target image I Goal The state space of the extended Markov decision is used to represent the current state of the robot, where the extended Markov decision MDP g The expression is as follows: MDP g =<S,A,P sa ,G,R> Among them, S is the state space, A is the action space, and P sa is the state transition function, G is the target space, and R is the reward function.

3. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 2, characterized in that: The expression of the state space S is as follows: s t =(O t ,I Goal ) S={s0,s1,……s t-1 ,s t ,s t+1 ……s n } Among them, s t is the state of the robot at time t. When t = 0, the robot is at the initial moment; O t is the current observed image, I Goal is the target image; s0 represents the state of the robot at the initial moment, and n represents the maximum time step of the state sequence; The expression of action space A is as follows: a t ={MoveAhead,MoveBack,MoveLeft,MoveRight,RotateLeft,RotateRight,Stop} A={a0,a1,……a t-1 ,a t ,a t+1 ……a n-1 } Among them, MoveAhead, MoveBack, MoveLeft, MoveRight, RotateLeft, RotateRight, and Stop represent moving forward, moving backward, moving left, moving right, turning left, turning right, and stopping respectively; the state is s t The robot takes navigation action a at time t t After that, it enters the state s at time t+1 t+1 ; a0 represents the action taken by the robot at the initial moment. a0 is the first action in the action sequence, corresponding to the process of the robot transitioning from the initial state s0 to the next state s1; State transition function P sa The expression is as follows: P sa =p(s t+1 |s t ,a t ) Among them, P sa Represents the robot in state s t Execute navigation action a t Then transfer to state s t+1 probability; The expression of the target space G is as follows: g i =I Goali (i=1,2,3,……,m) G={g1,g2,……,g m } Among them, g i Represents the target, which is an image containing the target object I Goal Specify; G is the target space, m is the total number of different targets in the target space; g i Represents the i-th target in the target space G, represented by the corresponding I Goali Indicates; I Goali Represents a given navigation target image in a task; Stage-by-stage dynamic reward function R t The expression is as follows: R t =f(s t ,a t ) Among them, R t Represents a robot performing a navigation action a at time t t The rewards obtained from the environment.

4. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 1, characterized in that: Current observation image O t is the observation image obtained by the robot at time t, O t+1 is the observation image obtained at time t+1, and the navigation action a t Indicates that O t With O t+1 The predicted action associated with two observed images, a t-1 is the action predicted at time t-1, a t gt is the correct action obtained from the simulation environment at time t, O t+1 gt Assume that the correct action a is taken at time t t gt The next correct observation image after the current observation information O t , navigation action a t and the next observation information O t+1 There is mutual information between them.

5. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 4 is characterized in that: The A3C algorithm based on information theory regularization is used in the variational reasoning module (3), generation module (4) and action prediction module (5); The variational reasoning module (3) calculates and outputs the KL divergence loss E3; in the generation module (4), the correct action a is assumed to be taken at time t obtained from the simulation environment t gt The next correct observation image O t+1 gt and the next observed image O generated t+1 The reconstruction loss E2 is formed between them; in the action prediction module (5), the navigation action a t and the correct action a obtained from the simulation environment at time t t gt Constitute the predictive control item E1.

6. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 5, characterized in that: The specific details of the forecast control item E1 are as follows: The reconstruction loss E2 is as follows: E2=‖O t+1 gt -Oh t+1 ‖ 2 In the actual navigation process, the correct action is a t gt A priori unknown, correct action a t gt Related to the target g, use the distribution q(z|O t ,g) to approximate This process is constrained by KL divergence, as follows: E3=KL[q(z|O t ,g)]‖p(z|a t gt ,O t )] The objective function of the target navigation task is used to constrain the action prediction module (5) to predict the navigation action a t , as follows: J(O t ,g)=αE1-βE2-γE3 Among them, α is the weight parameter of E1, β is the weight parameter of E2, and γ is the weight parameter of E3.

7. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 3, characterized in that: The reward shaping module (6) is based on the new state of the robot according to the staged dynamic reward function R t Calculate the reward value and output the reward value as follows: Among them, a t represents the navigation action at time t predicted by the action prediction module (5), r time Set the time step penalty for each time step; x t (x1, y1) is the position of the robot at time t, g(x gi ,y gi ) is the target g i The location of Geo(x t ,g) is the geodesic distance between the robot and the target position. The geodesic distance is calculated as the straight-line distance between the robot's current position and the target position. The geodesic distance calculation formula is: D is the set distance threshold, when Geo(x t ,g)>D, it means that the robot is far away from the target position, and the robot is considered to be in the exploration stage; Geo(x t ,g)≤D, it means that the robot position is close to the target position, and the robot is considered to be in the approaching stage; distance is the geodesic distance reward, which represents the difference between the robot's geodesic distance at the previous moment and the current moment t, λ far is the weight parameter of the geodesic distance reward in the exploration phase, λ near is the weight parameter of the geodesic distance reward in the approach phase; r visual is the visual feature reward, which is obtained by observing the robot's image O t and target image I Goal The cosine similarity of the extracted feature vectors is used to shape; β far is the weight parameter of the visual feature reward in the exploration phase, β near is the weight parameter of the visual feature reward in the approach phase; r distance The expression is as follows: r distance =Geo(x t-1 ,g)-Geo(x t ,g) When Geo(x t-1 ,g)>Geo(x t ,g), indicating that the robot has completed the current action a at time t t After that, it is moving towards the target direction, and the distance is getting smaller and smaller. At this time, r distance It is a positive reward; r visual The expression is as follows: in, is the current observation image O t The eigenvector of The target image I Goal If the eigenvector of and F g The more similar, the visual The closer the value of is to 1, the closer the current visual information is to the target visual information. and F g If the directions are not similar, then r visual The closer the value is to -1, the greater the reward value r visual It will be lower.

8. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 1, characterized in that: The active stopping module (7) calculates the current observation image O extracted by the image processing module (2) t The eigenvector of and target image I Goal The eigenvector of Cosine similarity between Compare with the set threshold τ to determine whether to issue a stop command. If the robot reaches the target position, it will issue a stop command and end the navigation task; if the threshold is not reached, it will continue with the next round of environment perception, action decision-making and navigation execution process.

9. The target navigation system based on deep reinforcement learning in a map-free environment according to claim 8, characterized in that: The expression of cosine similarity is as follows: in, is the current observation image O t The eigenvector of The target image I Goal The eigenvector of .

10. A target navigation method based on deep reinforcement learning in a map-free environment, characterized by: The steps include: S1. Image acquisition and preprocessing: Real-time acquisition of the robot's current observation image O t With the target image I Goal , extract O through convolutional neural network (CNN) t and I Goal High-dimensional feature vector of ; S2, navigation action prediction: According to O t and I Goal The high-dimensional feature vector of , outputs the latent variable z; According to the latent variable z, the next observation image O is generated t+1 ; According to the next observation image O t+1 The last layer of features, the current observation image O t The features and action a at time t-1 t-1 features, jointly predict the navigation action a at time t t ; S3, Navigation action execution and feedback: Control the robot to perform navigation action a at time t in the simulation environment t ; According to the navigation action a performed by the robot at time t t The reward value is calculated based on the staged dynamic reward function: if the robot reaches the target position and stops, it is given a positive reward; if it collides, exceeds the drivable area, or stops before reaching the target, it is given a negative reward. During normal movement, the reward is given according to the distance to the target and the similarity of visual features. The robot performs navigation action a t Afterwards, according to O t and I Goal The high-dimensional feature vector of real-time monitoring of the target achievement status, when O t and I Goal When the feature distance of the high-dimensional feature vector is higher than the threshold or the number of steps exceeds the limit, a termination signal is triggered, and the termination signal causes the action execution module to control the robot to stop working. If it is lower than the threshold, it returns to step S1.

Images