A navigation method and device, a mobile robot and a readable storage medium

Abstract

Embodiments of the present application provide a navigation method and device, a mobile robot and a readable storage medium. The method comprises: training a preset action network model according to the number of times that the mobile robot reaches a target position before each training, a preset random number, a reward of state information at each time in historical state information of the current environment in which the mobile robot is located, the state information at each time in the historical state information and a corresponding real action, obtaining a trained action network model, and then outputting an optimal executable action based on the current state information of the current environment in which the mobile robot is located and the target position from the trained action network model, so that the mobile robot moves based on the optimal executable action. In this way, the dependence of the mobile robot on curiosity can be reduced, the dependence on the number of times of reaching the target position is increased, and the interference of the environment on the navigation of the robot is reduced, and the navigation difficulty of the robot is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a navigation method and device, a mobile robot and a readable storage medium. BACKGROUND

[0002] Mobile robots are widely used in various industries such as industry, hotel, security, and hospital, and have broad application fields. Accurate autonomous navigation is the key to mobile robots, and exploratory navigation in unknown areas is still a challenge for robots. Reinforcement learning algorithm-based navigation methods have shown excellent exploration success rate and navigation performance in recent research. Robots learn strategies to maximize future reward totals by interacting with environmental states, enabling robots to achieve the most effective action behavior. Different reward mechanisms set in the same navigation task also affect the efficiency and accuracy of navigation.

[0003] There are two methods for setting internal rewards for reinforcement learning: one method is to drive robots to explore by designing different action behaviors for robots. Some people improve the optimal behavior strategy of robots through reward shaping functions. Another method drives robots to explore by evaluating environmental states. This method encourages robots to explore unknown areas by calculating the curiosity of robots about new exploration areas to assign reward weights.

[0004] However, the applicant found that in the prior art, when the path for the robot to pass through in the scene using the robot does not change, but only the decoration or decoration around the path changes, the robot may mistakenly judge a position that has been explored as a new position, for example, there is a TV in the environment, since the TV will switch pictures regularly, the picture is different each time the robot visits this position, at this time, the robot will judge the position where the TV is as a new position instead of the previously recorded position that has been explored, so that the robot always regards the position where the TV is as a new position, thereby causing the robot to be in a deadlock state. It can be seen that when the environment in the scene using the robot changes frequently, the environment will increase the interference to the navigation of the robot and increase the difficulty of navigation of the robot. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a navigation method, device, mobile robot and readable storage medium to reduce the interference of the environment to the navigation of the robot and reduce the difficulty of navigation of the robot. The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of the present application provide a navigation method, which comprises:

[0007] The current state information and target position of the mobile robot in its current environment are obtained; the current state information and target position are input into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot can move based on the best executable action;

[0008] The trained action network model is obtained by iteratively executing the following steps:

[0009] Input the historical state information of the mobile robot in its current environment into the i-th... t The action network model obtained from the training iteration, i-th... t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The predicted action output by the action network model obtained from the training; historical state information is the state information of the mobile robot at each moment in the current environment. The state information includes images marked with location information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position in the current environment to the target position; 1≤t≤N-1, where N is the total number of historical state information.

[0010] The state information at time t+1, the state information at time t, and the actual action at time t are input into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1. The state information at time t+1 is generated by the mobile robot interacting with the current environment based on the actual action at time t.

[0011] Input the state information at time t+1 and the state information at time t into the i-th... t The evaluation network model obtained from the training iterations yields the i-th... t The second reward value output by the evaluation network model obtained from the training, which corresponds to the state information at time t+1.

[0012] Update the i-th reward value based on the first and second reward values. t The model parameters of the evaluation network model obtained from the training iteration are used to obtain the i-th training iteration. t+1 The evaluation network model obtained from the i-th training iteration; and based on the i-th training iteration... t The predicted action output by the action network model obtained from the training iteration and the corresponding real action at time t are used to update the i-th action. t The model parameters of the action network model obtained from the training iteration are used to obtain the i-th... t+1 The action network model obtained from the training;

[0013] the action network model obtained by the i t+1 th training and the evaluation network model obtained by the i t+1 th training as the action network model obtained by the i t th training and the evaluation network model obtained by the i t th training.

[0014] the state information at the t+1th time in the historical state information as the state information at the tth time, and performing the step of inputting the historical state information of the mobile robot in the current environment to the action network model obtained by the i t th training, the action network model obtained by the i t th training determines the predicted action output by the action network model obtained by the i t th training according to the number of times that the mobile robot reaches the target position before the i t th training, the preset random number and the reward of the state information at each time in the historical state information.

[0015] Optionally, the action network model obtained by the i t th training determines the predicted action output by the action network model obtained by the i t th training according to the number of times that the mobile robot reaches the target position before the i t th training, the preset random number and the reward of the state information at each time in the historical state information, including:

[0016] the action network model obtained by the i t th training determines the predicted action output by the action network model obtained by the i t th training according to the number of times that the mobile robot reaches the target position before the i greed th training, i.e. i-1, through the following formula:

[0017] e greed =0.8-min(10,(i-1) / 10)×θ

[0018] calculating an exploration threshold e t ; θ is a control factor, 0.01≤θ≤0.08;

[0019] obtaining a random number; when the random number is less than the exploration threshold, randomly selecting any direction as the predicted action output by the action network model obtained by the i t th training;

[0020] when the random number is greater than or equal to the exploration threshold, obtaining the reward corresponding to the state information at the tth time in the historical state information before the i t th training, and taking the action that makes the total reward value of the state information at the tth time maximum as the predicted action output by the action network model obtained by the iThe action network model outputs a predicted action.

[0021] Optionally, the state information at the t+1 moment, the state information at the t moment and the real action corresponding to the t moment are input into the curiosity reward model to obtain a first reward value corresponding to the state information at the t+1 moment output by the curiosity reward model, including:

[0022] The state information at the t moment and the real action corresponding to the t moment are input into a feedforward predictor model in the curiosity reward model to obtain a predicted environmental feature corresponding to the t+1 moment output by the feedforward predictor model.

[0023] The state information at the t+1 moment and the state information at the t moment are input into a sliding window sub-model in the curiosity reward model to obtain a real environmental feature corresponding to the t+1 moment and a feature weight bias output by the sliding window sub-model.

[0024] Based on the predicted environmental feature corresponding to the t+1 moment, the real environmental feature corresponding to the t+1 moment and the feature weight bias, a first reward value corresponding to the state information at the t+1 moment output by the curiosity reward model is obtained.

[0025] Optionally, the state information at the t+1 moment and the state information at the t moment are input into a sliding window sub-model in the curiosity reward model to obtain a real environmental feature corresponding to the t+1 moment and a feature weight bias output by the sliding window sub-model, including:

[0026] The state information at the t+1 moment and the state information at the t moment are input into a sliding window sub-model in the curiosity reward model; the sliding window sub-model stores a plurality of sliding windows, and each sliding window includes a plurality of environmental features.

[0027] When the sliding window sub-model in the curiosity reward model simultaneously searches for real environmental features corresponding to the state information at the t moment and the state information at the t+1 moment in a first sliding window in the plurality of sliding windows, the real environmental feature corresponding to the t+1 moment is output, and a preset first feature weight bias is obtained as the feature weight bias; the first feature weight bias is greater than 0.5; the first sliding window is a sliding window in the plurality of sliding windows that contains at least one of the real environmental features corresponding to the state information at the t moment or the state information at the t+1 moment.

[0028] Based on the predicted environmental feature corresponding to the t+1 moment, the real environmental feature corresponding to the t+1 moment and the feature weight bias, a first reward value output by the curiosity reward model is obtained, including:

[0029] the predicted environment feature corresponding to the t+1 moment the real environment feature corresponding to the t+1 moment and a feature weight bias b; the first reward value corresponding to the state information of the t+1 moment is calculated by the following formula:

[0030]

[0031] the predicted environment feature corresponding to the t+1 moment

[0032] Optionally, before inputting the state information of the t+1 moment and the state information of the t moment into the sliding window sub-model in the curiosity reward model, the method further comprises:

[0033] obtaining each sliding window in the sliding window sub-model; and clustering each sliding window by a k-nearest neighbor clustering algorithm according to the real environment feature contained in each sliding window to obtain a clustered sliding window;

[0034] inputting the state information of the t+1 moment and the state information of the t moment into the sliding window sub-model in the curiosity reward model, comprising:

[0035] inputting the state information of the t+1 moment and the state information of the t moment into the sliding window sub-model in the curiosity reward model, comprising:

[0036] when the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment feature corresponding to the state information of the t moment and the state information of the t+1 moment in the first sliding window of the plurality of sliding windows, outputting the real environment feature corresponding to the t+1 moment, and obtaining a preset first feature weight bias as the feature weight bias, comprising:

[0037] when the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment feature corresponding to the state information of the t moment and the state information of the t+1 moment in the first sliding window of the plurality of sliding windows including the clustered sliding window, outputting the real environment feature corresponding to the t+1 moment, and obtaining a preset first feature weight bias as the feature weight bias.

[0038] In a second aspect, an embodiment of the present application further provides a navigation device, which comprises:

[0039] The navigation module is used to obtain the current state information and target position of the mobile robot in the current environment; the current state information and target position are input into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot can move based on the best executable action;

[0040] The training module is used to obtain the trained action network model through the following sub-modules:

[0041] The predicted action acquisition submodule is used to input the historical state information of the mobile robot in its current environment into the i-th... t The action network model obtained from the training iteration, i-th... t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The predicted action output by the action network model obtained from the training; historical state information is the state information of the mobile robot at each moment in the current environment. The state information includes images marked with location information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position in the current environment to the target position; 1≤t≤N-1, where N is the total number of historical state information.

[0042] The first reward value acquisition submodule is used to input the state information at time t+1, the state information at time t, and the real action corresponding to time t into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1; wherein, the state information at time t+1 is generated by the mobile robot interacting with the current environment based on the real action corresponding to time t.

[0043] The second reward value acquisition submodule is used to input the state information at time t+1 and the state information at time t into the i-th submodule. t The evaluation network model obtained from the training iteration yields the i-th... t The second reward value output by the evaluation network model obtained from the training, which corresponds to the state information at time t+1.

[0044] The update submodule is used to update the i-th reward value based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained from the training iteration are used to obtain the i-th training iteration. t+1 The evaluation network model obtained from the i-th training iteration; and based on the i-th training iteration... t The predicted action output by the action network model obtained from the training iteration and the corresponding real action at time t are used to update the i-th action. tthe action network model obtained by the second training, to obtain the i t+1 the action network model obtained by the second training;

[0045] The updating submodule is further configured to input the action network model obtained by the second training and the evaluation network model obtained by the i t+1 th training into the action network model obtained by the i t+1 th training, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i

[0046] Optionally, the prediction action obtaining submodule is specifically configured to:

[0047] the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and input the state information at the t+1th moment in the historical state information as the state information at the tth moment, and trigger the prediction action obtaining submodule to execute the step of inputting the historical state information of the mobile robot in the current environment into the action network model obtained by the i

[0048] e greed = 0.8-min(10, (i-1) / 10) x θ

[0049] calculate an exploration threshold e greed ; θ is a control factor, 0.01≤θ≤0.08;

[0050] obtain a random number; when the random number is less than the exploration threshold, randomly select any direction as the prediction action output by the action network model obtained by the i t th training;

[0051] when the random number is greater than or equal to the exploration threshold, obtain the reward corresponding to the state information at the tth moment in the historical state information before the i t th training, and input the action that makes the total reward value of the state information at the tth moment maximum as the prediction action output by the action network model obtained by the i t th training.

[0052] Optionally, the first reward value obtaining submodule is specifically configured to:

[0053] inputting the state information at the tth moment and the real action corresponding to the tth moment into a feedforward predictor model in the curiosity reward model to obtain a predicted environment feature corresponding to the (t+1)th moment output by the feedforward predictor model;

[0054] inputting the state information at the tth moment and the state information at the (t+1)th moment into a sliding window sub-model in the curiosity reward model to obtain a real environment feature corresponding to the (t+1)th moment and a feature weight bias output by the sliding window sub-model;

[0055] based on the predicted environment feature corresponding to the (t+1)th moment, the real environment feature corresponding to the (t+1)th moment and the feature weight bias, obtaining a first reward value corresponding to the state information at the (t+1)th moment output by the curiosity reward model.

[0056] Optionally, the first reward value obtaining sub-module is specifically configured to:

[0057] inputting the state information at the tth moment and the state information at the (t+1)th moment into a sliding window sub-model in the curiosity reward model; the sliding window sub-model stores a plurality of sliding windows, and each sliding window includes a plurality of environment features;

[0058] when the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment features corresponding to the state information at the tth moment and the state information at the (t+1)th moment in a first sliding window in the plurality of sliding windows, outputting the real environment feature corresponding to the (t+1)th moment and obtaining a preset first feature weight bias as the feature weight bias; the first feature weight bias is greater than 0.5; the first sliding window is a sliding window in the plurality of sliding windows that includes at least one of the real environment features corresponding to the state information at the tth moment or the state information at the (t+1)th moment;

[0059] based on the predicted environment feature corresponding to the (t+1)th moment the real environment feature corresponding to the (t+1)th moment and the feature weight bias b; through the following formula:

[0060]

[0061] obtaining the first reward value corresponding to the state information at the (t+1)th moment

[0062] Optionally, the training module further includes:

[0063] The clustering submodule is configured to: acquire each sliding window in the sliding window submodule before inputting the state information at the t+1 moment and the state information at the t moment into the sliding window submodule in the curiosity reward model; and perform clustering on each sliding window according to the real environment features contained in each sliding window by using a k-nearest neighbor clustering algorithm, to obtain clustered sliding windows.

[0064] The first reward value obtaining submodule is specifically configured to:

[0065] The state information at the t+1 moment and the state information at the t moment are input into the sliding window submodule in the curiosity reward model and including the clustered sliding windows.

[0066] When the sliding window submodule in the curiosity reward model simultaneously searches for the real environment features corresponding to the state information at the t moment and the state information at the t+1 moment in the first sliding window of the multiple sliding windows including the clustered sliding windows, the real environment features corresponding to the t+1 moment are output, and a preset first feature weight bias is acquired as the feature weight bias.

[0067] In a third aspect, an embodiment of the present application further provides a mobile robot, comprising a processor, a communication interface, a memory, an image sensor, a motor and a communication bus, wherein the processor, the communication interface, the memory, the image sensor and the motor complete mutual communication through the communication bus.

[0068] The memory is configured to store a computer program.

[0069] The image sensor is configured to collect an environment image.

[0070] The motor is configured to rotate based on the optimal executable action motion sent by the processor, so as to move the mobile robot.

[0071] The processor is configured to execute the program stored in the memory, and realize the steps of the navigation method shown in the first aspect.

[0072] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the navigation method shown in the first aspect.

[0073] An embodiment of the present application further provides a computer program product containing instructions, which, when executed on a computer, causes the computer to perform the steps of the navigation method shown in the first aspect.

[0074] An embodiment of the present application has the following beneficial effects:

[0075] The embodiment of the application provides a navigation method and device, a mobile robot and a readable storage medium, wherein the historical state information of the mobile robot in a current environment is input into an action network model obtained through i t times of training, the action network model obtained through i t times of training determines a predicted action output by the action network model obtained through i t times of training according to the number of times that the mobile robot reaches a target position before i t times of training, a preset random number and state information at each moment in the historical state information; the historical state information is state information of the mobile robot at each moment in the current environment, the state information comprises images marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times that the mobile robot moves from an initial position in the current environment to the target position; 1≤t≤N, and N is the total number of the historical state information; the state information at a t+1 moment, the state information at a t moment and the real action corresponding to the t moment are input into a curiosity reward model, so that a first reward value corresponding to the state information at the t+1 moment output by the curiosity reward model is obtained; wherein the state information at the t+1 moment is generated by the mobile robot based on the real action corresponding to the t moment and the current environment; then the state information at the t+1 moment and the state information at the t moment are input into the evaluation network model obtained through i t times of training, so that a second reward value corresponding to the state information at the t+1 moment output by the evaluation network model obtained through i t times of training is obtained; then the model parameters of the evaluation network model obtained through i t times of training are updated based on the first reward value and the second reward value, so that the evaluation network model obtained through i t+1 times of training is obtained; the predicted action output by the action network model obtained through i t times of training and the real action corresponding to the t moment are used to update the model parameters of the action network model obtained through i t times of training, so that the action network model obtained through i t+1 times of training is obtained; finally, the action network model obtained through i t+1 times of training and the evaluation network model obtained through i t+1 times of training are used as the action network model obtained through i t times of training and the evaluation network model obtained through i t times of training; the state information at the t+1 moment in the historical state information is used as the state information at the t moment, and the historical state information of the mobile robot in the current environment is input into the action network model obtained through i t times of training, the action network model obtained through i tThe action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The steps for predicting actions output by the action network model obtained from the previous training are as follows: The above steps are repeated cyclically to obtain a trained action network model. After obtaining the trained action network model, the current state information and target position of the mobile robot in its current environment are obtained. The current state information and target position are then input into the trained action network model to obtain the optimal executable action output by the trained action network model, allowing the mobile robot to move based on the optimal executable action. In this way, as training progresses, the mobile robot comprehensively considers the number of times it reaches the target position and the rewards from the state information at each moment in the historical state information when executing actions, rather than just acting based on curiosity. This reduces the mobile robot's reliance on curiosity and increases its reliance on the number of times it reaches the target position, thereby reducing environmental interference with robot navigation and reducing the difficulty of robot navigation. Of course, implementing any product or method of this invention does not necessarily require achieving all of the above advantages simultaneously. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0077] Figure 1 This is a flowchart illustrating a navigation method according to an embodiment of the present invention;

[0078] Figure 2 This is a schematic diagram of the structure of a navigation device according to an embodiment of the present invention;

[0079] Figure 3 This is a schematic diagram of the structure of a mobile robot according to an embodiment of the present invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0081] In order to solve the problems in the prior art, the embodiment of the present application provides a navigation method and device, a mobile robot and a readable storage medium, so as to reduce the interference of the environment on the robot navigation and reduce the navigation difficulty of the robot.

[0082] Firstly, a navigation method of the embodiment of the present application is introduced as follows, as shown in the figure, it is a flowchart of the navigation method of the embodiment of the present application; the method can include: Figure 1

[0083] S110, input the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, the action network model obtained by the i t th training determines the predicted action output by the action network model obtained by the i t th training according to the number of times that the mobile robot reaches the target position before the i t th training, the preset random number and the state information at each time point in the historical state information; the historical state information is the state information of the mobile robot at each time point in the current environment, the state information includes an image marked with position information, and the state information at each time point has a corresponding real action; i represents the number of times that the mobile robot moves from the initial position in the current environment to the target position; 1≤t≤N-1, and N is the total number of the historical state information;

[0084] S120, input the state information at the t+1 time point, the state information at the t time point and the real action corresponding to the t time point into the curiosity reward model, to obtain the first reward value corresponding to the state information at the t+1 time point and output by the curiosity reward model; wherein the state information at the t+1 time point is generated by the mobile robot based on the real action corresponding to the t time point and the interaction with the current environment;

[0085] S130, input the state information at the t+1 time point and the state information at the t time point into the evaluation network model obtained by the i t th training, to obtain the second reward value corresponding to the state information at the t+1 time point and output by the evaluation network model obtained by the i t th training;

[0086] S140, update the model parameters of the evaluation network model obtained by the i t th training based on the first reward value and the second reward value, to obtain the evaluation network model obtained by the i t+1 th training; and update the predicted action output by the action network model obtained by the i t th training and the real action corresponding to the t time point, to obtain the action network model obtained by the i t ​the action network model obtained in the i-th training, as the action network model obtained in the i-th training, and execute step S190. t+1 the action network model obtained in the i-th training;

[0087] S150, determine whether a training end condition is met, if yes, execute step S160, if no, execute step S170;

[0088] S160, take the action network model obtained in the i-th training as the action network model obtained in the training, and execute step S190. t+1

[0089] S170, take the action network model obtained in the i-th training and the evaluation network model obtained in the i-th training as the action network model obtained in the i-th training and the evaluation network model obtained in the i-th training; t+1 t+1 t t

[0090] S180, take the state information at the t+1-th time point in the historical state information as the state information at the t-th time point, and execute step S110.

[0091] S190, obtain the current state information of the current environment where the mobile robot is located and the target position, input the current state information and the target position into the action network model obtained in the training, obtain the optimal executable action output by the action network model obtained in the training, and make the mobile robot move based on the optimal executable action.

[0092] In some examples, in order to enable the mobile robot to navigate, the mobile robot can be trained in an environment to be navigated, so that the mobile robot can accurately identify the environment to be navigated, i.e. the current environment. It can be understood that the mobile robot can be trained in each new environment to be navigated, and then the mobile robot can be used for navigation after the training is completed.

[0093] When training the mobile robot, the historical data can be used to train the mobile robot, the historical data being the historical state information of the current environment, the historical state information being the state information of the mobile robot at each time point in the current environment, the state information including images marked with position information, and each time point of state information having a corresponding real action.

[0094] ​​​​​In some examples, training the mobile robot involves multiple movements from a starting position to a target position. Each movement from the starting position to the target position constitutes one training iteration. Once a robot has moved from one starting position to the target position, it completes one round of training. This process is repeated multiple times, completing multiple rounds of training. It's understood that a single training round can include multiple training iterations. The robot is considered to have completed one round of training by moving from the initial position to the target position once. Therefore, the i-th... t The training iteration can be considered as the i-th round of training multiplied by the t-th training iteration within the i-th round. When the number of training iterations in each round is the same, then the i-th... t The training iteration is (i-1)×N+t; the number of training iterations in each round is equal to the total number of state information in the historical state information. During the i-th round of training, the mobile robot needs to move from its initial position in the current environment to the target position. Therefore, i can represent the number of times the mobile robot will move from its initial position to the target position; that is, i-1 can represent the number of times the mobile robot has successfully moved from its initial position to the target position.

[0095] In some examples, during the first training iteration, i.e., the first training iteration in the first round, the i-th... t The action network model obtained from the i-th training and the i-th training t The evaluation network models obtained from each training session are both initial action network models and initial evaluation network models. The parameters in these initial action network models and initial evaluation network models can be pre-set parameters.

[0096] During each round of training, steps S110 to S160 can be followed multiple times to move from the initial position to the target position. Multiple rounds of training are then performed. During the training process, it can be determined whether the training termination condition is met. If it is met, the action network model obtained before the step of the determination process is taken as the completed action network model. For example, suppose that before the determination step 140, the i-th... t+1 The i-th training iteration yields the result. t+1 The evaluation network model obtained from the i-th training and the i-th training t+1 The action network model obtained from the training of the i-th time can then be used to train the i-th time network. t+1 The action network model obtained from the second training iteration is considered the completed action network model.

[0097] If the training termination conditions are not met, step S170 can be executed to continue training.

[0098] In some examples, the training end condition can be that the number of training reaches a preset number of training threshold. For example, assuming that the i-th training is the 110th training, and the preset number of training threshold is 110, then the 110th training reaches the preset number of training threshold, where i = 11 and t = 9. t+1

[0099] In some examples, the training end condition can also be that the sum of the first reward value and the second reward value is less than or equal to a preset reward value threshold. Since in the curiosity reward model, the higher the degree of exploration of the mobile robot, the higher the reward value, and the lower the degree of exploration of the mobile robot, the lower the reward value, the higher the degree of exploration of the mobile robot, the greater the possibility of using a new position, and the lower the degree of exploration of the mobile robot, the more familiar the robot is with the current environment, and thus the reward value is lower. At this time, the training can be ended. Therefore, when the sum of the first reward value and the second reward value is less than or equal to the preset reward value threshold, the training can be ended.

[0100] After the training is ended, the mobile robot can move in the current environment by navigation. Specifically, the current state information and the target position of the mobile robot in the current environment can be obtained first, and then the current state information and the target position are input into the trained action network model, and the trained action network model outputs the best executable action, and then the mobile robot can move based on the best executable action.

[0101] It can be understood that steps S120 and S130 can be executed simultaneously, or step S130 can be executed first and then step S120, Figure 1 The order of steps S120 and S130 shown in the above embodiment is only exemplary and is not a limitation on the order of steps S120 and S130. Similarly, steps S170 and S180 can be executed simultaneously, or step S180 can be executed first and then step S170, Figure 1 The order of steps S170 and S180 shown in the above embodiment is only exemplary and is not a limitation on the order of steps S170 and S180.

[0102] Through the navigation method of the embodiment of the present application, the mobile robot can consider the reward of the state information at each time in the historical state information and the number of times of reaching the target position to execute the action during the training, instead of only considering the curiosity-based exploration to execute the action, thereby reducing the dependence of the mobile robot on curiosity and increasing the dependence on the number of times of reaching the target position, and thus reducing the interference of the environment on the navigation of the robot and reducing the difficulty of navigation of the robot.

[0103] In some examples, the i-th training is the 110th training, and the preset number of training threshold is 110. t ​The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t When the action network model obtained from the training of the i-th iteration predicts the action, it can be determined by the i-th iteration. t The action network model obtained from the i-th training iteration is based on the... t The number of times the mobile robot reaches the target position before the training iteration, i-1, is determined by the following formula:

[0104] e greed =0.8-min(10,(i-1) / 10)×θ

[0105] Calculate the exploration threshold e greed θ is a control factor, 0.01≤θ≤0.08;

[0106] Get a random number rand; if the random number rand is less than the exploration threshold, randomly select any direction as the i-th direction. t The predicted action output by the action network model obtained from the training;

[0107] When the random number rand is greater than or equal to the exploration threshold, obtain the i-th... t Before the training iteration, the reward corresponding to the state information at time t in the historical state information is determined, and the action that maximizes the total reward value corresponding to the state information at time t is selected as the i-th action. t The predicted action output by the action network model obtained from the training.

[0108] In some examples, in order to reduce exploration and make more use of historical experience as the training progresses, an exploration threshold can be set. This threshold can decrease as the training progresses, thereby enabling the robot to use actions corresponding to historical state information more as the training progresses.

[0109] Specifically, by using the above formula, it is possible to ensure that as the number of training iterations increases, the i-th... t Before the next training iteration, the number of times the mobile robot reaches the target location (i-1) will also increase, thus causing the exploration threshold to decrease with the increase in the number of training iterations. It is understandable that during the i-th training iteration... t During the i-th training iteration, since it's the i-th training round, the mobile robot might not have reached the target position yet. Therefore, what we obtain here is the i-th... t The number of times the mobile robot reaches the target position before the training iteration is i-1, and i tThe number i-1 of times that the mobile robot reaches the target position before the next training is actually the number of rounds of training that have been completed before the i-th round of training is performed, that is, i-1 rounds of training have been completed.

[0110] That is, at the beginning of training, the exploration threshold is high, at this time, the random number has a high probability of being less than the exploration threshold, therefore, the probability of exploration is also relatively large, and the probability of using the action corresponding to the historical state information is relatively small. As the number of training increases, the exploration threshold gradually decreases, at this time, the probability that the random number is greater than or equal to the exploration threshold increases, and the probability that the random number is less than the exploration threshold decreases, therefore, the probability of exploration also decreases, and the probability of using the action corresponding to the historical state information also increases.

[0111] In some examples, if the random number is less than the exploration threshold, it can be indicated that the mobile robot performs exploration, and any direction can be randomly selected as the i-th t The predicted action output by the action network model obtained through the i-th training; if the random number rand is greater than or equal to the exploration threshold, it can be indicated that the mobile robot uses the action corresponding to the historical state information, at this time, the reward corresponding to the state information at the t-th moment in the historical state information can be obtained, and the action that maximizes the total reward value corresponding to the state information at the t-th moment is used as the predicted action output by the action network model obtained through the i-th training. t The predicted action output by the action network model obtained through the i-th training; if the random number rand is greater than or equal to the exploration threshold, it can be indicated that the mobile robot uses the action corresponding to the historical state information, at this time, the reward corresponding to the state information at the t-th moment in the historical state information can be obtained, and the action that maximizes the total reward value corresponding to the state information at the t-th moment is used as the predicted action output by the action network model obtained through the i-th training. t The predicted action output by the action network model obtained through the i-th training; if the random number rand is greater than or equal to the exploration threshold, it can be indicated that the mobile robot uses the action corresponding to the historical state information, at this time, the reward corresponding to the state information at the t-th moment in the historical state information can be obtained, and the action that maximizes the total reward value corresponding to the state information at the t-th moment is used as the predicted action output by the action network model obtained through the i-th training.

[0112] It can be seen that, through the embodiments of the present application, the mobile robot can use more historical state information and reduce exploration as the training progresses, so that the interference of the environment on the navigation of the robot can be further reduced, and the navigation difficulty of the robot can be reduced.

[0113] In some examples, when the state information at the t+1-th moment, the state information at the t-th moment, and the real action corresponding to the t-th moment are input into the curiosity reward model to obtain the first reward value corresponding to the state information at the t+1-th moment output by the curiosity reward model, the state information at the t-th moment and the real action corresponding to the t-th moment can be first input into the feedforward predictor model in the curiosity reward model to obtain the predicted environmental feature corresponding to the t+1-th moment output by the feedforward predictor model.

[0114] Then, the state information at the t+1-th moment and the state information at the t-th moment are input into the sliding window sub-model in the curiosity reward model to obtain the real environmental feature and the feature weight bias corresponding to the t+1-th moment output by the sliding window sub-model.

[0115] Finally, based on the predicted environment feature corresponding to the t+1th moment, the real environment feature corresponding to the t+1th moment and the feature weight deviation, a first reward value corresponding to the state information of the t+1th moment output by the curiosity reward model is obtained.

[0116] It can be understood that when the mobile robot is at a position, if the action performed at the position is executed, it will inevitably move to the next position, so when the state information of the tth moment and the real action corresponding to the tth moment are input into the feedforward predictor model, the feedforward predictor model can obtain the predicted environment feature corresponding to the t+1th moment; wherein the state information of the tth moment includes the position of the mobile robot at the tth moment, and the predicted environment feature corresponding to the t+1th moment is the feature of the environment of the position of the mobile robot at the t+1th moment.

[0117] In some examples, the feedforward predictor model shares part of the parameters with the action network model, so the feedforward predictor model can copy the model parameters from the action network model after each training to update its own model parameters.

[0118] In yet some examples, when the state information of the t+1th moment and the state information of the tth moment are input into the sliding window sub-model in the curiosity reward model to obtain the real environment feature corresponding to the t+1th moment and the feature weight deviation output by the sliding window sub-model, the state information of the t+1th moment and the state information of the tth moment can be input into the sliding window sub-model in the curiosity reward model; the sliding window sub-model in the curiosity reward model stores a plurality of sliding windows, and each sliding window includes a plurality of environment features.

[0119] Then the sliding window sub-model in the curiosity reward model can find a first sliding window in the plurality of sliding windows, and the first sliding window is a sliding window in the plurality of sliding windows that contains at least one of the following real environment features: the real environment feature corresponding to the state information of the tth moment or the real environment feature corresponding to the state information of the t+1th moment.

[0120] When the sliding window sub-model in the curiosity reward model simultaneously finds the real environment features corresponding to the state information of the tth moment and the state information of the t+1th moment in the first sliding window in the plurality of sliding windows, it outputs the real environment feature corresponding to the t+1th moment and obtains a preset first feature weight deviation as the feature weight deviation corresponding to the t+1th moment; the first feature weight deviation is greater than 0.5.

[0121] In this way, the real environment feature corresponding to the t+1th moment and the feature weight deviation can be obtained.

[0122] In yet some examples, after obtaining the predicted environment feature corresponding to the t+1th moment, the real environment feature corresponding to the t+1th moment, and the feature weight bias, the first reward value corresponding to the state information of the t+1th moment can be calculated based on the predicted environment feature corresponding to the t+1th moment, the real environment feature corresponding to the t+1th moment, and the feature weight bias b according to the following formula: the real environment feature corresponding to the t+1th moment and the feature weight bias b; through the following formula:

[0123]

[0124] the first reward value corresponding to the state information of the t+1th moment

[0125] In some examples, during the exploration navigation process of the robot, a new sliding window is generated every time the robot moves one step, and thus, if each sliding window is stored, a large amount of storage space of the mobile robot will be occupied. If the storage space is not large enough, the mobile robot will delete the old sliding window according to the first-in-first-out principle when storing a new sliding window, thereby causing the mobile robot to have a forgetting problem. Therefore, before inputting the state information of the t+1th moment and the state information of the tth moment into the sliding window sub-model in the curiosity reward model, each sliding window in the sliding window sub-model can be obtained; according to the real environment features contained in each sliding window, the k-nearest neighbor clustering algorithm is used to cluster each sliding window to obtain clustered sliding windows.

[0126] Specifically, during the clustering process, the distance between two adjacent sliding windows can be calculated first, and then a plurality of consecutive sliding windows with a distance less than a preset threshold value are taken as a cluster. After clustering, the cluster center of each cluster is the average value of the sliding windows in the cluster.

[0127] After obtaining the clustered sliding windows, the state information of the t+1th moment and the state information of the tth moment can be input into the sliding window sub-model in the curiosity reward model, which includes the clustered sliding windows.

[0128] When the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment features corresponding to the state information of the tth moment and the state information of the t+1th moment in the first sliding window among the plurality of sliding windows including the clustered sliding windows, the real environment feature corresponding to the t+1th moment is output, and a preset first feature weight bias is obtained as the feature weight bias.

[0129] Understandably, in this sliding window sub-model, multiple sliding windows can all be clustered, and there may also be sliding windows that do not belong to any cluster and cannot be clustered. Therefore, this sliding window sub-model can include both clustered sliding windows and sliding windows that cannot be clustered. This is also possible.

[0130] Corresponding to the above method embodiments, this invention also provides a navigation device, such as... Figure 2 The diagram shown is a structural schematic of a navigation device according to an embodiment of the present invention. The device may include:

[0131] The navigation module 210 is used to acquire the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot can move based on the best executable action;

[0132] Training module 220 is used to obtain the trained action network model through the following sub-modules:

[0133] The predicted action acquisition submodule 2201 is used to input the historical state information of the mobile robot in its current environment into the i-th... t The action network model obtained from the training iteration, i-th... t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The predicted action output by the action network model obtained from the training; historical state information is the state information of the mobile robot at each moment in the current environment. The state information includes images marked with location information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position in the current environment to the target position; 1≤t≤N-1, where N is the total number of historical state information.

[0134] The first reward value acquisition submodule 2202 is used to input the state information at time t+1, the state information at time t, and the real action corresponding to time t into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1; wherein, the state information at time t+1 is generated by the mobile robot interacting with the current environment based on the real action corresponding to time t;

[0135] The second reward value acquisition submodule 2203 is used to input the state information at time t+1 and the state information at time t into the i-th submodule. tThe evaluation network model obtained from the training iterations yields the i-th... t The second reward value output by the evaluation network model obtained from the training, which corresponds to the state information at time t+1.

[0136] Update submodule 2204 is used to update the i-th reward value based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained from the training iteration are used to obtain the i-th training iteration. t+1 The evaluation network model obtained from the i-th training iteration; and based on the i-th training iteration... t The predicted action output by the action network model obtained from the training iteration and the corresponding real action at time t are used to update the i-th action. t The model parameters of the action network model obtained from the training iteration are used to obtain the i-th... t+1 The action network model obtained from the training;

[0137] The update submodule 2204 is also used to update the i-th... t+1 The action network model obtained from the training of the i-th time and the i-th training time t+1 The evaluation network model obtained from the i-th training iteration is used as the i-th... t The action network model obtained from the training of the i-th time and the i-th training time t The evaluation network model obtained from the training; and the submodule 2201 that takes the state information at time t+1 in the historical state information as the state information at time t, and triggers the prediction action acquisition submodule 2201 to input the historical state information of the mobile robot in the current environment into the i-th time module. t The action network model obtained from the training iteration, i-th... t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The steps of the predicted action output by the action network model obtained after training.

[0138] According to a navigation device of this invention, as training progresses, the mobile robot can comprehensively consider the number of times it reaches the target location and the reward of the state information at each moment in the historical state information to perform actions, rather than just performing actions based on curiosity. This reduces the mobile robot's dependence on curiosity and increases its dependence on the number of times it reaches the target location, thereby reducing environmental interference on robot navigation and reducing the difficulty of robot navigation.

[0139] In some instances, the prediction action acquisition submodule 2201 is specifically used for:

[0140] i-th t The action network model obtained from the i-th training iteration is based on the... tThe number of times i-1 that the mobile robot reaches the target position before the (i-1)-th training is obtained by the following formula:

[0141] e greed = 0.8-min(10, (i-1) / 10) x θ

[0142] The exploration threshold e is calculated greed ; θ is a control factor, 0.01≤θ≤0.08;

[0143] A random number is obtained; when the random number is less than the exploration threshold, a random direction is selected as the predicted action output by the action network model obtained through the i-th training; t The predicted action output by the action network model obtained through the i-th training;

[0144] When the random number is greater than or equal to the exploration threshold, the reward corresponding to the state information at the t-th time in the historical state information before the i-th training is obtained, and the action that makes the total reward value corresponding to the state information at the t-th time maximum is taken as the predicted action output by the action network model obtained through the i-th training. t t The predicted action output by the action network model obtained through the i-th training.

[0145] In some examples, the first reward value obtaining submodule 2202 is specifically configured to:

[0146] The state information at the t-th time and the real action corresponding to the t-th time are input into a feedforward predictor model in the curiosity reward model to obtain a predicted environmental feature corresponding to the (t+1)-th time output by the feedforward predictor model;

[0147] The state information at the (t+1)-th time and the state information at the t-th time are input into a sliding window submodule in the curiosity reward model to obtain a real environmental feature corresponding to the (t+1)-th time and a feature weight bias output by the sliding window submodule;

[0148] Based on the predicted environmental feature corresponding to the (t+1)-th time, the real environmental feature corresponding to the (t+1)-th time, and the feature weight bias, a first reward value corresponding to the state information at the (t+1)-th time output by the curiosity reward model is obtained.

[0149] In some examples, the first reward value obtaining submodule 2202 is specifically configured to:

[0150] The state information at the (t+1)-th time and the state information at the t-th time are input into a sliding window submodule in the curiosity reward model; the sliding window submodule stores a plurality of sliding windows, and each sliding window includes a plurality of environmental features;

[0151] ​When the sliding window submodel in the curiosity reward model simultaneously looks up the real environment features corresponding to the state information at the tth moment and the state information at the t+1th moment in a first sliding window of multiple sliding windows, the real environment feature corresponding to the t+1th moment is output, and a preset first feature weight bias is obtained as the feature weight bias; the first feature weight bias is greater than 0.5; the first sliding window is a sliding window containing at least one of the following real environment features in the multiple sliding windows: the real environment feature corresponding to the state information at the tth moment or the real environment feature corresponding to the state information at the t+1th moment;

[0152] based on the predicted environment feature corresponding to the t+1th moment the real environment feature corresponding to the t+1th moment and the feature weight bias b; the first reward value corresponding to the state information at the t+1th moment is calculated by the following formula:

[0153]

[0154]

[0155] Optionally, the training module further comprises:

[0156] a clustering submodule, configured to obtain each sliding window in the sliding window submodel before inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel in the curiosity reward model; and perform clustering on each sliding window by using a k-nearest neighbor clustering algorithm according to the real environment features contained in each sliding window, to obtain clustered sliding windows.

[0157] The first reward value obtaining submodule 2202 is specifically configured to:

[0158] input the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel in the curiosity reward model, which comprises the clustered sliding windows;

[0159] When the sliding window submodel in the curiosity reward model simultaneously looks up the real environment features corresponding to the state information at the tth moment and the state information at the t+1th moment in a first sliding window of multiple sliding windows, the real environment feature corresponding to the t+1th moment is output, and a preset first feature weight bias is obtained as the feature weight bias.

[0160] The embodiment of the present application also provides a mobile robot, such as Figure 3 ​As shown, the mobile robot comprises a processor 301, a communication interface 302, a memory 303, an image sensor 304, a motor 305 and a communication bus 306, wherein the processor 301, the communication interface 302, the memory 303, the image sensor 304 and the motor 305 complete communication with each other through the communication bus 306,

[0161] The memory 303 is used to store a computer program.

[0162] The image sensor 304 is used to collect environment images.

[0163] The motor 305 is used to rotate based on the optimal executable action motion sent by the processor 301, so as to move the mobile robot.

[0164] The processor 301 is used to execute the program stored in the memory 303, and realize the steps of the navigation method shown in any of the above embodiments, for example, the following steps can be executed:

[0165] Obtain the current state information of the mobile robot in the current environment and the target position; input the current state information and the target position into the trained action network model to obtain the optimal executable action output by the trained action network model, so that the mobile robot moves based on the optimal executable action motion.

[0166] The trained action network model is obtained by repeatedly executing the following steps:

[0167] Input the historical state information of the mobile robot in the current environment into the action network model obtained by the i t th training, and the action network model obtained by the i t th training determines the predicted action output by the action network model obtained by the i t th training according to the number of times of the mobile robot reaching the target position before the i t th training, a preset random number and the reward of the state information at each time in the historical state information; the historical state information is the state information of the mobile robot at each time in the current environment, the state information includes images marked with position information, and the state information at each time has a corresponding real action; i represents the number of times of the mobile robot moving from the initial position in the current environment to the target position; 1≤t≤N-1, and N is the total number of historical state information.

[0168] The state information at time t+1, the state information at time t, and the actual action at time t are input into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1. The state information at time t+1 is generated by the mobile robot interacting with the current environment based on the actual action at time t.

[0169] Input the state information at time t+1 and the state information at time t into the i-th... t The evaluation network model obtained from the training iteration yields the i-th... t The second reward value output by the evaluation network model obtained from the training, which corresponds to the state information at time t+1.

[0170] Update the i-th reward value based on the first and second reward values. t The model parameters of the evaluation network model obtained from the training iteration are used to obtain the i-th training iteration. t+1 The evaluation network model obtained from the i-th training iteration; and based on the i-th training iteration... t The predicted action output by the action network model obtained from the training iteration and the corresponding real action at time t are used to update the i-th action. t The model parameters of the action network model obtained from the training iteration are used to obtain the i-th... t+1 The action network model obtained from the training;

[0171] The i-th t+1 The action network model obtained from the training of the i-th time and the i-th training time t+1 The evaluation network model obtained from the i-th training iteration is used as the i-th... t The action network model obtained from the training of the i-th time and the i-th training time t The evaluation network model obtained from the training;

[0172] The state information at time t+1 in the historical state information is used as the state information at time t, and the historical state information of the mobile robot in the current environment is input into the i-th time step. t The action network model obtained from the training iteration, i-th... t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The steps of the predicted action output by the action network model obtained after training.

[0173] Through the mobile robot in the embodiment of the present application, the mobile robot can perform actions by comprehensively considering the reward of the state information at each moment in the historical state information and the number of times of reaching the target position during the training, instead of only performing actions based on curiosity exploration, thereby reducing the dependence of the mobile robot on curiosity, increasing the dependence on the number of times of reaching the target position, and further reducing the interference of the environment on robot navigation and the difficulty of robot navigation.

[0174] The communication bus mentioned in the mobile robot can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0175] The communication interface is used for communication between the mobile robot and other devices.

[0176] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0177] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0178] In another embodiment of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the navigation method shown in any of the above embodiments are implemented, for example, the following steps can be executed:

[0179] The current state information and target position of the mobile robot in its current environment are obtained; the current state information and target position are input into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot can move based on the best executable action;

[0180] The trained action network model is obtained by iteratively executing the following steps:

[0181] Input the historical state information of the mobile robot in its current environment into the i-th... t The action network model obtained from the training iteration, i-th... t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time step is determined by the number of times the mobile robot reaches the target position before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The predicted action output by the action network model obtained from the training; historical state information is the state information of the mobile robot at each moment in the current environment. The state information includes images marked with location information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position in the current environment to the target position; 1≤t≤N-1, where N is the total number of historical state information.

[0182] The state information at time t+1, the state information at time t, and the actual action at time t are input into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1. The state information at time t+1 is generated by the mobile robot interacting with the current environment based on the actual action at time t.

[0183] Input the state information at time t+1 and the state information at time t into the i-th... t The evaluation network model obtained from the training iteration yields the i-th... t The second reward value output by the evaluation network model obtained from the training, which corresponds to the state information at time t+1.

[0184] Update the i-th reward value based on the first and second reward values. t The model parameters of the evaluation network model obtained from the training iteration are used to obtain the i-th training iteration. t+1 The evaluation network model obtained from the i-th training iteration; and based on the i-th training iteration... t The predicted action output by the action network model obtained from the training iteration and the corresponding real action at time t are used to update the i-th action. t The model parameters of the action network model obtained from the training iteration are used to obtain the i-th... t+1 The action network model obtained from the training;

[0185] the action network model obtained by the i t+1 th training and the evaluation network model obtained by the i t+1 th training as the action network model obtained by the i t th training and the evaluation network model obtained by the i t th training.

[0186] the state information at the t+1th time in the historical state information of the mobile robot in the current environment as the state information at the tth time, and performing the step of inputting the historical state information of the mobile robot in the current environment to the action network model obtained by the i t th training, determining the predicted action output by the action network model obtained by the i t th training according to the number of times that the mobile robot reaches the target position before the i t th training, the preset random number and the state information at each time in the historical state information. t th training.

[0187] By the computer readable storage medium provided by the embodiment of the present application, the mobile robot can consider the number of times that the mobile robot reaches the target position and the reward of the state information at each time in the historical state information to perform the action during the training, instead of only performing the action based on the curiosity, thereby reducing the dependence of the mobile robot on the curiosity, increasing the dependence on the number of times that the mobile robot reaches the target position, and further reducing the interference of the environment on the navigation of the robot and the difficulty of the navigation of the robot.

[0188] In another embodiment provided by the present application, a computer program product containing instructions is also provided, which, when running on a computer, causes the computer to perform the steps of the navigation method shown in any of the above embodiments, for example, the following steps can be performed:

[0189] obtaining the current state information of the mobile robot in the current environment and the target position, inputting the current state information and the target position to the trained action network model to obtain the best executable action output by the trained action network model, and causing the mobile robot to move based on the best executable action;

[0190] The trained action network model is obtained by repeatedly performing the following steps:

[0191] inputting the historical state information of the mobile robot in the current environment to the action network model obtained by the i t th training, determining the predicted action output by the action network model obtained by the i t th training according to the number of times that the mobile robot reaches the target position before the i tthe number of times the mobile robot reaches the target position before the second training, a preset random number, and a reward of state information at each time point in the historical state information, determine the prediction action output by the action network model obtained through the i t t t

[0192]

[0193] t t+1 t t t+1

[0194] t t+1 t t t+1

[0195] t+1 t+1 t t

[0196] t t ​​​​​​​​​​​​​The action network model obtained through the secondary training outputs a predicted action. t The number of times the mobile robot reaches the target position, a preset random number, and a reward of state information at each time point in the historical state information before the secondary training are determined. t The action network model obtained through the secondary training outputs a predicted action.

[0197] Through the computer program product including instructions of the embodiment of the present application, the mobile robot can consider the number of times of reaching the target position and the reward of state information at each time point in the historical state information to perform the action during the training, instead of only performing the action based on the curiosity exploration, thereby reducing the dependence of the mobile robot on the curiosity, increasing the dependence on the number of times of reaching the target position, and further reducing the interference of the environment on the robot navigation and the difficulty of the robot navigation.

[0198] In the above embodiment, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be achieved in the form of a computer program product, entirely or partially. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the entire or partial flow or function shown in the embodiment of the present application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0199] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other present or future devices, systems, methods, compositions of matter or combinations of matters can be constructed to practice the present application. It must be noted that, as used in this specification and the appended claims, the singular form "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" includes a plurality of such components and reference to "the element" is a reference to one or more elements and equivalents thereof. All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs unless otherwise specifically defined. The disclosures of each patent, patent application, and publication cited herein are hereby incorporated herein by reference in their entirety.

[0200] Each of the embodiments in the present specification is described in a relevant manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, mobile robot, readable storage medium, and computer program product containing instructions, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0201] The preferred embodiments of the present application are described above with reference to the drawings, but the present application is not limited to the above examples. Any modification, equivalent replacement, improvement, and the like within the spirit and principle of the present application are included in the scope of the present application.

Claims

1. A navigation method, characterized in that, The method includes: The current state information and target position of the mobile robot in its current environment are obtained; the current state information and the target position are input into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action; The trained action network model is obtained by repeatedly executing the following steps: The historical state information of the mobile robot in its current environment is input into the i-th... t The action network model obtained from the training of the i-th training iteration, t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time is determined by the number of times the mobile robot reaches the target location before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The predicted action output by the action network model obtained from the training; the historical state information is the state information of the mobile robot at each moment in the current environment, the state information includes an image marked with location information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position in the current environment to the target position; 1≤t≤N-1, where N is the total number of historical state information; The state information at time t+1, the state information at time t, and the actual action corresponding to time t are input into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1; wherein, the state information at time t+1 is generated by the mobile robot interacting with the current environment based on the actual action corresponding to time t. Input the state information at time t+1 and the state information at time t into the i-th... t The evaluation network model obtained from the training iterations yields the i-th... t The second reward value output by the evaluation network model obtained from the training, which corresponds to the state information at the (t+1)th time; Update the i-th reward value based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained from the training iteration are used to obtain the i-th training iteration. t+1 The evaluation network model obtained from the i-th training; and based on the i-th training... t The predicted action output by the action network model obtained from the training iteration and the actual action corresponding to the t-th time step are used to update the i-th time step. t The model parameters of the action network model obtained from the training iteration are used to obtain the i-th... t+1 The action network model obtained from the training; The i-th t+1 The action network model obtained from the training of the i-th time and the i-th training time t+1 The evaluation network model obtained from the i-th training iteration is used as the i-th... t The action network model obtained from the training of the i-th time and the i-th training time t The evaluation network model obtained from the training; The state information at time t+1 in the historical state information is used as the state information at time t, and the process of inputting the historical state information of the mobile robot in the current environment into the i-th time step is executed. t The action network model obtained from the training of the i-th training iteration, t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time is determined by the number of times the mobile robot reaches the target location before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The steps of the predicted action output by the action network model obtained after training.

2. The method according to claim 1, characterized in that, The i-th t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time is determined by the number of times the mobile robot reaches the target location before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The predicted actions output by the action network model obtained from this training include: The i-th t The action network model obtained from the i-th training iteration is based on the... t The number of times the mobile robot reaches the target location before the training iteration, i-1, is determined by the following formula: e greed =0.8-min(10,(i-1) / 10)×θ Calculate the exploration threshold e greed ; The θ is a control factor, 0.01≤θ≤0.08; Obtain a random number; when the random number is less than the exploration threshold, randomly select any direction as the i-th direction. t The predicted action output by the action network model obtained from the training; When the random number is greater than or equal to the exploration threshold, obtain the i-th... t Before the training iteration, the reward corresponding to the state information at time t in the historical state information is used, and the action that maximizes the total reward value corresponding to the state information at time t is taken as the i-th action. t The predicted action output by the action network model obtained from the training.

3. The method according to claim 1, characterized in that, The process of inputting the state information at time t+1, the state information at time t, and the actual action corresponding to time t into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1 includes: The state information at time t and the actual action corresponding to time t are input into the feedforward prediction sub-model in the curiosity reward model to obtain the predicted environment features corresponding to time t+1 output by the feedforward prediction sub-model. The state information at time t+1 and the state information at time t are input into the sliding window sub-model in the curiosity reward model to obtain the real environment features and feature weight deviations output by the sliding window sub-model corresponding to time t+1. Based on the predicted environment features corresponding to the (t+1)th time, the real environment features corresponding to the (t+1)th time, and the feature weight deviation, the first reward value output by the curiosity reward model corresponding to the state information at the (t+1)th time is obtained.

4. The method according to claim 3, characterized in that, The step of inputting the state information at time t+1 and the state information at time t into the sliding window sub-model in the curiosity reward model to obtain the deviation of the real environment features and feature weights output by the sliding window sub-model from those at time t+1 includes: The state information at time t+1 and the state information at time t are input into the sliding window sub-model in the curiosity reward model; the sliding window sub-model stores multiple sliding windows, each of which includes multiple environmental features; When the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment features corresponding to the state information at time t and the state information at time t+1 in the first sliding window of the plurality of sliding windows, it outputs the real environment features corresponding to time t+1 and obtains a preset first feature weight deviation as the feature weight deviation; the first feature weight deviation is greater than 0.5; the first sliding window is a sliding window among the plurality of sliding windows that contains at least one of the following real environment features: the real environment feature corresponding to the state information at time t or the real environment feature corresponding to the state information at time t+1; The process of obtaining the first reward value output by the curiosity reward model based on the predicted environment features at time t+1, the actual environment features at time t+1, and the feature weight deviation includes: Based on the predicted environmental features corresponding to the (t+1)th time. The real environment features corresponding to the (t+1)th time. And the feature weight deviation b; expressed by the following formula: The first reward value corresponding to the state information at time t+1 is calculated.

5. The method according to claim 4, characterized in that, Before inputting the state information at time t+1 and the state information at time t into the sliding window sub-model in the curiosity reward model, the method further includes: Obtain each sliding window in the sliding window sub-model; based on the real environment features contained in each sliding window, cluster each sliding window using the k-nearest neighbor clustering algorithm to obtain the clustered sliding windows; The step of inputting the state information at time t+1 and the state information at time t into the sliding window sub-model in the curiosity reward model includes: The state information at time t+1 and the state information at time t are input into the sliding window sub-model of the curiosity reward model, which includes the clustered sliding window; When the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment features corresponding to the state information at time t and the state information at time t+1 in the first sliding window of the plurality of sliding windows, it outputs the real environment features corresponding to time t+1 and obtains a preset first feature weight deviation as the feature weight deviation, including: When the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment features corresponding to the state information at time t and the state information at time t+1 in the first sliding window of multiple sliding windows including the clustered sliding window, it outputs the real environment features corresponding to time t+1 and obtains the preset first feature weight deviation as the feature weight deviation.

6. A navigation device, characterized in that, The device includes: The navigation module is used to acquire the current state information and target position of the mobile robot in its current environment; input the current state information and the target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action; The training module is used to obtain the trained action network model through the following sub-modules: The predicted action acquisition submodule is used to input the historical state information of the mobile robot in the current environment into the i-th... t The action network model obtained from the training of the i-th training iteration, t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time is determined by the number of times the mobile robot reaches the target location before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The predicted action output by the action network model obtained from the training; the historical state information is the state information of the mobile robot at each moment in the current environment, the state information includes an image marked with location information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position in the current environment to the target position; 1≤t≤N-1, where N is the total number of historical state information; The first reward value acquisition submodule is used to input the state information at time t+1, the state information at time t, and the actual action corresponding to time t into the curiosity reward model to obtain the first reward value output by the curiosity reward model corresponding to the state information at time t+1; wherein, the state information at time t+1 is generated by the mobile robot interacting with the current environment based on the actual action corresponding to time t; The second reward value acquisition submodule is used to input the state information at time t+1 and the state information at time t into the i-th submodule. t The evaluation network model obtained from the training iterations yields the i-th... t The second reward value output by the evaluation network model obtained from the training, which corresponds to the state information at the (t+1)th time; The update submodule is used to update the i-th reward value based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained from the training iteration are used to obtain the i-th training iteration. t+1 The evaluation network model obtained from the i-th training; and based on the i-th training... t The predicted action output by the action network model obtained from the training iteration and the actual action corresponding to the t-th time step are used to update the i-th time step. t The model parameters of the action network model obtained from the training iteration are used to obtain the i-th... t+1 The action network model obtained from the training; The update submodule is also used to update the i-th... t+1 The action network model obtained from the training of the i-th time and the i-th training time t+1 The evaluation network model obtained from the i-th training iteration is used as the i-th... t The action network model obtained from the training of the i-th time and the i-th training time t The evaluation network model obtained from the training iterations; and the state information at time t+1 in the historical state information as the state information at time t, triggering the prediction action acquisition submodule to execute the step of inputting the historical state information of the mobile robot in the current environment to the i-th time step. t The action network model obtained from the training of the i-th training iteration, t The action network model obtained from the i-th training iteration is based on the... t The reward for the i-th time is determined by the number of times the mobile robot reaches the target location before the training iteration, the preset random number, and the state information at each moment in the historical state information. t The steps of the predicted action output by the action network model obtained after training.

7. The apparatus according to claim 6, characterized in that, The prediction action acquisition submodule is specifically used for: The i-th t The action network model obtained from the i-th training iteration is based on the... t The number of times the mobile robot reaches the target location before the training iteration, i-1, is determined by the following formula: e greed =0.8-min(10,(i-1) / 10)×θ Calculate the exploration threshold e greed ; The θ is a control factor, 0.01≤θ≤0.08; Get random numbers; When the random number is less than the exploration threshold, any direction is randomly selected as the i-th direction. t The predicted action output by the action network model obtained from the training; When the random number is greater than or equal to the exploration threshold, obtain the i-th... t Before the training iteration, the reward corresponding to the state information at time t in the historical state information is used, and the action that maximizes the total reward value corresponding to the state information at time t is taken as the i-th action. t The predicted action output by the action network model obtained from the training.

8. The apparatus according to claim 6, characterized in that, The first reward value acquisition submodule is specifically used for: The state information at time t and the actual action corresponding to time t are input into the feedforward prediction sub-model in the curiosity reward model to obtain the predicted environment features corresponding to time t+1 output by the feedforward prediction sub-model. The state information at time t+1 and the state information at time t are input into the sliding window sub-model in the curiosity reward model to obtain the real environment features and feature weight deviations output by the sliding window sub-model corresponding to time t+1. Based on the predicted environment features corresponding to the (t+1)th time, the real environment features corresponding to the (t+1)th time, and the feature weight deviation, the first reward value output by the curiosity reward model corresponding to the state information at the (t+1)th time is obtained.

9. A mobile robot, characterized in that, It includes a processor, a communication interface, a memory, an image sensor, a motor, and a communication bus, wherein the processor, the communication interface, the memory, the image sensor, and the motor communicate with each other through the communication bus; The memory is used to store computer programs; The image sensor is used to acquire environmental images; The motor is used to rotate based on the optimal executable motion sent by the processor, so as to move the mobile robot; When the processor executes the program stored in the memory, it implements the steps of the method described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-5.

Images