Navigation method and device, mobile robot and readable storage medium

Abstract

The embodiment of the invention provides a navigation method and device, a mobile robot and a readable storage medium. The method comprises the following steps that: a mobile robot obtains a target position according to the number of times that the mobile robot arrives at the target position before each training, a preset random number, an award of state information of the mobile robot at each moment in historical state information of a current environment, the state information of the mobile robot at each moment in the historical state information and a corresponding real action; a preset action network model is trained to obtain a trained action network model, and then the trained action network model outputs an optimal executable action based on the current state information of the mobile robot in the current environment and the target position, so that the mobile robot moves based on the optimal executable action; therefore, the dependence of the mobile robot on curiosity can be reduced, the dependence on the number of times of arriving at the target position is increased, the interference of the environment on robot navigation is reduced, and the navigation difficulty of the robot is reduced.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to a navigation method, device, mobile robot and readable storage medium. Background Art

[0002] Mobile robots are widely used in various industries such as industry, hotels, security, and hospitals, and have a wide range of applications. Accurate autonomous navigation is the key to mobile robots, and exploratory navigation in unknown areas is still a challenge for robots. Navigation methods based on reinforcement learning algorithms have shown excellent exploration success rate and navigation performance in recent studies. Robots learn strategies to maximize the total number of future rewards by interacting with environmental states, so that robots can achieve the most effective action behaviors. Setting different reward mechanisms in the same navigation task will also affect the efficiency of navigation and the accuracy of the results.

[0003] There are currently two methods for setting intrinsic rewards for reinforcement learning: one method is to drive robot exploration by designing different action behaviors of the robot. Some people use reward shaping functions to improve the robot's optimal behavior strategy. The other method drives robot navigation and exploration by evaluating the state of the environment. This method encourages the robot to explore unknown areas and assigns reward weights by calculating the robot's curiosity about the new exploration area.

[0004] However, the applicant found that in the prior art, when the passageway for the robot to pass through in the scene where the robot is used does not change, but only the decoration or decoration around the passageway changes, the robot may mistakenly judge the location that has been explored as a new location. For example, there is a TV in the environment. Since the TV switches the screen regularly, the screen is different every time the robot visits this location. At this time, the robot will judge the location of the TV as a new location instead of the previously recorded location that has been explored, so that the robot always regards the location of the TV as a new location, which causes the robot to fall into a deadlock state. It can be seen that when the environment in the scene where the robot is used changes frequently, it will increase the interference of the environment on the robot's navigation and increase the difficulty of the robot's navigation. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a navigation method, device, mobile robot and readable storage medium to reduce the interference of the environment on the robot's navigation and reduce the navigation difficulty of the robot. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a navigation method, the method comprising:

[0007] Obtain the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action;

[0008] The trained action network model is obtained by cyclically executing the following steps:

[0009] Input the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information;

[0010] The state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment are input into the curiosity reward model to obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment;

[0011] Input the state information at the t+1th moment and the state information at the tth moment into the i-th t The evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training;

[0012] Update the i-th reward value based on the first reward value and the second reward value t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training;

[0013] The i t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by training;

[0014] The state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the historical state information of the mobile robot in the current environment is input into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The steps of predicting actions output by the action network model trained this time.

[0015] Optional, first t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted actions output by the action network model obtained through training include:

[0016] i-th t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before training i-1 is calculated using the following formula:

[0017] e greed =0.8-min(10,(i-1) / 10)×θ

[0018] Calculate the exploration threshold e greed ; θ is the control factor, 0.01≤θ≤0.08;

[0019] Get a random number; when the random number is less than the exploration threshold, randomly select any direction as the i-th direction. t The predicted action output by the action network model obtained through training;

[0020] When the random number is greater than or equal to the exploration threshold, get the i-th t Before the training, the reward corresponding to the state information at the tth moment in the historical state information is calculated, and the action that maximizes the total reward value corresponding to the state information at the tth moment is taken as the i-th action. tThe predicted action output by the action network model trained this time.

[0021] Optionally, the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment are input into the curiosity reward model to obtain a first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model, including:

[0022] The state information at the tth moment and the real action corresponding to the tth moment are input into the feedforward prediction submodel in the curiosity reward model, and the predicted environment features corresponding to the t+1th moment output by the feedforward prediction submodel are obtained;

[0023] The state information at the t+1th moment and the state information at the tth moment are input into the sliding window sub-model in the curiosity reward model, and the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model are obtained;

[0024] Based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment, and the feature weight deviation, a first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model is obtained.

[0025] Optionally, the state information at the t+1th moment and the state information at the tth moment are input into the sliding window sub-model in the curiosity reward model to obtain the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model, including:

[0026] The state information at the t+1th moment and the state information at the tth moment are input into the sliding window submodel in the curiosity reward model; the sliding window submodel stores a plurality of sliding windows, each of which includes a plurality of environmental features;

[0027] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to the state information at the t-th moment and the state information at the t+1-th moment in the first sliding window of the multiple sliding windows, the real environment features corresponding to the t+1-th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation; the first feature weight deviation is greater than 0.5; the first sliding window is a sliding window among the multiple sliding windows that contains at least one of the following real environment features: the real environment feature corresponding to the state information at the t-th moment or the real environment feature corresponding to the state information at the t+1-th moment;

[0028] Based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment, and the feature weight deviation, the first reward value output by the curiosity reward model is obtained, including:

[0029] Based on the predicted environmental features corresponding to the t+1th moment The real environment characteristics corresponding to the t+1th moment And feature weight bias b; through the following formula:

[0030]

[0031] Calculate the first reward value corresponding to the state information at the t+1th moment

[0032] Optionally, before inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window sub-model in the curiosity reward model, the method further includes:

[0033] Obtain each sliding window in the sliding window sub-model; cluster each sliding window by using a k-nearest neighbor clustering algorithm according to the real environment features contained in each sliding window to obtain a clustered sliding window;

[0034] The state information at the t+1th moment and the state information at the tth moment are input into the sliding window sub-model in the curiosity reward model, including:

[0035] Input the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel including the clustered sliding window in the curiosity reward model;

[0036] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to the state information at the t-th moment and the state information at the t+1-th moment in the first sliding window of the multiple sliding windows, the real environment features corresponding to the t+1-th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation, including:

[0037] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to state information at the tth moment and state information at the t+1th moment in the first sliding window of multiple sliding windows including the clustered sliding windows, the real environment features corresponding to the t+1th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation.

[0038] In a second aspect, an embodiment of the present invention further provides a navigation device, the device comprising:

[0039] The navigation module is used to obtain the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action;

[0040] The training module is used to obtain the trained action network model through the following submodules:

[0041] The predicted action acquisition submodule is used to input the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information;

[0042] A first reward value obtaining submodule is used to input the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment into the curiosity reward model, and obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment;

[0043] The second reward value acquisition submodule is used to input the state information at the t+1th moment and the state information at the tth moment into the i-th moment. t The evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training;

[0044] An updating submodule, configured to update the i-th reward value based on the first reward value and the second reward value t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated tThe model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training;

[0045] Update submodules and also to update the i t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by training for the first time; and the state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the prediction action acquisition submodule is triggered to execute the historical state information of the mobile robot in the current environment input into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The steps of predicting actions output by the action network model trained this time.

[0046] Optionally, the predicted action acquisition submodule is used to:

[0047] i-th t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before training i-1 is calculated using the following formula:

[0048] e greed =0.8-min(10,(i-1) / 10)×θ

[0049] Calculate the exploration threshold e greed ; θ is the control factor, 0.01≤θ≤0.08;

[0050] Get a random number; when the random number is less than the exploration threshold, randomly select any direction as the i-th direction. t The predicted action output by the action network model obtained through training;

[0051] When the random number is greater than or equal to the exploration threshold, get the i-th t Before the training, the reward corresponding to the state information at the tth moment in the historical state information is calculated, and the action that maximizes the total reward value corresponding to the state information at the tth moment is taken as the i-th action. t The predicted action output by the action network model trained this time.

[0052] Optionally, the first reward value obtaining submodule is specifically used to:

[0053] The state information at the tth moment and the real action corresponding to the tth moment are input into the feedforward prediction submodel in the curiosity reward model, and the predicted environment features corresponding to the t+1th moment output by the feedforward prediction submodel are obtained;

[0054] The state information at the t+1th moment and the state information at the tth moment are input into the sliding window sub-model in the curiosity reward model, and the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model are obtained;

[0055] Based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment, and the feature weight deviation, a first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model is obtained.

[0056] Optionally, the first reward value obtaining submodule is specifically used to:

[0057] The state information at the t+1th moment and the state information at the tth moment are input into the sliding window submodel in the curiosity reward model; the sliding window submodel stores a plurality of sliding windows, each of which includes a plurality of environmental features;

[0058] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to the state information at the t-th moment and the state information at the t+1-th moment in the first sliding window of the multiple sliding windows, the real environment features corresponding to the t+1-th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation; the first feature weight deviation is greater than 0.5; the first sliding window is a sliding window among the multiple sliding windows that contains at least one of the following real environment features: the real environment feature corresponding to the state information at the t-th moment or the real environment feature corresponding to the state information at the t+1-th moment;

[0059] Based on the predicted environmental features corresponding to the t+1th moment The real environment characteristics corresponding to the t+1th moment And feature weight bias b; through the following formula:

[0060]

[0061] Calculate the first reward value corresponding to the state information at the t+1th moment

[0062] Optionally, the training module also includes:

[0063] The clustering submodule is used to obtain each sliding window in the sliding window submodel before inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel in the curiosity reward model; cluster each sliding window by a k-nearest neighbor clustering algorithm according to the real environment features contained in each sliding window to obtain a clustered sliding window;

[0064] The first reward value obtaining submodule is specifically used for:

[0065] Input the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel including the clustered sliding window in the curiosity reward model;

[0066] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to state information at the tth moment and state information at the t+1th moment in the first sliding window of multiple sliding windows including the clustered sliding windows, the real environment features corresponding to the t+1th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation.

[0067] In a third aspect, an embodiment of the present invention further provides a mobile robot, comprising a processor, a communication interface, a memory, an image sensor, a motor and a communication bus, wherein the processor, the communication interface, the memory, the image sensor and the motor communicate with each other via the communication bus;

[0068] Memory, used to store computer programs;

[0069] An image sensor, used to collect environmental images;

[0070] The motor is configured to rotate based on the best executable action motion sent by the processor to move the mobile robot;

[0071] The processor is used to implement the steps of the navigation method shown in the first aspect when executing the program stored in the memory.

[0072] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the navigation method shown in the first aspect are implemented.

[0073] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the navigation method shown in the first aspect above.

[0074] Beneficial effects of the embodiments of the present invention:

[0075] The embodiment of the present invention provides a navigation method, a device, a mobile robot and a readable storage medium, firstly inputting the historical state information of the mobile robot in the current environment into the i t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N, N is the total number of historical state information; then the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment are input into the curiosity reward model to obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein, the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment; then the state information at the t+1th moment and the state information at the tth moment are input into the i-th t The evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training; and then updating the i-th moment based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training; finally, the i-th t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by training; the state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the historical state information of the mobile robot in the current environment is input into the i-th t The action network model obtained by training, tThe action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The steps of predicting the action output by the action network model obtained by training for the first time; obtaining the trained action network model by executing the above steps in a loop, and then obtaining the current state information and target position of the mobile robot in the current environment after obtaining the trained action network model; and inputting the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action. In this way, as the training proceeds, the mobile robot can perform actions by comprehensively considering the number of times the target position is reached and the rewards of the state information at each moment in the historical state information, rather than just performing actions based on curiosity exploration, thereby reducing the mobile robot's dependence on curiosity and increasing its dependence on the number of times the target position is reached, thereby reducing the interference of the environment on the robot's navigation and reducing the difficulty of the robot's navigation. Of course, implementing any product or method of the present invention does not necessarily require all of the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0077] Figure 1 A flowchart of a navigation method according to an embodiment of the present invention;

[0078] Figure 2 is a structural schematic diagram of a navigation device according to an embodiment of the present invention;

[0079] Figure 3 The figure is a schematic diagram of the structure of a mobile robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0081] In order to solve the problems existing in the prior art, the embodiments of the present invention provide a navigation method, an apparatus, a mobile robot and a readable storage medium, so as to reduce the interference of the environment on the robot's navigation and reduce the navigation difficulty of the robot.

[0082] Below, a navigation method according to an embodiment of the present invention is first introduced. Figure 1 FIG. 1 is a flowchart of a navigation method according to an embodiment of the present invention; the method may include:

[0083] S110, inputting the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information;

[0084] S120, inputting the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment into the curiosity reward model, and obtaining a first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein the state information at the t+1th moment is generated by the mobile robot interacting with the current environment based on the real action corresponding to the tth moment;

[0085] S130, input the state information at the t+1th moment and the state information at the tth moment into the i-th t The evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training;

[0086] S140, updating the i-th reward value based on the first reward value and the second reward value t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated tThe model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training;

[0087] S150, determining whether the training end condition is met, if yes, executing step S160, if no, executing step S170;

[0088] S160, the i t+1 The action network model obtained by the training is used as the trained action network model; and step S190 is executed.

[0089] S170, the i t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by training;

[0090] S180, taking the state information at the t+1th moment in the historical state information as the state information at the tth moment, and executing step S110.

[0091] S190, obtain the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model, and obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action.

[0092] In some examples, in order to use the mobile robot for navigation, the mobile robot may be trained in an environment to be navigated so that the mobile robot can accurately identify the environment to be navigated, which is also the current environment. It is understandable that each time the mobile robot arrives at a new environment to be navigated, it may be trained in the environment to be navigated, and the mobile robot may be used for navigation after the training is completed.

[0093] When training the mobile robot, the mobile robot may be trained first using historical data, where the historical data is historical state information of the current environment, where the historical state information is state information of the mobile robot at each moment in the current environment, where the state information includes an image marked with position information, and where the state information at each moment has a corresponding real action;

[0094] In some examples, when training the mobile robot, the mobile robot needs to move from the starting position to the target position multiple times in order to train the mobile robot, and each time from the starting position to the target position, each time it moves to a position, it is trained once. When it moves from a starting position to a target position, the mobile robot completes a round of training, and then continues to move from a starting position to the target position multiple times, thereby completing multiple rounds of training. It can be understood that a round of training can include multiple trainings. The mobile robot is considered to have completed a round of training when it reaches the target position once from the initial position. Therefore, the above i-th t The training can be considered as the i-th round of training multiplied by the t-th training in the i-th round of training. When the number of training rounds is the same, the i-th round of training is t The number of training rounds is (i-1)×N+t; the number of each training round is the total number of state information in the historical state information. When performing the i-th round of training, the mobile robot needs to move from the initial position of the current environment to the target position. Therefore, i can represent the number of times the mobile robot will move from the initial position of the current environment to the target position; that is, i-1 can represent the number of times the mobile robot has successfully moved from the initial position of the current environment to the target position.

[0095] In some examples, when performing the first training, that is, the first training of the first round, the i-th t The action network model obtained by training and the i-th t The evaluation network models obtained by the training are both initial action network models and initial evaluation network models, and the parameters in the initial action network model and the initial evaluation network model can be pre-set parameters.

[0096] In each round of training, multiple trainings can be performed according to steps S110 to S160 to move from the initial position to the target position. Then, multiple rounds of training are performed. During the training process, it can be determined whether the training end condition is met. If it is met, the action network model obtained before the step of executing the judgment process is used as the action network model that has completed the training. For example, assuming that before executing the judgment step 140, the i-th action network model is performed. t+1 The training result is the i-th t+1 The evaluation network model obtained by training and the i-th t+1 The action network model obtained by training can be used to t+1 The action network model obtained through training is used as the action network model that has completed training.

[0097] If the training end condition is not met, step S170 may be executed to continue the training.

[0098] In some examples, the training end condition may be that the number of training times reaches a preset training times threshold. t+1 The training is the 110th training, and the preset training times threshold is 110, then the 110th training reaches the preset training times threshold, wherein i=11, t=9.

[0099] In some examples, the training end condition may also be that the sum of the first reward value and the second reward value is less than or equal to a preset reward value threshold. In the curiosity reward model, if the mobile robot explores more, the reward value is higher, and if the mobile robot explores less, the reward value is lower. When the mobile robot explores more, the possibility of using a new location is greater. When the mobile robot explores less, it means that the robot is already familiar with the current environment, so the reward value will be lower. At this time, the training can be ended. Therefore, when the sum of the first reward value and the second reward value is less than or equal to the preset reward value threshold, the training can be ended.

[0100] After the training is completed, the mobile robot can move in the current environment through navigation. Specifically, the current state information and target position of the mobile robot in the current environment can be obtained first, and then the current state information and target position can be input into the trained action network model. The trained action network model outputs the best executable action, and the mobile robot can move based on the best executable action.

[0101] It is understandable that step S120 and step S130 may be performed simultaneously, or step S130 may be performed first and then step S120. Figure 1 The order of steps S120 and S130 shown in the figure is only an exemplary description and does not limit the order of steps S120 and S130. Similarly, step S170 and step S180 can be executed at the same time, or step S180 can be executed first and then S170. Figure 1 The order of steps S170 and S180 shown in the figure is only an exemplary description and does not limit the order of steps S170 and S180.

[0102] Through a navigation method of an embodiment of the present invention, as training progresses, the mobile robot can comprehensively consider the number of times it reaches the target position and the rewards of the state information at each moment in the historical state information to perform actions, rather than just performing actions based on curiosity exploration, thereby reducing the mobile robot's dependence on curiosity and increasing its dependence on the number of times it reaches the target position, thereby reducing the interference of the environment on the robot's navigation and reducing the robot's navigation difficulty.

[0103] In some examples, in the i tThe action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t When the predicted action output by the action network model obtained by training for the first time is t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before training i-1 is calculated using the following formula:

[0104] e greed =0.8-min(10,(i-1) / 10)×θ

[0105] Calculate the exploration threshold e greed ; θ is the control factor, 0.01≤θ≤0.08;

[0106] Get the random number rand; when the random number rand is less than the exploration threshold, randomly select any direction as the i-th direction. t The predicted action output by the action network model obtained through training;

[0107] When the random number rand is greater than or equal to the exploration threshold, get the i-th t Before the training, the reward corresponding to the state information at the tth moment in the historical state information is calculated, and the action that maximizes the total reward value corresponding to the state information at the tth moment is taken as the i-th action. t The predicted action output by the action network model trained this time.

[0108] In some examples, in order to reduce the exploration of the mobile robot and make more use of historical experience as the training times progress, an exploration threshold can be set. The exploration threshold can become smaller as the training times progress, so that as the training times progress, more actions corresponding to historical state information can be used.

[0109] Specifically, by using the above formula, as the number of training times increases, the i-th t The number of times the mobile robot reaches the target position before the i-th training will also increase, so that the exploration threshold will decrease with the increase of the number of training times. It can be understood that t During the training, since it is the i-th round of training, the mobile robot may not have reached the target position during the i-th round of training. Therefore, the i-th round of training is obtained here. t The number of times the mobile robot reaches the target position before training i-1, and i tThe number of times the mobile robot reaches the target position before the i-th training is actually the number of training rounds that have been completed before the i-th training, that is, the i-1-th round of training has been completed.

[0110] That is to say, at the beginning of training, the exploration threshold is high. At this time, the random number has a greater probability of being less than the exploration threshold. Therefore, the probability of exploration will be relatively high, and the probability of using the action corresponding to the historical state information will be relatively low. As the number of training times increases, the exploration threshold will gradually decrease. At this time, the probability of the random number being greater than or equal to the exploration threshold will increase, and the probability of the random number being less than the exploration threshold will decrease. Therefore, the probability of exploration will also decrease, and the probability of using the action corresponding to the historical state information will also increase.

[0111] In some examples, if the random number is less than the exploration threshold, it can be represented as a mobile robot exploring, and any direction can be randomly selected as the i-th direction. t The predicted action output by the action network model obtained by training; if the random number rand is greater than or equal to the exploration threshold, it can indicate that the mobile robot is going to use the action corresponding to the historical loading. At this time, the i-th t Before the training, the reward corresponding to the state information at the tth moment in the historical state information is calculated, and the action that maximizes the total reward value corresponding to the state information at the tth moment is taken as the i-th action. t The predicted action output by the action network model trained this time.

[0112] It can be seen that, through the embodiments of the present invention, as the training proceeds, the mobile robot can use more historical state information and reduce exploration, thereby further reducing the interference of the environment on the robot's navigation and reducing the navigation difficulty of the robot.

[0113] In some examples, when the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment are input into the curiosity reward model to obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model, the state information at the tth moment and the real action corresponding to the tth moment can be first input into the feedforward prediction sub-model in the curiosity reward model to obtain the predicted environmental features corresponding to the t+1th moment output by the feedforward prediction sub-model;

[0114] Then, the state information at the t+1th moment and the state information at the tth moment are input into the sliding window sub-model in the curiosity reward model, and the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model are obtained;

[0115] Finally, based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment, and the feature weight deviation, the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model is obtained.

[0116] It can be understood that when the mobile robot is at a position, if it performs an action at that position, it will inevitably move to the next position. Therefore, when the state information at the tth moment and the actual action corresponding to the tth moment are input into the feedforward prediction sub-model, the feedforward prediction sub-model can obtain the predicted environmental features corresponding to the t+1th moment; wherein, the state information at the tth moment includes the position of the mobile robot at the tth moment, and the predicted environmental features corresponding to the t+1th moment are the predicted features of the environment at the position of the mobile robot at the t+1th moment.

[0117] In some examples, the feedforward prediction sub-model shares some parameters with the action network model, so the feedforward prediction sub-model can copy the model parameters from the action network model after each training to update its own model parameters.

[0118] In some other examples, when the state information at the t+1th moment and the state information at the tth moment are input into the sliding window sub-model in the curiosity reward model, and the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model are obtained, the state information at the t+1th moment and the state information at the tth moment can be input into the sliding window sub-model in the curiosity reward model; the sliding window sub-model stores a plurality of sliding windows, each of which includes a plurality of environment features;

[0119] Then, the sliding window submodel in the curiosity reward model may search for a first sliding window in the multiple sliding windows, where the first sliding window is a sliding window in the multiple sliding windows that contains at least one of the following real environment features: a real environment feature corresponding to the state information at the t-th moment or a real environment feature corresponding to the state information at the t+1-th moment;

[0120] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to the state information at the t-th moment and the state information at the t+1-th moment in the first sliding window of the multiple sliding windows, the real environment features corresponding to the t+1-th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation corresponding to the t+1-th moment; the first feature weight deviation is greater than 0.5;

[0121] In this way, the real environment features and feature weight deviations corresponding to the t+1th moment can be obtained.

[0122] In some other examples, after obtaining the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment, and the feature weight deviation, the predicted environmental features corresponding to the t+1th moment can be used to calculate the predicted environmental features corresponding to the t+1th moment. The real environment characteristics corresponding to the t+1th moment And feature weight bias b; through the following formula:

[0123]

[0124] Calculate the first reward value corresponding to the state information at the t+1th moment

[0125] In some examples, when the robot is exploring and navigating, each step forward will generate a new sliding window. Therefore, if each sliding window is stored, it will inevitably occupy a large amount of storage space of the mobile robot. If the storage space is not large enough, the mobile robot will delete the old sliding window according to the first-in-first-out principle when storing a new sliding window, which will cause the mobile robot to forget. Therefore, before inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel in the curiosity reward model, each sliding window in the sliding window submodel can also be obtained; according to the real environment features contained in each sliding window, each sliding window is clustered by the k-nearest neighbor clustering algorithm to obtain the clustered sliding window;

[0126] Specifically, during the clustering process, the distance between two adjacent sliding windows can be calculated first, and then a plurality of consecutive sliding windows whose distance is less than a preset threshold are grouped as a cluster. After the clustering is completed, the cluster center of each cluster is the average value of the sliding windows in the cluster.

[0127] After obtaining the clustered sliding window, the state information at the t+1th moment and the state information at the tth moment may be input into the sliding window sub-model including the clustered sliding window in the curiosity reward model;

[0128] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to state information at the tth moment and state information at the t+1th moment in the first sliding window of multiple sliding windows including the clustered sliding windows, the real environment features corresponding to the t+1th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation.

[0129] It is understandable that the multiple sliding windows in the sliding window sub-model may all be clustered, or there may be sliding windows that do not belong to any cluster and cannot be clustered, so the sliding window sub-model may include both clustered sliding windows and sliding windows that cannot be clustered. This is also possible.

[0130] Corresponding to the above method embodiment, the present invention also provides a navigation device, such as Figure 2 FIG. 1 is a schematic diagram of a navigation device according to an embodiment of the present invention. The device may include:

[0131] The navigation module 210 is used to obtain the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action;

[0132] The training module 220 is used to obtain a trained action network model through the following submodules:

[0133] The predicted action acquisition submodule 2201 is used to input the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information;

[0134] The first reward value obtaining submodule 2202 is used to input the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment into the curiosity reward model, and obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment;

[0135] The second reward value obtaining submodule 2203 is used to input the state information at the t+1th moment and the state information at the tth moment into the i-th moment. tThe evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training;

[0136] The updating submodule 2204 is used to update the i-th reward value based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training;

[0137] The updating submodule 2204 is also used to update the i-th t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by the training; and the state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the predicted action acquisition submodule 2201 is triggered to execute the input of the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The steps of predicting actions output by the action network model trained this time.

[0138] Through a navigation device of an embodiment of the present invention, as training progresses, the mobile robot can comprehensively consider the number of times it reaches the target position and the rewards of the state information at each moment in the historical state information to perform actions, rather than just performing actions based on curiosity exploration, thereby reducing the mobile robot's dependence on curiosity and increasing its dependence on the number of times it reaches the target position, thereby reducing the interference of the environment on the robot's navigation and reducing the robot's navigation difficulty.

[0139] In some examples, the predicted action obtaining submodule 2201 is specifically used to:

[0140] i-th t The action network model obtained by training is based on the i tThe number of times the mobile robot reaches the target position before training i-1 is calculated using the following formula:

[0141] e greed =0.8-min(10,(i-1) / 10)×θ

[0142] Calculate the exploration threshold e greed ; θ is the control factor, 0.01≤θ≤0.08;

[0143] Get a random number; when the random number is less than the exploration threshold, randomly select any direction as the i-th direction. t The predicted action output by the action network model obtained through training;

[0144] When the random number is greater than or equal to the exploration threshold, get the i-th t Before the training, the reward corresponding to the state information at the tth moment in the historical state information is calculated, and the action that maximizes the total reward value corresponding to the state information at the tth moment is taken as the i-th action. t The predicted action output by the action network model trained this time.

[0145] In some examples, the first reward value obtaining submodule 2202 is specifically configured to:

[0146] The state information at the tth moment and the real action corresponding to the tth moment are input into the feedforward prediction submodel in the curiosity reward model, and the predicted environment features corresponding to the t+1th moment output by the feedforward prediction submodel are obtained;

[0147] The state information at the t+1th moment and the state information at the tth moment are input into the sliding window sub-model in the curiosity reward model, and the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model are obtained;

[0148] Based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment, and the feature weight deviation, a first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model is obtained.

[0149] In some examples, the first reward value obtaining submodule 2202 is specifically configured to:

[0150] The state information at the t+1th moment and the state information at the tth moment are input into the sliding window submodel in the curiosity reward model; the sliding window submodel stores a plurality of sliding windows, each of which includes a plurality of environmental features;

[0151] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to the state information at the t-th moment and the state information at the t+1-th moment in the first sliding window of the multiple sliding windows, the real environment features corresponding to the t+1-th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation; the first feature weight deviation is greater than 0.5; the first sliding window is a sliding window among the multiple sliding windows that contains at least one of the following real environment features: the real environment feature corresponding to the state information at the t-th moment or the real environment feature corresponding to the state information at the t+1-th moment;

[0152] Based on the predicted environmental features corresponding to the t+1th moment The real environment characteristics corresponding to the t+1th moment And feature weight bias b; through the following formula:

[0153]

[0154] Calculate the first reward value corresponding to the state information at the t+1th moment

[0155] Optionally, the training module also includes:

[0156] The clustering submodule is used to obtain each sliding window in the sliding window submodel before inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel in the curiosity reward model; cluster each sliding window by a k-nearest neighbor clustering algorithm according to the real environment features contained in each sliding window to obtain a clustered sliding window;

[0157] The first reward value obtaining submodule 2202 is specifically used for:

[0158] Input the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel including the clustered sliding window in the curiosity reward model;

[0159] When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to state information at the tth moment and state information at the t+1th moment in the first sliding window of multiple sliding windows including the clustered sliding windows, the real environment features corresponding to the t+1th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation.

[0160] The embodiment of the present invention also provides a mobile robot, such as Figure 3As shown, it includes a processor 301, a communication interface 302, a memory 303, an image sensor 304, a motor 305 and a communication bus 306, wherein the processor 301, the communication interface 302, the memory 303, the image sensor 304 and the motor 305 communicate with each other through the communication bus 306.

[0161] Memory 303, used for storing computer programs;

[0162] An image sensor 304 is used to collect environmental images;

[0163] The motor 305 is used to rotate based on the best executable action motion sent by the processor 301 to make the mobile robot move;

[0164] The processor 301 is used to execute the program stored in the memory 303 to implement the steps of the navigation method shown in any of the above embodiments. For example, the following steps may be performed:

[0165] Obtain the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action;

[0166] The trained action network model is obtained by cyclically executing the following steps:

[0167] Input the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information;

[0168] The state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment are input into the curiosity reward model to obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment;

[0169] Input the state information at the t+1th moment and the state information at the tth moment into the i-th t The evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training;

[0170] Update the i-th reward value based on the first reward value and the second reward value t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training;

[0171] The i t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by training;

[0172] The state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the historical state information of the mobile robot in the current environment is input into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The steps of predicting actions output by the action network model trained this time.

[0173] Through the mobile robot of the embodiment of the present invention, as the training progresses, the mobile robot can comprehensively consider the number of times it reaches the target position and the rewards of the state information at each moment in the historical state information to perform actions, rather than just performing actions based on curiosity exploration, thereby reducing the mobile robot's dependence on curiosity and increasing its dependence on the number of times it reaches the target position, thereby reducing the interference of the environment on the robot's navigation and reducing the robot's navigation difficulty.

[0174] The communication bus mentioned in the mobile robot can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0175] The communication interface is used for communication between the mobile robot and other devices.

[0176] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0177] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0178] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the navigation method shown in any of the above embodiments are implemented. For example, the following steps may be performed:

[0179] Obtain the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action;

[0180] The trained action network model is obtained by cyclically executing the following steps:

[0181] Input the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information;

[0182] The state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment are input into the curiosity reward model to obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment;

[0183] Input the state information at the t+1th moment and the state information at the tth moment into the i-th t The evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training;

[0184] Update the i-th reward value based on the first reward value and the second reward value t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training;

[0185] The i t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by training;

[0186] The state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the historical state information of the mobile robot in the current environment is input into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The steps of predicting actions output by the action network model trained this time.

[0187] Through the computer-readable storage medium of the embodiment of the present invention, as the training progresses, the mobile robot can comprehensively consider the number of times it reaches the target position and the rewards of the state information at each moment in the historical state information to perform actions, rather than just performing actions based on curiosity exploration, thereby reducing the mobile robot's dependence on curiosity and increasing its dependence on the number of times it reaches the target position, thereby reducing the interference of the environment on the robot's navigation and reducing the robot's navigation difficulty.

[0188] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the steps of the navigation method shown in any of the above embodiments. For example, the following steps may be executed:

[0189] Obtain the current state information and target position of the mobile robot in the current environment; input the current state information and target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action;

[0190] The trained action network model is obtained by cyclically executing the following steps:

[0191] Input the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training, t The action network model obtained by training is based on the i tThe number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, and the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information;

[0192] The state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment are input into the curiosity reward model to obtain the first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model; wherein the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment;

[0193] Input the state information at the t+1th moment and the state information at the tth moment into the i-th t The evaluation network model obtained by training is t The second reward value corresponding to the state information at the t+1th moment output by the evaluation network model obtained by the training;

[0194] Update the i-th reward value based on the first reward value and the second reward value t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by training; and based on the i t The predicted action output by the action network model obtained by training and the actual action corresponding to the tth moment are updated t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training;

[0195] The i t+1 The action network model obtained by training and the i-th t+1 The evaluation network model obtained by training is used as the i-th t The action network model obtained by training and the i-th t The evaluation network model obtained by training;

[0196] The state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the historical state information of the mobile robot in the current environment is input into the i-th t The action network model obtained by training, tThe action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are used to determine the i-th t The steps of predicting actions output by the action network model trained this time.

[0197] Through the computer program product containing instructions of the embodiment of the present invention, as the training progresses, the mobile robot can comprehensively consider the number of times it reaches the target position and the rewards of the state information at each moment in the historical state information to perform actions, rather than just performing actions based on curiosity exploration, thereby reducing the mobile robot's dependence on curiosity and increasing its dependence on the number of times it reaches the target position, thereby reducing the interference of the environment on the robot's navigation and reducing the robot's navigation difficulty.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function shown in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0199] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0200] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, mobile robot, readable storage medium, and computer program product embodiments containing instructions, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0201] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A navigation method, characterized in that: The method comprises: Acquire the current state information and target position of the mobile robot in the current environment; input the current state information and the target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action; The trained action network model is obtained by cyclically executing the following steps: The historical state information of the mobile robot in the current environment is input into the i-th t The action network model obtained by training the i-th t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are determined to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information; Inputting the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment into the curiosity reward model, obtaining a first reward value output by the curiosity reward model and corresponding to the state information at the t+1th moment; wherein the state information at the t+1th moment is generated by the mobile robot interacting with the current environment based on the real action corresponding to the tth moment; The state information at the t+1th moment and the state information at the tth moment are input to the i-th t The evaluation network model obtained by training the i-th t A second reward value corresponding to the state information at the t+1th moment and output by the evaluation network model obtained by the training; The i-th reward value is updated based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by the i-th training; and based on the i-th t The predicted action output by the action network model obtained by training and the real action corresponding to the t-th moment are used to update the i-th moment. t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training; The i t+1 The action network model obtained by training the i-th t+1 The evaluation network model obtained by the training is used as the i-th t The action network model obtained by training the i-th t The evaluation network model obtained by training; The state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the historical state information of the mobile robot in the current environment is input into the i-th state information. t The action network model obtained by training the i-th t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are determined to determine the i-th t The steps of predicting actions output by the action network model trained this time.

2. The method according to claim 1, characterized in that The i t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are determined to determine the i-th t The predicted actions output by the action network model obtained through training include: The i t The action network model obtained by the training is based on the i-th t The number of times the mobile robot reaches the target position before training i-1 is calculated by the following formula: e greed =0.8-min(10,(i-1) / 10)×θ Calculate the exploration threshold e greed ; The θ is a control factor, 0.01≤θ≤0.08; Get a random number; when the random number is less than the exploration threshold, randomly select any direction as the i-th direction t The predicted action output by the action network model trained this time; When the random number is greater than or equal to the exploration threshold, obtain the i-th t Before the training, the reward corresponding to the state information at the t-th moment in the historical state information is calculated, and the action that maximizes the total reward value corresponding to the state information at the t-th moment is taken as the i-th action. t The predicted action output by the action network model trained this time.

3. The method according to claim 1, characterized in that The step of inputting the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment into the curiosity reward model to obtain a first reward value output by the curiosity reward model corresponding to the state information at the t+1th moment includes: Inputting the state information at the t-th moment and the real action corresponding to the t-th moment into the feedforward prediction submodel in the curiosity reward model, and obtaining the predicted environmental features corresponding to the t+1-th moment output by the feedforward prediction submodel; Inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window sub-model in the curiosity reward model, and obtaining the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model; Based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment and the feature weight deviation, a first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model is obtained.

4. The method according to claim 3, characterized in that The step of inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window sub-model in the curiosity reward model, and obtaining the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model, comprises: Inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel in the curiosity reward model; the sliding window submodel stores a plurality of sliding windows, each sliding window including a plurality of environmental features; When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to the state information at the t-th moment and the state information at the t+1-th moment in the first sliding window of the multiple sliding windows, the real environment features corresponding to the t+1-th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation; the first feature weight deviation is greater than 0.5; the first sliding window is a sliding window among the multiple sliding windows that contains at least one of the following real environment features: the real environment feature corresponding to the state information at the t-th moment or the real environment feature corresponding to the state information at the t+1-th moment; The obtaining, based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment, and the feature weight deviation, a first reward value output by the curiosity reward model includes: Based on the predicted environmental features corresponding to the t+1th moment The real environment features corresponding to the t+1th moment And the feature weight deviation b; through the following formula: Calculate the first reward value corresponding to the state information at the t+1th moment 5. The method according to claim 4, characterized in that Before inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel in the curiosity reward model, the method further includes: Acquire each sliding window in the sliding window submodel; cluster each sliding window by a k-nearest neighbor clustering algorithm according to the real environment features contained in each sliding window to obtain a clustered sliding window; The step of inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window sub-model in the curiosity reward model includes: Inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window submodel including the clustered sliding window in the curiosity reward model; When the sliding window sub-model in the curiosity reward model simultaneously searches for the real environment features corresponding to the state information at the t-th moment and the state information at the t+1-th moment in the first sliding window of the multiple sliding windows, outputting the real environment features corresponding to the t+1-th moment, and obtaining a preset first feature weight deviation as the feature weight deviation, including: When the sliding window submodel in the curiosity reward model simultaneously searches for real environment features corresponding to the state information at the tth moment and the state information at the t+1th moment in the first sliding window of multiple sliding windows including the clustered sliding windows, the real environment features corresponding to the t+1th moment are output, and a preset first feature weight deviation is obtained as the feature weight deviation.

6. A navigation device, characterized in that: The device comprises: A navigation module is used to obtain the current state information and the target position of the mobile robot in the current environment; input the current state information and the target position into the trained action network model to obtain the best executable action output by the trained action network model, so that the mobile robot moves based on the best executable action; The training module is used to obtain the trained action network model through the following submodules: The predicted action acquisition submodule is used to input the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training the i-th t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are determined to determine the i-th t The predicted action output by the action network model obtained by training; the historical state information is the state information of the mobile robot at each moment in the current environment, the state information includes an image marked with position information, and the state information at each moment has a corresponding real action; i represents the number of times the mobile robot will move from the initial position of the current environment to the target position; 1≤t≤N-1, N is the total number of historical state information; A first reward value obtaining submodule is used to input the state information at the t+1th moment, the state information at the tth moment, and the real action corresponding to the tth moment into the curiosity reward model, and obtain the first reward value output by the curiosity reward model and corresponding to the state information at the t+1th moment; wherein the state information at the t+1th moment is generated by the interaction between the mobile robot and the current environment based on the real action corresponding to the tth moment; The second reward value obtaining submodule is used to input the state information at the t+1th moment and the state information at the tth moment into the i-th moment. t The evaluation network model obtained by training the i-th t A second reward value corresponding to the state information at the t+1th moment and output by the evaluation network model obtained by the training; An updating submodule, configured to update the i-th reward value based on the first reward value and the second reward value. t The model parameters of the evaluation network model obtained by training are obtained t+1 The evaluation network model obtained by the i-th training; and based on the i-th t The predicted action output by the action network model obtained by training and the real action corresponding to the t-th moment are used to update the i-th moment. t The model parameters of the action network model obtained by training are obtained t+1 The action network model obtained by training; The updating submodule is further used to update the i-th t+1 The action network model obtained by training the i-th t+1 The evaluation network model obtained by the training is used as the i-th t The action network model obtained by training the i-th t The evaluation network model obtained by the training; and the state information at the t+1th moment in the historical state information is used as the state information at the tth moment, and the predicted action acquisition submodule is triggered to execute the input of the historical state information of the mobile robot in the current environment into the i-th t The action network model obtained by training the i-th t The action network model obtained by training is based on the i t The number of times the mobile robot reaches the target position before the training, the preset random number, and the reward of the state information at each moment in the historical state information are determined to determine the i-th t The steps of predicting actions output by the action network model trained this time.

7. The device according to claim 6, characterized in that The predicted action obtaining submodule is specifically used for: The i t The action network model obtained by the training is based on the i-th t The number of times the mobile robot reaches the target position before training i-1 is calculated by the following formula: e greed =0.8-min(10,(i-1) / 10)×θ Calculate the exploration threshold e greed ; The θ is a control factor, 0.01≤θ≤0.08; Get a random number; When the random number is less than the exploration threshold, any direction is randomly selected as the i-th direction. t The predicted action output by the action network model trained this time; When the random number is greater than or equal to the exploration threshold, obtain the i-th t Before the training, the reward corresponding to the state information at the t-th moment in the historical state information is calculated, and the action that maximizes the total reward value corresponding to the state information at the t-th moment is taken as the i-th action. t The predicted action output by the action network model trained this time.

8. The device according to claim 6, characterized in that The first reward value obtaining submodule is specifically used for: Inputting the state information at the t-th moment and the real action corresponding to the t-th moment into the feedforward prediction submodel in the curiosity reward model, and obtaining the predicted environmental features corresponding to the t+1-th moment output by the feedforward prediction submodel; Inputting the state information at the t+1th moment and the state information at the tth moment into the sliding window sub-model in the curiosity reward model, and obtaining the real environment features and feature weight deviations corresponding to the t+1th moment output by the sliding window sub-model; Based on the predicted environmental features corresponding to the t+1th moment, the real environmental features corresponding to the t+1th moment and the feature weight deviation, a first reward value corresponding to the state information at the t+1th moment output by the curiosity reward model is obtained.

9. A mobile robot, characterized in that: It includes a processor, a communication interface, a memory, an image sensor, a motor and a communication bus, wherein the processor, the communication interface, the memory, the image sensor and the motor communicate with each other through the communication bus; The memory is used to store computer programs; The image sensor is used to collect environmental images; The motor is configured to rotate based on the best executable action motion sent by the processor to move the mobile robot; The processor is used to implement the method steps described in any one of claims 1-5 when executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images