A robot movement method, device, equipment and storage medium

Abstract

The present application provides a robot movement method, device, equipment and storage medium, which belongs to the field of robot control technology. The method includes: obtaining the actual state information of the robot, the actual state information includes: current state information and target state information; inputting the actual state information into the pre-trained target neural network model to obtain the robot's to-be-executed action parameters, wherein the target neural network model is trained based on the reinforcement learning reward function, and the reinforcement learning reward function is used to calculate the reward result based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle; controlling the robot to move to the target point according to the to-be-executed action parameters. The present application can avoid collisions between the robot and obstacles and reduce the risk during the robot's movement.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robot movement method, device, equipment and storage medium. Background Art

[0002] In the process of robot navigation and control, there are usually obstacles blocking the robot's planned route, which leads to the robot's urgent need for an obstacle avoidance method to avoid these obstacles and reach the target point.

[0003] In the prior art, after obtaining the location of an obstacle, another route is usually planned to bypass it. However, in actual scenarios, the obstacle is likely to be a moving person or object, that is, a dynamic obstacle. If a new route is planned, it may result in the situation of competing for the dynamic obstacle.

[0004] This may lead to collisions with dynamic obstacles due to lane grabbing, increasing the risk of the robot moving. Summary of the invention

[0005] The purpose of the present application is to provide a robot movement method, device, equipment and storage medium, which can avoid collision between the robot and obstacles and reduce the risk during the robot movement process.

[0006] The embodiment of the present application is implemented as follows:

[0007] In one aspect of an embodiment of the present application, a robot movement method is provided, comprising:

[0008] Obtain the actual state information of the robot, which includes: current state information and target state information;

[0009] The actual state information is input into the pre-trained target neural network model to obtain the parameters of the robot's action to be executed, wherein the target neural network model is trained based on the reinforcement learning reward function, and the reinforcement learning reward function is used to calculate the reward result according to the distance from the robot's current position to the target point and the distance between the robot and the target obstacle;

[0010] Control the robot to move to the target point according to the parameters of the action to be executed.

[0011] Optionally, before obtaining the status information of the robot, the method further includes:

[0012] Collecting the state information of the robot during actual operation as sample state information, the sample state information includes: current state information and target state information;

[0013] Input the sample state information into the initial neural network model to obtain the sample output result;

[0014] Determine the distance from the robot's current position to the target point and the distance between the robot and the target obstacle based on the sample state information;

[0015] According to the distance from the robot's current position to the target point, the distance between the robot and the target obstacle, and the reinforcement learning reward function, the sample reward result is obtained;

[0016] The model parameters of the initial neural network model are updated according to the sample reward results.

[0017] Optionally, a sample reward result is obtained according to the distance from the current position of the robot to the target point, the distance between the robot and the target obstacle, and the reinforcement learning reward function, including:

[0018] Using the reinforcement learning reward function, the first reward result corresponding to the distance from the robot's current position to the target point and the second reward result corresponding to the distance between the robot and the target obstacle are calculated respectively, and the sample reward result is calculated based on the first reward result and the second reward result.

[0019] Optionally, after updating the model parameters of the initial neural network model according to the sample reward result, the method includes:

[0020] Determine an updated sample output result based on the sample state information and the updated neural network model;

[0021] Control the robot to move to the target point according to the updated sample output result;

[0022] If the moving time is greater than a preset threshold or the robot collides with an obstacle, the model parameters of the updated neural network model are updated.

[0023] Optionally, controlling the robot to move to a target point according to the updated sample output result includes:

[0024] If the moving time is less than or equal to the preset threshold and the robot does not collide with the obstacle, the updated neural network model is determined to be the target neural network model.

[0025] Optionally, the robot is a wheeled robot, and the parameters of the action to be executed include the magnitude and direction of the speed of the robot to be moved; controlling the robot to move to the target point according to the parameters of the action to be executed includes:

[0026] Determine the wheel speed direction and wheel speed magnitude of the wheeled robot based on the wheel differential speed model and the magnitude and direction of the robot's to-be-moved speed;

[0027] Move to the target point according to the direction and size of the wheel speed.

[0028] Optionally, the current state information includes: current position information, speed information, and distance information of the robot; the target state information includes: target position information, preset speed information, and preset steering information of the robot.

[0029] Another aspect of an embodiment of the present application provides a robot moving device, including: an acquisition module, a calculation module and a moving module;

[0030] The acquisition module is used to obtain the actual state information of the robot, and the actual state information includes: current state information and target state information;

[0031] A calculation module is used to input the actual state information into a pre-trained target neural network model to obtain the parameters of the robot's to-be-executed action, wherein the target neural network model is trained based on a reinforcement learning reward function, and the reinforcement learning reward function is used to calculate the reward result based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle;

[0032] The mobile module is used to control the robot to move to the target point according to the parameters of the action to be executed.

[0033] Optionally, the acquisition module is also used to collect the state information of the robot during actual operation as sample state information, and the sample state information includes: current state information and target state information; the calculation module is also used to input the sample state information into the initial neural network model to obtain a sample output result; determine the distance from the current position of the robot to the target point and the distance between the robot and the target obstacle based on the sample state information; obtain a sample reward result based on the distance from the current position of the robot to the target point, the distance between the robot and the target obstacle and the reinforcement learning reward function; and update the model parameters of the initial neural network model based on the sample reward result.

[0034] Optionally, the calculation module is specifically used to use the reinforcement learning reward function to calculate the first reward result corresponding to the distance from the robot's current position to the target point and the second reward result corresponding to the distance between the robot and the target obstacle, and calculate the sample reward result based on the first reward result and the second reward result.

[0035] Optionally, the computing module is also used to determine an updated sample output result based on the sample state information and the updated neural network model; control the robot to move to the target point according to the updated sample output result; if the moving time is greater than a preset threshold or the robot collides with an obstacle, update the model parameters of the updated neural network model.

[0036] Optionally, the calculation module is also used to determine that the updated neural network model is the target neural network model if the moving time is less than or equal to a preset threshold and the robot does not collide with the obstacle.

[0037] Optionally, the robot is a wheeled robot, and the parameters of the action to be executed include the magnitude and direction of the robot's speed to be moved; the moving module is specifically used to determine the wheel speed direction and magnitude of the wheeled robot based on the wheel differential model and the magnitude and direction of the robot's speed to be moved; and move to the target point according to the wheel speed direction and magnitude.

[0038] Optionally, in the device, the current state information includes: current position information, speed information, and distance information of the robot; the target state information includes: target position information, preset speed information, and preset steering information of the robot.

[0039] Another aspect of an embodiment of the present application provides a computer device, including: a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the computer program, the steps of the above-mentioned robot movement method are implemented.

[0040] In another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned robot movement method are implemented.

[0041] The beneficial effects of the embodiments of the present application include:

[0042] In a robot movement method, device, equipment and storage medium provided by an embodiment of the present application, the actual state information of the robot can be obtained, and the actual state information includes: current state information and target state information; the actual state information is input into a pre-trained target neural network model to obtain the parameters of the robot's action to be executed, wherein the target neural network model is trained based on a reinforcement learning reward function, and the reinforcement learning reward function is used to calculate the reward result based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle; the robot is controlled to move to the target point according to the parameters of the action to be executed. Among them, the reward result is calculated based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle. The position of the target point and the safe distance of the surrounding obstacles can be comprehensively considered in the process of training the neural network model, so that when the robot uses the target neural network to move, it can maintain the distance from other obstacles while ensuring that it reaches the target point, prevent the situation of grabbing the road, avoid the possible collision between the robot and the obstacle, and reduce the risk of the robot during movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of an application scenario of the robot movement method provided in an embodiment of the present application;

[0045] Figure 2 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 1 ;

[0046] Figure 3 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 2 ;

[0047] Figure 4 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 3 ;

[0048] Figure 5 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 4 ;

[0049] Figure 6 A schematic diagram of robot movement provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of the structure of a robot mobile device provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0053] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0054] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0055] In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0056] The specific implementation and application scenarios of the robot movement method provided in the embodiments of the present application are explained in detail below.

[0057] Figure 1 For a schematic diagram of an application scenario of the robot movement method provided in the embodiment of the present application, please refer to Figure 1 The scene may include a robot 100 and an obstacle 200 . The obstacle 200 may be one or more, and the obstacle 200 may be a dynamic obstacle, such as a person, a car, etc.

[0058] by Figure 1 As shown in the example, the robot 100 needs to go from the current position to the target point 300, and the obstacle 200 blocks the pre-planned path of the robot 100. Therefore, the robot 100 needs to avoid the obstacle 200 and go around the obstacle 200 to reach the target point 300.

[0059] Optionally, the robot 100 may be a humanoid robot, a wheeled robot, etc., and may specifically be a robot used for logistics transportation.

[0060] The specific implementation process of the robot movement method provided in the embodiment of the present application is explained in detail below.

[0061] Figure 2 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 1 , please refer to Figure 2 , robot movement method, including:

[0062] S210: Acquire the actual status information of the robot.

[0063] The actual state information includes: current state information and target state information.

[0064] Optionally, the execution subject of the method may be a controller of the robot, and the robot may execute the method when it obtains the existence of an obstacle within a preset range through an environment acquisition device.

[0065] Optionally, the actual state information may specifically be multiple parameter information of the robot, wherein the current state information may be the observable state information of the robot, that is, the state information that can be directly obtained based on the current position and current working state of the robot; the target state information may be the unobservable state information of the robot, that is, the relevant state information that cannot be directly obtained based on the current position and current working state of the robot and needs to be read through the control of the robot.

[0066] Optionally, the actual state information can be obtained by calculating based on a preset calculation formula after the robot's relevant environment acquisition equipment, such as a laser radar (point cloud data acquisition), a binocular camera, an ultrasonic sensor and other equipment collects the environment information.

[0067] S220: Input the actual state information into the pre-trained target neural network model to obtain the robot's action parameters to be executed.

[0068] Among them, the target neural network model is trained based on the reinforcement learning reward function, and the reinforcement learning reward function is used to calculate the reward result based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle.

[0069] Optionally, the target neural network model may be a neural network model obtained through pre-training, and specifically may be a strategy network, through which the motion strategy of the robot can be characterized.

[0070] Optionally, the target neural network can be specifically trained based on a neural network training method of reinforcement learning, wherein the reinforcement learning reward function is used to calculate the reward result based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle.

[0071] Optionally, the distance between the robot and the target obstacle can be multiple, that is, multiple distances of multiple target obstacles within a certain range of the robot distance.

[0072] Optionally, the parameter of the robot's action to be executed may specifically be a speed parameter, that is, the robot should move at a specific speed and direction.

[0073] S230: Control the robot to move to the target point according to the parameters of the action to be executed.

[0074] Optionally, after determining the parameters of the action to be executed, the robot can be controlled to move to the target point based on the corresponding speed magnitude and speed direction.

[0075] Correspondingly, if an obstacle is found again during the movement process, the above steps can be repeated to obtain new parameters of the action to be executed, and then the robot can be controlled to move to the target point according to the new parameters of the action to be executed, which will not be elaborated here.

[0076] In a robot movement method provided in an embodiment of the present application, the actual state information of the robot can be obtained, and the actual state information includes: current state information and target state information; the actual state information is input into a pre-trained target neural network model to obtain the parameters of the robot's action to be executed, wherein the target neural network model is trained based on a reinforcement learning reward function, and the reinforcement learning reward function is used to calculate the reward result based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle; the robot is controlled to move to the target point according to the parameters of the action to be executed. Among them, the reward result is calculated based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle. The position of the target point and the safe distance of the surrounding obstacles can be comprehensively considered in the process of training the neural network model, so that when the robot uses the target neural network to move, it can maintain the distance from other obstacles while ensuring reaching the target point, preventing the situation of grabbing the road, avoiding the possible collision between the robot and the obstacle, and reducing the risk of the robot during movement.

[0077] Another specific implementation process of the robot movement method provided in the embodiment of the present application is explained in detail below.

[0078] Figure 3 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 2 , please refer to Figure 3 Before obtaining the status information of the robot, the method further includes:

[0079] S310: Collecting the actual running state information of the robot as sample state information.

[0080] The sample status information includes: current status information and target status information.

[0081] Optionally, the state information of the robot during the actual operation process can be collected by an environmental information collection device. After the state information is obtained, the state information can be used as sample state information.

[0082] It should be noted that the information content included in the sample state information is the same as the content information included in the actual state information, and both may be current state information and target state information.

[0083] S320: Input the sample state information into the initial neural network model to obtain the sample output result.

[0084] Optionally, the sample state information may be input into an initial neural network model, wherein the initial neural network model may be an untrained target neural network model, or specifically may be a strategy model for characterizing the robot's motion strategy.

[0085] Optionally, the sample state information is input into the initial neural network model to obtain a corresponding sample output result, which may also be a related speed parameter, similar to the aforementioned parameter of the action to be executed.

[0086] S330: Determine the distance from the current position of the robot to the target point and the distance between the robot and the target obstacle based on the sample state information.

[0087] Optionally, after obtaining the sample output result, the distance from the current position to the target point and the distance between the robot and the target obstacle can be determined based on the sample status information, that is, the distance from the robot to the target point and the distance from the robot to the target obstacle can be determined based on the relevant data collected by the environment acquisition device.

[0088] S340: Obtain a sample reward result based on the distance from the robot's current position to the target point, the distance between the robot and the target obstacle, and the reinforcement learning reward function.

[0089] Optionally, the reinforcement learning reward function can be a preset functional relationship formula, and the reinforcement learning reward function can be used to calculate based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle to obtain a sample reward result.

[0090] S350: Update the model parameters of the initial neural network model according to the sample reward results.

[0091] Optionally, after determining the sample reward result, the model parameters of the initial neural network model can be updated based on the sample reward result until the above-mentioned target neural network model is obtained.

[0092] Optionally, a sample reward result is obtained based on the distance from the robot's current position to the target point, the distance between the robot and the target obstacle, and a reinforcement learning reward function, including: using the reinforcement learning reward function to calculate a first reward result corresponding to the distance from the robot's current position to the target point and a second reward result corresponding to the distance between the robot and the target obstacle, and calculating the sample reward result based on the first reward result and the second reward result.

[0093] Optionally, the reinforcement learning reward function is as follows:

[0094] r all =α 1· r goal +α2· r collison ;

[0095] Among them, r all is the sample reward result; α1 and α2 are weight parameters, usually preset constants; r goal is the first reward result; r collison This is the second reward result.

[0096] It should be noted that the first reward result can be a parameter related to the distance from the robot's current position to the target point, and the second reward result can be a parameter related to the distance between the robot and the target obstacle. Specifically, it can be a proportional relationship, an inverse relationship or other relationship, and can be set according to the actual needs of the user. No specific restrictions are made here.

[0097] Optionally, in the actual calculation process, the first reward result can be determined based on the distance from the current position of the robot to the target point, and the second reward result can be determined based on the distance between the robot and the target obstacle. Then, calculation is performed based on the above-mentioned reinforcement learning reward function to obtain a sample reward result.

[0098] Another specific implementation process of the robot movement method provided in the embodiment of the present application is explained in detail below.

[0099] Figure 4 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 3 , please refer to Figure 4 After updating the model parameters of the initial neural network model according to the sample reward result, the method includes:

[0100] S410: Determine an updated sample output result based on the sample state information and the updated neural network model.

[0101] Optionally, after the initial neural network model is updated, the above-mentioned sample state information can be input into the updated neural network model, and an updated sample output result can be obtained based on the updated neural network model. The updated sample output result can specifically be a speed parameter, which is used to indicate that the robot moves at a specific speed magnitude and a speed direction.

[0102] Optionally, when updating parameters, a PPO (Proximal Policy Optimization, deep reinforcement learning) algorithm may be used to refine the parameters of the neural network, specifically, to update the network weights so that the neural network tends to converge.

[0103] S420: Control the robot to move to the target point according to the updated sample output result.

[0104] Optionally, after determining the updated sample output result, the robot can be moved to the target point based on the sample output result; that is, the robot moves to the target point with a speed magnitude and a speed direction.

[0105] S430: If the moving time is greater than a preset threshold or the robot collides with an obstacle, the model parameters of the updated neural network model are updated.

[0106] Optionally, during the movement process, if the robot collides with an obstacle; or the cumulative movement time after moving to the target point exceeds a preset threshold, it can be determined that the current neural network model has not converged, and it is necessary to continue to update and adjust the neural network parameters. After the adjustment is completed, repeat the above steps S410-S430 until the movement time is less than or equal to the preset threshold and the robot does not collide with the obstacle.

[0107] Optionally, controlling the robot to move to a target point according to the updated sample output result includes:

[0108] S440: If the moving time is less than or equal to the preset threshold and the robot does not collide with the obstacle, determine that the updated neural network model is the target neural network model.

[0109] Optionally, if the moving time is less than or equal to a preset threshold and the robot does not collide with the obstacle, it can be determined that the updated neural network model has converged, and the updated neural network model can be used as a target neural network model for practical application.

[0110] In a robot movement method provided in an embodiment of the present application, an updated sample output result can be determined based on sample state information and an updated neural network model; the robot is controlled to move to a target point according to the updated sample output result; if the movement time is greater than a preset threshold or the robot collides with an obstacle, the model parameters of the updated neural network model are updated. If the movement time is less than or equal to the preset threshold and the robot does not collide with an obstacle, the updated neural network model is determined to be a target neural network model. Among them, cyclically updating the parameters of the neural network model in the above manner can make the model tend to meet user needs. After being determined as a target neural network, when the robot is controlled to move by the target neural network, obstacles can be avoided more accurately, and the target point can be reached more quickly, thereby avoiding collisions between the robot and obstacles while ensuring the movement speed, and reducing the risk during the robot movement.

[0111] The specific implementation process of controlling the robot to move to the target point in the robot movement method provided in the embodiment of the present application is explained in detail below.

[0112] Figure 5 Schematic diagram of the process of the robot movement method provided in the embodiment of the present application Figure 4 , please refer to Figure 5 The robot is a wheeled robot, and the action parameters to be executed include the magnitude and direction of the robot's movement speed; controlling the robot to move to the target point according to the action parameters to be executed includes:

[0113] S510: Determine the wheel speed direction and wheel speed magnitude of the wheeled robot based on the wheel differential model and the magnitude and direction of the speed at which the robot is to move.

[0114] Optionally, the wheel differential model can specifically be a model for converting the magnitude and direction of the speed to be moved into the wheel speed direction and magnitude of the wheel speed of the wheels of the wheeled robot. A certain mapping relationship can be set. After the magnitude and direction of the speed to be moved are input, the wheel speed direction and magnitude of the wheel speed of the wheeled robot can be obtained through the wheel differential model.

[0115] S520: Move to the target point according to the direction and magnitude of the wheel speed.

[0116] Optionally, after determining the direction and magnitude of the wheel speed, the wheeled robot may be controlled to move in this manner to reach a target point.

[0117] It should be noted that the above is only an example for a wheeled robot. For other types of robots, such as bipedal robots, flying robots, and other different types of robots, different calculation models can be used to obtain corresponding movement methods. As long as they can move to the target point, there is no specific restriction here.

[0118] Optionally, the current state information includes: current position information, speed information, and distance information of the robot; the target state information includes: target position information, preset speed information, and preset steering information of the robot.

[0119] Optionally, the current position information of the robot can be specifically represented as a coordinate point, for example: (p x ,p y ), the speed information can be specifically expressed as the horizontal speed and the vertical speed, for example: (v x ,v y ), the distance information may specifically be the distance r between the robot and the obstacle.

[0120] Optionally, the target position information of the robot can be specifically represented as a coordinate point, for example: (p gx ,p gy ), the preset speed information can be specifically the current preset speed of the robot, which can be specifically expressed as a speed value vpref The preset steering information may specifically be the current preset steering direction of the robot, that is, the direction of the preset speed, which may specifically be expressed as a steering angle ψ.

[0121] Optionally, the current state information can be specifically represented as s o =[p x ,p y ,v x v y ,r]∈R 5 , the target state information can be specifically expressed as s h =[p gx ,p gy ,v pref ,ψ]∈R 4 .

[0122] Optionally, the sample status information can be expressed as s t =[s o ,s h ].

[0123] Optionally, the robot's action parameter u can be t Expressed as the velocity v at time t t , that is, u t =v t ; then the target neural network model can be represented as a control strategy, which can be represented as π, then π:

[0124] Among them, s t is the sample status information, For the expected estimated sample state information, the calculation relationship in the specific model is as follows:

[0125] At time t, the model can be expressed as:

[0126]

[0127]

[0128]

[0129]

[0130] in, This is the expectation of the model; is the estimated distance from the robot to the obstacle; P t is the speed at time t, is the estimated speed at time t; Δt is the interval time.

[0131] Figure 6 For the robot movement diagram provided in the embodiment of the present application, please refer to Figure 6 , Figure 6 The figures show feasible obstacle avoidance methods, where a is an obstacle avoidance trajectory when the robot 100 and the dynamic obstacle 200 are moving towards each other; b is an obstacle avoidance trajectory when the robot 100 and the dynamic obstacle 200 are moving in an intersecting manner; and c is an obstacle avoidance trajectory when the robot 100 and the dynamic obstacle 200 are moving in the same direction.

[0132] The following describes the devices, equipment, storage media, etc. corresponding to the robot movement method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0133] Figure 7 For a schematic diagram of the structure of the robot mobile device provided in the embodiment of the present application, please refer to Figure 7 , the robot moving device comprises: an acquisition module 710, a calculation module 720 and a moving module 730;

[0134] The acquisition module 710 is used to acquire the actual state information of the robot, and the actual state information includes: current state information and target state information;

[0135] A calculation module 720 is used to input the actual state information into a pre-trained target neural network model to obtain the parameters of the robot's to-be-executed action, wherein the target neural network model is trained based on a reinforcement learning reward function, and the reinforcement learning reward function is used to calculate the reward result based on the distance from the robot's current position to the target point and the distance between the robot and the target obstacle;

[0136] The moving module 730 is used to control the robot to move to a target point according to the parameters of the action to be executed.

[0137] Optionally, the acquisition module 710 is also used to collect the state information of the robot during actual operation as sample state information, and the sample state information includes: current state information and target state information; the calculation module 720 is also used to input the sample state information into the initial neural network model to obtain a sample output result; determine the distance from the current position of the robot to the target point and the distance between the robot and the target obstacle based on the sample state information; obtain a sample reward result based on the distance from the current position of the robot to the target point, the distance between the robot and the target obstacle and the reinforcement learning reward function; and update the model parameters of the initial neural network model according to the sample reward result.

[0138] Optionally, the calculation module 720 is specifically used to use the reinforcement learning reward function to calculate the first reward result corresponding to the distance from the robot's current position to the target point and the second reward result corresponding to the distance between the robot and the target obstacle, and calculate the sample reward result based on the first reward result and the second reward result.

[0139] Optionally, the computing module 720 is also used to determine an updated sample output result based on the sample state information and the updated neural network model; control the robot to move to the target point according to the updated sample output result; if the moving time is greater than a preset threshold or the robot collides with an obstacle, update the model parameters of the updated neural network model.

[0140] Optionally, the calculation module 720 is further used to determine that the updated neural network model is the target neural network model if the moving time is less than or equal to a preset threshold and the robot does not collide with the obstacle.

[0141] Optionally, the robot is a wheeled robot, and the parameters of the action to be executed include the magnitude and direction of the robot's speed to be moved; the moving module 730 is specifically used to determine the wheel speed direction and the wheel speed magnitude of the wheeled robot based on the wheel differential model and the magnitude and direction of the robot's speed to be moved; and move to the target point according to the wheel speed direction and the wheel speed magnitude.

[0142] Optionally, in the device, the current state information includes: current position information, speed information, and distance information of the robot; the target state information includes: target position information, preset speed information, and preset steering information of the robot.

[0143] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0144] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0145] Figure 8 For a schematic diagram of the structure of the computer device provided in the embodiment of the present application, please refer to Figure 8 The computer device includes: a memory 810 and a processor 820. The memory 810 stores a computer program that can be run on the processor 820. When the processor 820 executes the computer program, the steps of the above-mentioned robot movement method are implemented.

[0146] In another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned robot movement method are implemented.

[0147] Optionally, the above-mentioned computer device may specifically be a controller of the robot.

[0148] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0151] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0153] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robot moving method, characterized in that: include: Acquire actual state information of the robot, wherein the actual state information includes: current state information and target state information; Inputting the actual state information into a pre-trained target neural network model to obtain parameters of the robot's to-be-executed action, wherein the target neural network model is trained based on a reinforcement learning reward function, and the reinforcement learning reward function is used to calculate a reward result based on the distance from the robot's current position to a target point and the distance between the robot and a target obstacle; Control the robot to move to the target point according to the parameters of the action to be executed; Before obtaining the status information of the robot, the method further includes: Collecting the state information of the robot during actual operation as sample state information, wherein the sample state information includes: current state information and target state information; Inputting the sample state information into an initial neural network model to obtain a sample output result; Determine the distance from the current position of the robot to the target point and the distance between the robot and the target obstacle based on the sample state information; Obtaining a sample reward result according to the distance from the current position of the robot to the target point, the distance between the robot and the target obstacle, and the reinforcement learning reward function; Updating the model parameters of the initial neural network model according to the sample reward result; After updating the model parameters of the initial neural network model according to the sample reward result, the method includes: Determine an updated sample output result based on the sample state information and the updated neural network model; Control the robot to move to the target point according to the updated sample output result; If the moving time is greater than a preset threshold or the robot collides with an obstacle, the model parameters of the updated neural network model are updated.

2. The method according to claim 1, characterized in that The obtaining of a sample reward result according to the distance from the current position of the robot to the target point, the distance between the robot and the target obstacle, and the reinforcement learning reward function includes: Using the reinforcement learning reward function, the first reward result corresponding to the distance from the robot's current position to the target point and the second reward result corresponding to the distance between the robot and the target obstacle are calculated respectively, and the sample reward result is calculated based on the first reward result and the second reward result.

3. The method according to claim 1, characterized in that The controlling the robot to move to the target point according to the updated sample output result comprises: If the moving time is less than or equal to a preset threshold and the robot does not collide with an obstacle, the updated neural network model is determined to be the target neural network model.

4. The method according to claim 1, characterized in that The robot is a wheeled robot, and the parameters of the action to be executed include the magnitude and direction of the speed of the robot to be moved; The step of controlling the robot to move to the target point according to the parameters of the action to be executed comprises: Determining the direction and magnitude of the wheel speed of the wheeled robot based on the wheel differential model and the magnitude and direction of the speed of the robot to be moved; Move to the target point according to the wheel speed direction and wheel speed magnitude.

5. The method according to claim 1, characterized in that The current state information includes: the current position information, speed information, and distance information of the robot; the target state information includes: the target position information, preset speed information, and preset steering information of the robot.

6. A robot mobile device, characterized in that: include: Acquisition module, calculation module and movement module; The acquisition module is used to acquire actual state information of the robot, wherein the actual state information includes: current state information and target state information; The calculation module is used to input the actual state information into a pre-trained target neural network model to obtain parameters of the robot's to-be-executed actions, wherein the target neural network model is trained based on a reinforcement learning reward function, and the reinforcement learning reward function is used to calculate a reward result based on a distance from the robot's current position to a target point and a distance between the robot and a target obstacle; The moving module is used to control the robot to move to the target point according to the parameters of the action to be executed; The acquisition module is further used to collect state information of the robot during actual operation as sample state information, wherein the sample state information includes: current state information and target state information; The calculation module is further used to input the sample state information into the initial neural network model to obtain a sample output result; determine the distance from the current position of the robot to the target point and the distance between the robot and the target obstacle based on the sample state information; obtain a sample reward result according to the distance from the current position of the robot to the target point, the distance between the robot and the target obstacle and the reinforcement learning reward function; and update the model parameters of the initial neural network model according to the sample reward result; The calculation module is also used to determine the updated sample output result based on the sample state information and the updated neural network model; control the robot to move to the target point according to the updated sample output result; if the movement time is greater than a preset threshold or the robot collides with an obstacle, update the model parameters of the updated neural network model.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1 to 5.

Images