A motion control decision generation method, device, electronic device and storage medium

Abstract

The present invention provides a motion control decision generation method, device, electronic device and storage medium, including: determining a depth prediction map corresponding to the RGB image information based on first point cloud information of a target area acquired by a laser radar and RGB image information of the target area acquired by a camera; determining target point cloud information based on second point cloud information projected into a three-dimensional space based on the depth prediction map, and obtaining third point cloud information based on the target point cloud information and the first point cloud information; determining target multivariate state data based on the third point cloud information, and inputting the target multivariate state data into a deep reinforcement learning control decision model after multi-stage training to obtain a target motion control decision. The method provided by the present invention can effectively overcome the planar limitations of radar detection, efficiently obtain target motion control decisions, and achieve better mobile obstacle avoidance effects.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a motion control decision generation method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, intelligent robots have been widely used in major service places such as restaurants, shopping malls, and banks. As a key link in the application of intelligent robots, obstacle avoidance technology has attracted great attention from the industry. Traditional robot obstacle avoidance methods mainly include ultrasonic sensor obstacle avoidance, infrared sensor obstacle avoidance, and lidar obstacle avoidance.

[0003] Obstacle avoidance methods based on LiDAR are often limited by cost, power consumption, and simulation difficulty, and single-line radar is often used. However, single-line radar can only detect a fixed plane. If the robot is at a high altitude, only detecting a certain plane cannot achieve a high obstacle avoidance effect. For example, when the radar height is set low, the single-line radar cannot detect obstacles at high places; when the radar height is set high, the single-line radar cannot detect obstacles close to the ground, thereby generating unfavorable motion control decisions and causing collisions.

[0004] Therefore, in the prior art, inaccurate obstacle detection may result in poor control decision-making effect for mobile obstacle avoidance. Summary of the invention

[0005] The present invention provides a motion control decision generation method, device, electronic device and storage medium, which are used to solve the problem that inaccurate obstacle detection leads to poor control decision effect of mobile obstacle avoidance.

[0006] The present invention provides a motion control decision generation method, comprising:

[0007] Determine a depth prediction map corresponding to the RGB image information based on first point cloud information of the target area acquired by the laser radar and RGB image information of the target area acquired by the camera;

[0008] Determine target point cloud information based on second point cloud information projected into the three-dimensional space by the depth prediction map, and obtain third point cloud information based on the target point cloud information and the first point cloud information;

[0009] According to the third point cloud information, target multivariate state data is determined, and the target multivariate state data is input into a deep reinforcement learning control decision model after multi-stage training to obtain a target motion control decision.

[0010] According to a motion control decision generation method provided by the present invention, before determining the target point cloud information, the method further includes:

[0011] Calculate the first height of each pixel in the world coordinate system according to the depth information of each pixel in the depth prediction map;

[0012] Processing the depth prediction map according to the first height of each pixel and a preset height threshold to obtain a filtered target depth map;

[0013] Based on the target depth map, determining the second point cloud information;

[0014] The preset height threshold is determined based on the height of the ground in the world coordinate system and the height of the camera in the world coordinate system.

[0015] According to a motion control decision generation method provided by the present invention, the second point cloud information projected into the three-dimensional space based on the depth prediction map is used to determine the target point cloud information, including:

[0016] Based on coordinate information of the second point cloud information in the world coordinate system, grouping the second point cloud information to obtain multiple groups of fourth point cloud information;

[0017] Target point cloud information is determined according to the distance between each group of the fourth point cloud information and the camera.

[0018] According to a motion control decision generation method provided by the present invention, obtaining third point cloud information according to the target point cloud information and the first point cloud information includes:

[0019] Convert the target point cloud information into a radar coordinate system to obtain fifth point cloud information corresponding to the target point cloud information;

[0020] According to the fifth point cloud information, the first point cloud information is updated to obtain third point cloud information;

[0021] Wherein, the radar coordinate system is determined based on the first point cloud information.

[0022] According to a motion control decision generation method provided by the present invention, before inputting the target multivariate state data into a deep reinforcement learning control decision model after multi-stage training, the method further includes:

[0023] Taking a multivariate state data sample including radar point cloud sample data as a group of training samples, obtaining multiple groups of training samples;

[0024] The multiple groups of training samples are used to perform multi-stage training on the deep reinforcement learning control decision model. When the preset convergence conditions are met, the training is stopped to obtain the deep reinforcement learning control decision model after the multi-stage training.

[0025] According to a motion control decision generation method provided by the present invention, the multiple groups of training samples are used to perform multi-stage training on the deep reinforcement learning control decision model, including:

[0026] For any group of the training samples, the training samples are input into the deep reinforcement learning control decision model for the first stage training, and the first benefit reward value corresponding to the training samples is obtained by using the first value function; when the first benefit reward value converges, the first stage training is stopped to obtain the first deep reinforcement learning control decision model;

[0027] For any group of the training samples, input the training samples into the first deep reinforcement learning control decision model for second-stage training, and obtain a second benefit reward value corresponding to the training samples using a second value function; when the second benefit reward value converges, stop the second-stage training to obtain a second deep reinforcement learning control decision model;

[0028] For any group of the training samples, the training samples are input into the second deep reinforcement learning control decision model for third-stage training, and the third benefit reward value corresponding to the training samples is obtained using the third value function; when the third benefit reward value converges, the third-stage training is stopped to obtain the deep reinforcement learning control decision model after multi-stage training.

[0029] According to a motion control decision generation method provided by the present invention, the first value function is determined according to a preset Euclidean distance reduction term, a collision penalty term and a rotation angle penalty term;

[0030] The second value function is determined according to the first value function and a preset global reward and penalty item;

[0031] The third value function is determined based on the first value function, a preset maximum time step and an average time step of the second stage training.

[0032] The present invention also provides a motion control decision generating device, comprising:

[0033] A prediction module, configured to determine a depth prediction map corresponding to the RGB image information based on first point cloud information of the target area acquired by the laser radar and RGB image information of the target area acquired by the camera;

[0034] An updating module, configured to determine target point cloud information based on second point cloud information projected into a three-dimensional space by the depth prediction map, and to obtain third point cloud information based on the target point cloud information and the first point cloud information;

[0035] A decision module is used to determine target multivariate state data based on the third point cloud information, and input the target multivariate state data into a deep reinforcement learning control decision model after multi-stage training to obtain a target motion control decision.

[0036] The present invention also provides an obstacle avoiding robot, comprising an obstacle avoiding robot body, wherein the obstacle avoiding robot body is provided with an obstacle avoiding processor, a monocular camera and a single-line radar, the monocular camera is arranged on the top of the obstacle avoiding robot body, the single-line radar is arranged on the obstacle avoiding robot body, and the scanning field of view of the single-line radar at least partially overlaps with the shooting range of the monocular camera; and also includes a memory and a program or instruction stored in the memory and executable on the obstacle avoiding processor, wherein when the program or instruction is executed by the obstacle avoiding processor, the steps of the motion control decision generating method as described in any one of the above are executed.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned motion control decision-making methods are implemented.

[0038] The present invention provides a motion control decision generation method, device, electronic device and storage medium. The method obtains a depth prediction map corresponding to the RGB image information based on the first point cloud information of the target area obtained by the laser radar and the RGB image information of the target area obtained by the camera. Then, from the depth prediction map, the distance from the ground to the point closest to the camera in the height range of the camera top is determined, and the distance is used to update the laser radar data so that the updated radar data contains point cloud data of obstacles that cannot be detected by the radar, thereby overcoming the planar limitations of radar detection. According to the updated radar data, a deep reinforcement learning control decision model is trained using a multi-stage value function, and a global consideration is achieved by transferring the result of each training to each time step through the value function. The total time step is transferred to each time step through the value function to achieve speed optimization and improvement, and the target motion control decision is efficiently obtained, achieving a better mobile obstacle avoidance effect, and the practicality is extremely high. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the motion control decision making method provided by the present invention;

[0041] Figure 2 It is a schematic diagram of a front view of the structure of the obstacle avoidance robot provided by the present invention;

[0042] Figure 3 is a schematic diagram of a structural side view of the obstacle avoidance robot provided by the present invention;

[0043] Figure 4 It is a schematic diagram of the overall flow of the motion control decision making method provided by the present invention;

[0044] Figure 5 It is a structural schematic diagram of the motion control decision making device provided by the present invention;

[0045] Figure 6 It is a structural schematic diagram of the obstacle avoidance robot provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Combine the following Figure 1-Figure 6 The invention describes a motion control decision making method, device, electronic device and storage medium.

[0048] Figure 1 is a flow chart of the motion control decision making method provided by the present invention, such as Figure 1 As shown, including:

[0049] Step S110, determining a depth prediction map corresponding to the RGB image information based on first point cloud information of the target area acquired by the laser radar and RGB image information of the target area acquired by the camera;

[0050] Specifically, the target area described in the present invention refers to the same area in the same environment scanned by the lidar and photographed by the camera.

[0051] The first point cloud information described in the present invention refers to the point cloud information obtained after scanning the target area by a laser radar.

[0052] In an embodiment of the present invention, a laser radar may be a single-line radar or a multi-line radar, and a camera may be a monocular camera or a multi-camera. The single-line radar or the multi-line radar may be installed on the lower half of the motion device to detect obstacles close to the ground. At the same time, the monocular camera or the multi-camera may be set to the top of the motion device to obtain a better camera field of view and detect obstacles at high places. The field of view of the camera is set perpendicular to the direction of gravity to collect radar data and RGB image data of the target area, and then calibration may be performed using the radar data and RGB image data to obtain first point cloud information and RGB image information.

[0053] In an embodiment of the present invention, the motion device may be an obstacle avoidance robot, a car, or other devices that can achieve intelligent movement.

[0054] Furthermore, the camera and the laser radar are calibrated based on their relative positions, angles, and camera internal parameters. For the original point cloud information obtained after the radar scans the target area, each point P in the radar coordinate system is calibrated. l , both use the camera's intrinsic parameter matrix R and the relative position T of the camera and the radar, and convert P l Transform to camera coordinates and convert to new point P c ,Right now

[0055] P c =RP l +T;

[0056] Furthermore, the points in the original point cloud information of the radar that are beyond the range of the camera's RGB image can be filtered out, the calibration of the radar data and the RGB image data can be completed, and the first point cloud information of the radar and the RGB image information of the camera can be obtained.

[0057] It should be noted that in the prior art, the mobile obstacle avoidance solution based solely on visual information and deep reinforcement learning models has the problem that the model cannot be well applied from simulation scenarios to real scenarios, which greatly reduces the success rate of obstacle avoidance in real scenarios.

[0058] Since a single-line radar can only detect a fixed plane, a mobile obstacle avoidance solution based solely on single-line radar information and deep reinforcement learning cannot achieve obstacle avoidance when the moving equipment of the single-line radar is set at a high altitude.

[0059] Since the use cost of single-line radar and monocular camera is relatively low, the use of single-line radar and monocular camera can save a lot of cost. In this regard, the present invention designs a robot structure that combines the obstacle avoidance method of single-line radar and monocular camera. In an embodiment of the present invention, the single-line radar and monocular camera are mounted on the obstacle avoidance robot, so that the single-line radar and monocular camera collect data at the same time. The robot performs depth calculation based on the data collected by the single-line radar and monocular camera to obtain motion control decisions, which can achieve a better mobile obstacle avoidance effect.

[0060] Figure 2 : is a schematic diagram of the front view of the structure of the obstacle avoidance robot provided by the present invention, such as Figure 2 As shown, the obstacle avoidance robot mainly includes a monocular camera, a single-line radar and a mobile chassis. The single-line radar is installed on the lower part of the robot to detect obstacles close to the ground. The monocular camera is installed on the top of the robot to obtain a better camera field of view and detect obstacles at high places. The camera's field of view is set perpendicular to the direction of gravity.

[0061] Figure 3 FIG. 1 is a schematic diagram of a side view of the structure of the obstacle avoidance robot provided by the present invention. Figure 3 As shown, the monocular camera is set in front of the top of the robot to obtain a better camera field of view.

[0062] The depth prediction map described in the present invention refers to an RGB image with predicted depth information obtained based on the first point cloud information and the RGB image information and based on a trained image depth prediction network.

[0063] Since the radar data obtains relatively accurate depth information, the radar data and the camera RGB image are used as input data at the same time, that is, the first point cloud information and the RGB image information are input into the trained depth prediction network. This can obtain a more accurate image depth estimation result than the RGB image data alone, that is, the required depth prediction map can be obtained.

[0064] It should be noted that the depth prediction network is used to perform depth estimation on the calibrated radar point cloud information and the camera RGB image information to obtain a depth prediction map corresponding to the RGB image. The depth prediction network can be obtained based on an existing deep neural network, which can be a deep convolutional neural network or other deep neural networks for obtaining the required depth prediction map, which is not specifically limited in the present invention.

[0065] In an embodiment of the present invention, the depth prediction network can adopt a deep convolutional neural network, which consists of an encoder and a decoder. The encoder is composed of a 3×3 convolution module, and the decoder is composed of 4 3×3 kernel transposed convolution modules. The learning rate of the network starts from 0.01 and decreases by 20% every 5 epochs. The batch size is set to 8, and a total of 15 epochs are trained. By inputting the first point cloud information and RGB image information into the deep convolutional neural network, the depth prediction map corresponding to the RGB graphic can be obtained.

[0066] In an embodiment of the present invention, after obtaining the first point cloud information and RGB image information, the depth data of the laser radar can be corresponded to the pixel data in the RGB image through the first point cloud information and RGB image information, and the first point cloud information and RGB image information are input into the trained image depth prediction network to obtain the depth prediction map corresponding to the RGB image.

[0067] Step S120, determining target point cloud information based on second point cloud information projected into the three-dimensional space by the depth prediction map, and obtaining third point cloud information according to the target point cloud information and the first point cloud information;

[0068] Specifically, the second point cloud in the present invention specifically refers to the point cloud information of projecting the depth prediction map into the three-dimensional space, which is in the range from the ground height to the camera height. Since the camera is often set on the top of the motion device, the obstacles that affect the passage of the motion device can be fully considered through the second point cloud.

[0069] The target point cloud information described in the present invention refers to the point cloud information formed by the points closest to the camera in each grouped point cloud obtained after grouping the second point cloud. Since the road condition information during the movement will change constantly, and there may be obstacles that are constantly moving, the application often needs to continuously update the motion decision, and the obstacle closest to the camera is the priority factor to be considered, so the basis for generating the current decision needs to fully consider the target point cloud information closest to the camera.

[0070] In the present invention, since the point cloud information scanned by the radar is often limited by the fixed plane, the radar cannot collect obstacle information outside the height of the fixed plane it scans. By using the target point cloud information obtained based on the depth prediction map corresponding to the RGB image, the first point cloud information is updated according to the target point cloud information, so that the point cloud data of obstacles that cannot be detected by the radar can be reflected in the radar data, effectively overcoming the planar limitation of radar detection.

[0071] Step S130, determining target multivariate state data based on the third point cloud information, and inputting the target multivariate state data into a deep reinforcement learning control decision model after multi-stage training to obtain a target motion control decision.

[0072] Specifically, the target multi-state data described in the present invention refers to multi-state data containing radar third point cloud information. In an embodiment of the present invention, the target multi-state data may specifically include the third point cloud information, the current speed of the moving device, the current position of the moving device and the target position.

[0073] Furthermore, through existing monitoring and positioning technical means, the current speed of the motion device, the current position of the motion device and the target position can be obtained. Therefore, based on the third point cloud information, the target multi-state data can be further determined, and then the target multi-state data can be input into the deep reinforcement learning control decision model after multi-stage training to obtain the target motion control decision.

[0074] In an embodiment of the present invention, the deep reinforcement learning control decision model after multi-stage training is a model obtained after multi-stage training of multi-state data samples including radar point cloud sample data in sequence, and is used to perform deep processing on the input target multi-state data and output target motion control decisions; wherein the training samples are composed of multiple groups of multi-state data samples, and the multi-state data samples are real-time data samples in the training process, including point cloud sample data acquired by the radar in real time, the real-time speed of the motion device, the real-time position of the motion device and the target position.

[0075] In an embodiment of the present invention, during the multi-stage training of the deep reinforcement learning control decision model, each training stage will correspond to a value function, and each training stage will update the corresponding benefit reward value of each stage according to the corresponding value function, and then complete the model training of each stage through the condition of convergence of the benefit reward value. After completing the training of the current stage, the next stage will be entered for training until the training of all stages is completed.

[0076] The method of the present invention trains a deep reinforcement learning control decision model by using a multi-stage value function, so that during the training process, the result of each simulation can be transferred to each time step through the value function, thereby achieving global consideration; at the same time, the total time step can be transferred to each time step through the value function, thereby optimizing and improving the efficiency of the algorithm.

[0077] In the embodiment of the present invention, based on the target motion control decision, the motion device can make an accurate motion control strategy, thereby effectively avoiding the obstacles ahead.

[0078] The method according to the embodiment of the present invention can be specifically applied to technical fields such as robot obstacle avoidance or autonomous driving, and can bring better mobile obstacle avoidance effects for robot obstacle avoidance or autonomous driving.

[0079] The method of the embodiment of the present invention obtains a depth prediction map corresponding to the RGB image based on the first point cloud information of the target area obtained by the laser radar and the RGB image information of the target area obtained by the camera, and then determines the distance from the ground to the point closest to the camera in the height range of the camera top from the depth prediction map, and uses this distance to update the laser radar data, so that the updated radar data contains point cloud data of obstacles that cannot be detected by the radar, thereby overcoming the planar limitations of radar detection; based on the updated radar data, a multi-stage value function is used to train a deep reinforcement learning control decision model, and global considerations are achieved by transferring the results of each training to each time step through the value function, and the total time step is transferred to each time step through the value function to achieve speed optimization and improvement, and efficiently obtain target motion control decisions, achieve better mobile obstacle avoidance effects, and have extremely high practicality.

[0080] Optionally, before determining the target point cloud information, the method further includes:

[0081] Calculate the first height of each pixel in the world coordinate system according to the depth information of each pixel in the depth prediction map;

[0082] Processing the depth prediction map according to the first height of each pixel and a preset height threshold to obtain a filtered target depth map;

[0083] Based on the target depth map, determining the second point cloud information;

[0084] The preset height threshold is determined based on the height of the ground in the world coordinate system and the height of the camera in the world coordinate system.

[0085] Specifically, the depth information described in the present invention refers to the distance information between each point in the scene of the target area and the camera.

[0086] The first height described in the present invention refers to the height in the world coordinate system corresponding to the depth information of each pixel in the depth prediction image.

[0087] The preset height threshold described in the present invention refers to a pixel point filtering range threshold determined based on the height of the ground in the world coordinate system and the height of the camera in the world coordinate system, and is used to filter out all pixels in the depth prediction map that are within the preset height threshold range above and below the ground height, and at the same time filter out all pixels in the depth prediction map that are above the camera height.

[0088] It should be noted that the camera in the embodiment of the present invention is arranged on the top of the motion device to obtain a better camera field of view and detect obstacles at a high position of the radar. For example, when the motion device is a robot, the camera is placed on the top of the robot; when the motion device is a car, the camera can be placed on the roof with a good field of view.

[0089] The target depth map described in the present invention refers to a depth map obtained by filtering the pixels of the depth prediction map and retaining only all pixels within the interval between the ground height and the camera height.

[0090] Furthermore, according to the depth information of all pixels in the depth prediction map, the height corresponding to these depth information in the world coordinate system is calculated. For a depth prediction map M∈R with a size of (u,v) u*v For any pixel (x, y) sampled from the depth prediction map, the height H in the world coordinate system corresponding to the depth information d of the pixel can be calculated using the focal length intrinsic parameter f of the camera. y To calculate, that is

[0091]

[0092] Next, all points at ground height and above the camera height in the depth prediction map are filtered out. Here, the present invention assumes that the motion device will only collide with objects above the ground. When the height of the camera in the world coordinate system is h, set the filter range threshold ε, and filter out the pixels within the range of ε above and below the ground height, and set the pixel points M(x0,y0)=0 within the following range:

[0093] |H+h|<ε;

[0094] Furthermore, the points at the ground height in the depth prediction map are filtered out, and the points above the camera height are filtered out according to the height of the camera in the world coordinate system, so as to obtain a filtered target depth prediction map.

[0095] After obtaining the filtered target depth prediction map, the target depth prediction map is projected into a three-dimensional space, and point cloud information within the range from ground height to camera height can be obtained, thereby obtaining the second point cloud information.

[0096] The method of the embodiment of the present invention calculates the height corresponding to the depth information of all pixels in the depth prediction image in the world coordinate system, and can effectively filter out unnecessary pixels in the calculation process according to a preset filter range threshold, thereby reducing the computational load of the algorithm and improving the efficiency of the algorithm.

[0097] Optionally, determining the target point cloud information based on the second point cloud information projected into the three-dimensional space by the depth prediction map includes:

[0098] Based on coordinate information of the second point cloud information in the world coordinate system, grouping the second point cloud information to obtain multiple groups of fourth point cloud information;

[0099] Target point cloud information is determined according to the distance between each group of the fourth point cloud information and the camera.

[0100] Specifically, the fourth point cloud information described in the present invention refers to point cloud information of each group obtained by grouping the second point cloud information based on the coordinates of the second point cloud information in the world coordinate system.

[0101] In an embodiment of the present invention, among the multiple groups of fourth point cloud information, the point cloud formed by the points in each group of point cloud information with the smallest distance from the camera is used as the target point cloud.

[0102] For sampling in the depth prediction map M∈R u*v For any pixel point, assuming that the coordinates of the pixel point are (i, j), the points with the same coordinate i value are grouped into a group, and then the second point cloud information is grouped, so that multiple groups of point sets with the same coordinate i value can be obtained, and the fourth point cloud information can be obtained based on these point sets with the same coordinate i value.

[0103] In the fourth point cloud information, determine the distance L of the point closest to the camera in each column of points with the same i value from the ground height to the camera height, that is,

[0104] L(j)=minM(,:j),j∈1,2,3...,v,where M(i,j)≠0;

[0105] Among them, M(i,j) represents the distance between each point in the fourth point cloud information and the camera; i represents the horizontal coordinate of the pixel point in the world coordinate system, i∈1,2,3...,u; j represents the vertical coordinate of the pixel point in the world coordinate system, j∈1,2,3...,v, (u,v) represents the size of the depth prediction map.

[0106] Furthermore, the target point cloud information can be obtained based on the points that satisfy the L(j) condition.

[0107] The method of the embodiment of the present invention obtains target point cloud information by projecting the depth prediction map into the point cloud information of the three-dimensional space, determining the point closest to the camera in each column of the point cloud obtained by grouping in the range from the ground height to the camera top height, thereby determining the point cloud data of obstacles that cannot be detected by the laser radar, which is helpful to solve the planar limitations of the laser radar during the scanning detection process.

[0108] Optionally, obtaining third point cloud information according to the target point cloud information and the first point cloud information includes:

[0109] Convert the target point cloud information into a radar coordinate system to obtain fifth point cloud information corresponding to the target point cloud information;

[0110] According to the fifth point cloud information, the first point cloud information is updated to obtain third point cloud information;

[0111] Wherein, the radar coordinate system is determined based on the first point cloud information.

[0112] Specifically, the fifth point cloud information described in the present invention refers to point cloud information obtained by converting target point cloud information into a radar coordinate system.

[0113] In an embodiment of the present invention, the radar coordinate system is determined based on the first point cloud information. According to the coordinate information of the first point cloud information in the world coordinate system, the position information of the laser radar in the world coordinate system is used as the coordinate origin of the radar coordinate system, and then the radar coordinate system of the laser radar and the fixed plane information scanned by the laser radar can be determined.

[0114] Since the field of view of the lidar is larger than that of the camera, the target point cloud information only corresponds to part of the data in the radar data.

[0115] Furthermore, according to the existing coordinate system conversion algorithm, the target point cloud information can be converted to the radar coordinate system to obtain the fifth point cloud information. The fifth point cloud information will correspond to the fixed plane scanned by the laser radar. Each point in the fifth point cloud information will replace each point in the first point cloud information corresponding to its coordinate position, thereby realizing the update of the first point cloud information and obtaining the updated point cloud information, that is, the third point cloud information.

[0116] The method of the embodiment of the present invention determines the distance from the ground height to the point closest to the camera in the camera top height range from the point cloud information of the depth prediction map, and then uses this distance to update the lidar data, so that the updated radar data contains point cloud data of obstacles that cannot be detected by the radar, thereby effectively overcoming the planar limitations of lidar detection.

[0117] Optionally, before inputting the target multivariate state data into the deep reinforcement learning control decision model after multi-stage training, the method further includes:

[0118] Taking a multivariate state data sample including radar point cloud sample data as a group of training samples, obtaining multiple groups of training samples;

[0119] The multiple groups of training samples are used to perform multi-stage training on the deep reinforcement learning control decision model. When the preset convergence conditions are met, the training is stopped to obtain the deep reinforcement learning control decision model after the multi-stage training.

[0120] Specifically, the multi-state data sample containing radar point cloud sample data described in the present invention can be a data sample obtained by acquiring real-time state data of a moving device, which can be a four-tuple data sample composed of point cloud information obtained by the current laser radar detection, the moving speed of the current moving device, the position information of the current moving device and the target position information.

[0121] The preset convergence condition described in the present invention refers to the convergence of the revenue reward value corresponding to the last stage in the multi-stage model training process.

[0122] Before inputting the target multi-state data into the deep reinforcement learning control decision model after multi-stage training, the deep reinforcement learning control decision model needs to be trained as follows:

[0123] A four-tuple data sample consisting of point cloud sample data obtained by the current radar, the moving speed of the current motion device, the position information of the current motion device and the target position information is used as a group of training samples, thereby obtaining multiple groups of four-tuple data training samples.

[0124] Then, after obtaining multiple groups of training samples, the multiple groups of four-tuple data training samples are input into the deep reinforcement learning control decision model in sequence, that is, the four-tuple data samples consisting of the point cloud data obtained by the current lidar detection, the current moving speed of the motion device, the current position information of the motion device and the target position information in each group of training samples are input into the deep reinforcement learning control decision model at the same time. According to each output result of the deep reinforcement learning control decision model, the benefit reward value of the deep reinforcement learning control decision model is calculated, and the parameters of the deep reinforcement learning control decision model are adjusted. Finally, the training process of the deep reinforcement learning control decision model is completed, and the deep reinforcement learning control decision model after multi-stage training is obtained.

[0125] Through the method of the embodiment of the present invention, a multivariate state data sample containing radar point cloud sample data is used as a group of training samples, and a deep reinforcement learning control decision model is trained using multiple groups of training samples, which is beneficial to improving the accuracy of the trained model.

[0126] Optionally, the multiple groups of training samples are used to perform multi-stage training on the deep reinforcement learning control decision model, including:

[0127] For any group of the training samples, the training samples are input into the deep reinforcement learning control decision model for the first stage training, and the first benefit reward value corresponding to the training samples is obtained by using the first value function; when the first benefit reward value converges, the first stage training is stopped to obtain the first deep reinforcement learning control decision model;

[0128] For any group of the training samples, input the training samples into the first deep reinforcement learning control decision model for second-stage training, and obtain a second benefit reward value corresponding to the training samples using a second value function; when the second benefit reward value converges, stop the second-stage training to obtain a second deep reinforcement learning control decision model;

[0129] For any group of the training samples, the training samples are input into the second deep reinforcement learning control decision model for third-stage training, and the third benefit reward value corresponding to the training samples is obtained using the third value function; when the third benefit reward value converges, the third-stage training is stopped to obtain the deep reinforcement learning control decision model after multi-stage training.

[0130] Specifically, the first value function described in the present invention refers to the value function used in the first stage training of the deep reinforcement learning control decision model.

[0131] The first deep reinforcement learning control decision model described in the present invention refers to the deep reinforcement learning control decision model obtained after completing the first stage of training.

[0132] Similarly, the second value function described in the present invention refers to the value function used in the second stage training of the deep reinforcement learning control decision model.

[0133] The second deep reinforcement learning control decision model described in the present invention refers to the deep reinforcement learning control decision model obtained after completing the second stage of training.

[0134] The third value function described in the present invention refers to the value function used in the third stage training of the deep reinforcement learning control decision model.

[0135] Furthermore, for any set of training samples, the training samples are input into the deep reinforcement learning control decision model for three stages of training in sequence, and the value function corresponding to each stage is used to update the Q value of the deep reinforcement learning control decision model, determine the motion control decision corresponding to the Q value, and then obtain the benefit reward value of executing the motion control decision. When the benefit reward value corresponding to each stage converges, the model training of each stage is completed. After completing the training of the current stage, the next stage will be entered for training until the multi-stage deep reinforcement learning control decision model training is completed, and the deep reinforcement learning control decision model after multi-stage training is obtained.

[0136] The method of the embodiment of the present invention uses the value function to perform multi-stage model training on the deep reinforcement learning control decision model, and continuously optimizes and improves the results of each stage of training, so as to efficiently obtain a high-precision deep reinforcement learning control decision model.

[0137] Optionally, the first value function is determined according to a preset Euclidean distance reduction term, a collision penalty term and a rotation angle penalty term;

[0138] The second value function is determined according to the first value function and a preset global reward and penalty item;

[0139] The third value function is determined based on the first value function, a preset maximum time step and an average time step of the second stage training.

[0140] Specifically, in an embodiment of the present invention, when performing the first stage of deep reinforcement learning control decision model training, a first value function r1 is set, which includes the Euclidean distance r reduced from the current position of the motion device to the target position. d , the penalty term r for whether the sports equipment collides c and the penalty term r representing the rotation angle of the motion device w , using t to represent the current time step and i to represent the number of the motion device, a deep reinforcement learning network is trained through multiple motion devices equipped with cameras and lidars, that is, the formula of the first value function is

[0141]

[0142] When training the second-stage deep reinforcement learning control decision model, add the global reward and punishment term r result When the result is that the target position is successfully reached, the value is added to the entire moving trajectory ; when the result is a collision, add a value of The penalty term is set as follows:

[0143]

[0144] When training the third-stage deep reinforcement learning control decision model, a time penalty term is added. The range of the average time step is determined based on the second-stage training, and the penalty constant ρ, t, is determined based on this range. max is the maximum time step set, then the value function r3 of the third stage is

[0145]

[0146] Finally, the updated radar third point cloud information, the current moving speed of the motion device, the current position information of the motion device and the target position information are input into the deep reinforcement learning network after three-stage training to obtain the target motion control decision of the mobile carrier.

[0147] The method of the embodiment of the present invention uses a multi-stage value function to train a deep reinforcement learning control decision model. By transferring the result of each training to each time step through the value function, global consideration can be achieved. The total time step is transferred to each time step through the value function, thereby optimizing the algorithm speed and facilitating the efficient completion of multi-stage deep reinforcement learning control decision model training.

[0148] Figure 4 is a schematic diagram of the overall flow of the motion control decision making method provided by the present invention, such as Figure 4 As shown, including:

[0149] Step S1, install a single-line radar on the lower part of the robot to detect obstacles close to the ground, place a monocular camera on the top of the robot to obtain a better camera field of view and detect obstacles at high places, and make the camera's field of view perpendicular to the direction of gravity, and collect single-line radar data and monocular camera RGB image data.

[0150] Step S2, calibrate the monocular camera and the single-line radar, convert the radar data to the camera coordinate system, so that the depth data of the single-line radar can correspond to the pixel data in the RGB image. Input the calibrated single-line radar data and RGB image data into the trained image depth prediction network to obtain the depth estimation map corresponding to the RGB image, that is, the depth prediction map.

[0151] Step S3, according to the height of the robot, find the distance from the ground height to the point closest to the robot in the height range of the robot top from the depth prediction map, and use this distance to update the data of the single-line radar.

[0152] Wherein, for step S3, it also includes:

[0153] Step S30, calculating the heights of all pixels in the depth prediction image in the world coordinate system;

[0154] Step S31, according to the fixed height of the camera, all points at the ground height are filtered out. Here, the present invention assumes that the robot will only collide with objects above the ground;

[0155] Step S32, from the filtered depth prediction map, find the point in each column of data that is closest to the robot in the range from the ground height to the top height of the robot, and obtain a depth correction map;

[0156] Step S33, converting the points of the closest distance in the depth correction map into the radar coordinate system, and then replacing the points of the corresponding coordinate positions in the single-line radar data with these converted points to obtain updated single-line radar data.

[0157] The updated single-line radar data, the current speed and target position of the robot obtained in step S3 are input into the deep reinforcement learning network after three-stage training to obtain the motion control decision of the obstacle avoidance robot.

[0158] The motion control decision method provided by the embodiment of the present invention solves the obstacle avoidance problem of the obstacle avoidance robot by using a single-line radar and a monocular camera at the same time, which not only overcomes the generalization problem caused by the gap between the simulation scene and the real scene in the single camera solution, but also overcomes the planar limitation of obstacle detection in the single radar solution. At the same time, due to the low-cost single-line radar and monocular camera, the solution can better solve the obstacle avoidance problem of the obstacle avoidance robot at a very low cost.

[0159] In the scenario of the present invention, it is necessary to calibrate the radar data and camera data according to the relative positions of the radar and the camera and the internal parameters of the camera. After the calibration is completed, since the radar data obtains relatively accurate depth, using the radar depth and RGB as input data at the same time can obtain a more accurate image depth prediction result than the RGB image data alone. A more accurate image depth prediction result is very important for subsequent obstacle avoidance.

[0160] In an embodiment of the present invention, the distance from the ground to the point closest to the robot in the height range of the top of the robot in the predicted depth image is found according to the height of the robot. The present invention uses this distance to update the data of the single-line radar, so that the high obstacle data collected by the camera can also be reflected in the updated radar data, thereby overcoming the planar limitations of the single-line radar.

[0161] The present invention designs a multi-stage value function based on the existing deep reinforcement learning model. During the training process, the result of each simulation is passed to each time step through the value function to achieve global consideration; the total time step is passed to each time step through the value function to achieve speed optimization.

[0162] Figure 5 FIG. 1 is a schematic diagram of the structure of the motion control decision making device provided by the present invention. Figure 5 As shown, including:

[0163] A prediction module 510 is used to determine a depth prediction map corresponding to the RGB image information based on first point cloud information of the target area acquired by the laser radar and RGB image information of the target area acquired by the camera;

[0164] An updating module 520, configured to determine target point cloud information based on second point cloud information projected into a three-dimensional space by the depth prediction map, and to obtain third point cloud information according to the target point cloud information and the first point cloud information;

[0165] The decision module 530 is used to determine the target multivariate state data according to the third point cloud information, and input the target multivariate state data into the deep reinforcement learning control decision model after multi-stage training to obtain the target motion control decision.

[0166] The device described in this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0167] Figure 6 : is a schematic diagram of the structure of the obstacle avoidance robot provided by the present invention, such as Figure 6 As shown, it includes an obstacle avoiding robot body 6, in which an obstacle avoiding processor, a monocular camera 61 and a single-line radar 62 are provided. The monocular camera 61 is arranged on the top of the obstacle avoiding robot body 6, and the single-line radar 62 is arranged on the obstacle avoiding robot body 6. The scanning field of view of the single-line radar 62 at least partially overlaps with the shooting range of the monocular camera 61.

[0168] The obstacle avoidance robot of the present invention also includes a memory and a program or instruction stored in the memory and executable on the obstacle avoidance processor. The above-mentioned obstacle avoidance processor can call the logic instructions in the memory to execute the motion control decision generation method described in the present invention, which includes: determining the depth prediction map corresponding to the RGB image information based on the first point cloud information of the target area acquired by the laser radar and the RGB image information of the target area acquired by the camera; determining the target point cloud information based on the second point cloud information projected into the three-dimensional space by the depth prediction map, so as to update the first point cloud information according to the target point cloud information and obtain the third point cloud information; determining the target multivariate state data according to the third point cloud information, and inputting the target multivariate state data into the deep reinforcement learning control decision model after multi-stage training to obtain the target motion control decision; wherein, the deep reinforcement learning control decision model after multi-stage training is obtained after multi-stage training of the multivariate state data sample containing the radar point cloud sample data.

[0169] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0170] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the computer, the computer can execute the motion control decision generation method provided by the above-mentioned methods, and the method includes: based on the first point cloud information of the target area acquired by the laser radar and the RGB image information of the target area acquired by the camera, determining the depth prediction map corresponding to the RGB image information; based on the second point cloud information projected into the three-dimensional space by the depth prediction map, determining the target point cloud information, and obtaining the third point cloud information based on the target point cloud information and the first point cloud information; based on the third point cloud information, determining the target multi-state data, and inputting the target multi-state data into the deep reinforcement learning control decision model after multi-stage training to obtain the target motion control decision.

[0171] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned motion control decision generation methods, the methods comprising: determining a depth prediction map corresponding to the RGB image information based on first point cloud information of a target area acquired by a laser radar and RGB image information of the target area acquired by a camera; determining target point cloud information based on second point cloud information projected into a three-dimensional space from the depth prediction map, and obtaining third point cloud information based on the target point cloud information and the first point cloud information; determining target multivariate state data based on the third point cloud information, and inputting the target multivariate state data into a deep reinforcement learning control decision model after multi-stage training to obtain a target motion control decision.

[0172] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion control decision generation method, characterized in that: include: Determine a depth prediction map corresponding to the RGB image information based on first point cloud information of the target area acquired by the laser radar and RGB image information of the target area acquired by the camera; Based on the second point cloud information projected into the three-dimensional space by the depth prediction map, target point cloud information is determined, and third point cloud information is obtained according to the target point cloud information and the first point cloud information; the target point cloud information refers to the point cloud information consisting of the points closest to the camera in the point clouds of each group obtained after the second point cloud is grouped; Determine target multivariate state data according to the third point cloud information, and input the target multivariate state data into a deep reinforcement learning control decision model after multi-stage training to obtain a target motion control decision; The step of obtaining third point cloud information according to the target point cloud information and the first point cloud information includes: Convert the target point cloud information into a radar coordinate system to obtain fifth point cloud information corresponding to the target point cloud information; According to the fifth point cloud information, the first point cloud information is updated to obtain third point cloud information; Wherein, the radar coordinate system is determined based on the coordinate information of the first point cloud information in the world coordinate system; Using multiple sets of training samples, the deep reinforcement learning control decision model is trained in multiple stages, including: For any group of the training samples, the training samples are input into the deep reinforcement learning control decision model for the first stage training, and the first benefit reward value corresponding to the training samples is obtained by using the first value function; when the first benefit reward value converges, the first stage training is stopped to obtain the first deep reinforcement learning control decision model; For any group of the training samples, input the training samples into the first deep reinforcement learning control decision model for second-stage training, and obtain a second benefit reward value corresponding to the training samples using a second value function; when the second benefit reward value converges, stop the second-stage training to obtain a second deep reinforcement learning control decision model; For any group of the training samples, the training samples are input into the second deep reinforcement learning control decision model for third-stage training, and the third benefit reward value corresponding to the training samples is obtained using the third value function; when the third benefit reward value converges, the third-stage training is stopped to obtain the deep reinforcement learning control decision model after multi-stage training.

2. The motion control decision generation method according to claim 1, characterized in that: Before determining the target point cloud information, the method further includes: Calculate the first height of each pixel in the world coordinate system according to the depth information of each pixel in the depth prediction map; Processing the depth prediction map according to the first height of each pixel and a preset height threshold to obtain a filtered target depth map; Based on the target depth map, determining the second point cloud information; The preset height threshold is determined based on the height of the ground in the world coordinate system and the height of the camera in the world coordinate system.

3. The motion control decision generation method according to claim 1, characterized in that: The determining target point cloud information based on the second point cloud information projected into the three-dimensional space by the depth prediction map includes: Based on coordinate information of the second point cloud information in the world coordinate system, grouping the second point cloud information to obtain multiple groups of fourth point cloud information; Target point cloud information is determined according to the distance between each group of the fourth point cloud information and the camera.

4. The motion control decision generation method according to claim 1, characterized in that: Before inputting the target multivariate state data into the deep reinforcement learning control decision model after multi-stage training, it also includes: Taking a multivariate state data sample including radar point cloud sample data as a group of training samples, obtaining multiple groups of training samples; The multiple groups of training samples are used to perform multi-stage training on the deep reinforcement learning control decision model. When the preset convergence conditions are met, the training is stopped to obtain the deep reinforcement learning control decision model after the multi-stage training.

5. The motion control decision making method according to claim 1, characterized in that: The first value function is determined according to a preset Euclidean distance reduction term, a collision penalty term and a rotation angle penalty term; The second value function is determined according to the first value function and a preset global reward and penalty item; The third value function is determined based on the first value function, a preset maximum time step and an average time step of the second stage training.

6. A motion control decision making device, characterized in that: include: A prediction module, configured to determine a depth prediction map corresponding to the RGB image information based on first point cloud information of the target area acquired by the laser radar and RGB image information of the target area acquired by the camera; An updating module, configured to determine target point cloud information based on the second point cloud information projected into the three-dimensional space by the depth prediction map, and to obtain third point cloud information according to the target point cloud information and the first point cloud information; the target point cloud information refers to point cloud information consisting of points closest to the camera in the point clouds of each group obtained after the second point cloud is grouped; A decision module, used to determine target multivariate state data according to the third point cloud information, and input the target multivariate state data into a deep reinforcement learning control decision model after multi-stage training to obtain a target motion control decision; The step of obtaining third point cloud information according to the target point cloud information and the first point cloud information includes: Convert the target point cloud information into a radar coordinate system to obtain fifth point cloud information corresponding to the target point cloud information; According to the fifth point cloud information, the first point cloud information is updated to obtain third point cloud information; Wherein, the radar coordinate system is determined based on the coordinate information of the first point cloud information in the world coordinate system; Using multiple sets of training samples, the deep reinforcement learning control decision model is trained in multiple stages, including: For any group of the training samples, the training samples are input into the deep reinforcement learning control decision model for the first stage training, and the first benefit reward value corresponding to the training samples is obtained by using the first value function; when the first benefit reward value converges, the first stage training is stopped to obtain the first deep reinforcement learning control decision model; For any group of the training samples, input the training samples into the first deep reinforcement learning control decision model for second-stage training, and obtain a second benefit reward value corresponding to the training samples using a second value function; when the second benefit reward value converges, stop the second-stage training to obtain a second deep reinforcement learning control decision model; For any group of the training samples, the training samples are input into the second deep reinforcement learning control decision model for third-stage training, and the third benefit reward value corresponding to the training samples is obtained using the third value function; when the third benefit reward value converges, the third-stage training is stopped to obtain the deep reinforcement learning control decision model after multi-stage training.

7. An obstacle avoiding robot, characterized in that: The present invention comprises an obstacle avoiding robot body, in which an obstacle avoiding processor, a monocular camera and a single-line radar are arranged, the monocular camera is arranged on the top of the obstacle avoiding robot body, the single-line radar is arranged on the obstacle avoiding robot body, and the scanning field of view of the single-line radar at least partially overlaps with the shooting range of the monocular camera; and the present invention also comprises a memory and a program or instruction stored in the memory and executable on the obstacle avoiding processor, wherein when the program or instruction is executed by the obstacle avoiding processor, the steps of the motion control decision generating method as claimed in any one of claims 1 to 5 are executed.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the motion control decision making method according to any one of claims 1 to 5 are implemented.

Images