Intelligent robot control method and system based on deep learning

Abstract

The invention relates to the technical field of intelligent robot control analysis, and particularly discloses an intelligent robot control method and system based on deep learning. Comprising an image data feature analysis module, a video data feature analysis module, a visual feature vector analysis module, an internal sensor data analysis module, an external positioning data analysis module and a pose state control analysis module. High-precision positioning and map construction can be realized through operations such as voxel filtering, feature point extraction and matching, pose map construction and optimization and the like of laser radar point cloud data; based on Kalman filtering, internal and external sensor data are fused, position, attitude and speed information is comprehensively considered, the advantages of each sensor can be fully played, noise and uncertainty are effectively processed, position and attitude estimation of the robot is continuously optimized, high-precision positioning, attitude control and motion planning are realized, and the method is suitable for large-scale popularization and application. And the autonomy, adaptability and task execution capability of the robot in a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robot control analysis technology, and specifically to an intelligent robot control method and system based on deep learning. Background Art

[0002] With the continuous development of science and technology, intelligent robots have been widely used in many fields, such as industrial production, logistics distribution, medical services, etc. In these application scenarios, robots need to have high-precision environmental perception, accurate target recognition and tracking, and precise operation and control capabilities. However, the complex and changeable real environment poses a huge challenge to the control of intelligent robots. Traditional robot control methods often rely on a single type of sensor data, which is difficult to meet the requirements for robot autonomy and adaptability in complex environments. For example, relying solely on internal sensors may lead to inaccurate positioning and posture estimation due to cumulative errors, while relying solely on external sensors may have data processing delays and insufficient adaptability to environmental changes. Therefore, developing a method that can integrate multi-source data for effective control has become a key requirement for the development of intelligent robots.

[0003] At present, in the visual perception of intelligent robots, although deep learning-based target detection algorithms such as YOLO and FasterR-CNN have been applied, most of these algorithms only focus on target recognition at the image level. There is still room for improvement in how to make full use of dynamic information in video data and how to effectively fuse image and video data to enhance the robot's comprehensive perception of the environment. In terms of robot positioning and attitude estimation, the use of inertial measurement units can provide relatively fast position and attitude information in a short time, but due to the problem of error accumulation, long-term use will lead to large deviations. Although the external positioning method based on laser radar can obtain high-precision positioning results, the data processing is relatively complex and may be interfered by environmental factors in some scenarios, such as the influence of light changes on laser reflection. In addition, when fusing internal sensor data with external positioning data, existing methods often cannot balance real-time and accuracy well, resulting in unsatisfactory control effects of robots in complex environments. Summary of the invention

[0004] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides an intelligent robot control method and system based on deep learning to solve the above-mentioned technical defects.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an intelligent robot control method based on deep learning, comprising the following steps:

[0006] Step 1: Extract image data features: Use a deep learning-based target detection algorithm to identify the target object in the image, then input the identified image into the visual feature analysis module, divide the input image into several regions, and calculate the average, maximum, and minimum values ​​of the pixels in each region. The average, maximum, and minimum values ​​of the pixels in each region form a feature vector for each region. Finally, the feature vectors of several regions are concatenated to obtain an image feature vector.

[0007] Step 2: Extract video data features: Use the optical flow algorithm to calculate the optical flow field for each frame in the video data, divide the optical flow field into several regions, calculate the mean and standard deviation of the optical flow vector in each region, and obtain a regional optical flow feature vector. Finally, perform temporal pooling on the regional optical flow feature vectors of several frames in the video data to obtain a video feature vector.

[0008] Step 3: Fusion of visual feature vectors: By constructing an attention mechanism module, inputting the image feature vector and the video feature vector into the attention mechanism module, mapping the image feature vector and the video feature vector to an intermediate representation space through a shared fully connected layer, obtaining the image feature mapping vector and the video feature mapping vector, which are denoted as FI and FV respectively, and finally calculating the comprehensive visual feature vector ST through the formula ST = w1*FI+w2*FV, where w1 and w2 represent the weight coefficients corresponding to the image feature mapping vector and the video feature mapping vector respectively;

[0009] Step 4: Internal sensor data analysis: Let the quaternion corresponding to the initial posture of the intelligent robot be q 0 =(q 0w ,q 0x ,q 0y ,q 0z ), the angular velocity data provided by the inertial measurement unit is ω=(ω x ,ω y ,ω z ), the sampling period is Δt, where ω x ,ω y ,ω z They are expressed as angular velocities around the x, y, and z axes respectively;

[0010] By formula Calculate the modulus of the angular velocity, and then use the formula

[0011] Calculate the quaternion increment Δq, and finally update the formula based on the quaternion Calculate the quaternion vector q of the intelligent robot's posture at each motion moment t+1 ;

[0012] Assume the acceleration provided by the inertial measurement unit is a = (a x , a y , a z ), initial velocity v 0 =(v 0x , v 0y , v 0z ), the initial position is x 0 =(x 0x , x 0y , x 0z ), therefore, the velocity update formula is v t+1 =vt+atΔt, the position update formula is x t+1 =xt+vtΔt;

[0013] Step 5: External positioning data analysis: Assume that the point cloud data obtained by the laser radar of the intelligent robot each time is P = {p 1 , p 2 ,…,p n}, where P i =(x i ,y i , z i ) is represented as the three-dimensional coordinates of the i-th point, and the voxel filtering method is used to remove noise points. Then the size of the voxel grid is determined, and the point cloud space is divided into a number of cubes with a side length of l, namely voxels. At the same time, the centroid of the point in each voxel is calculated as the representative point of the voxel;

[0014] Normal vector estimation and curvature calculation are used to extract feature points from the preprocessed point cloud data. i By calculating the covariance matrix of the points in the spherical neighborhood with a radius of r, the covariance matrix is ​​decomposed by eigenvalues. The eigenvector corresponding to the minimum eigenvalue is the point P i The normal vector n i , and according to the neighborhood point to point P i The curvature c is calculated based on the distance and the change of the normal vector i , if the curvature is greater than the set threshold, point P i Marked as feature points, let the extracted feature point set be J = {f 1 , f 2 , …, f m}, in the point cloud data at each motion moment, the corresponding feature point pairs are found through the nearest neighbor search algorithm;

[0015] The pose graph is constructed based on the matched feature point pairs. The nodes in the pose graph represent the poses of the intelligent robot at each motion moment. Assuming that in a two-dimensional plane, x i =(x ix , x iy ,θ i), where x ix and x iy are expressed as position coordinates, θ i Expressed as attitude angle;

[0016] The constraint relationship between the poses is represented by the edge, and the constraint is calculated by matching feature point pairs. Suppose two matching feature point pairs (f i , f j ) in two different positions x i and x j The coordinate transformation relationship under T is ij , define the error function of the pose graph Where C represents the set of all constraint edges, It is expressed as the coordinate transformation relationship obtained by feature matching, T ij (x i , x j ) is represented by the coordinate transformation relationship calculated according to the current posture;

[0017] By using the Gauss-Newton method to minimize the error function E(x), we first calculate the Jacobian matrix J of the error function with respect to the pose variable x, and then use the iterative formula Δx = (J T J) -1 J T e updates the pose estimate x, where e is represented as the error vector;

[0018] Step 6. Posture state control analysis: Assume that the state vector of the intelligent robot after fusion is in is represented as the transpose of the position vector, Expressed as the transpose of the attitude vector, Expressed as the transpose of the velocity vector;

[0019] The posture state is predicted based on the data of the internal sensors, the position and velocity are predicted through the integral formula based on acceleration, and the posture state is predicted through the quaternion update formula based on angular velocity;

[0020] Assume the state transfer function is f(X k ,u k ), where X k It is represented as the state vector at the k-th motion moment, u k is represented by the control input obtained through the inertial measurement unit, then the predicted state vector is At the same time, the covariance matrix of the predicted state is calculated According to Kalman filter theory, where F k It is expressed as the Jacobian matrix of the state transfer function f with respect to the state vector X, P kIt is represented as the covariance matrix of the state vector at k motion moments, It is represented as the transpose of the covariance matrix of the state vector at k motion moments, Q k It is represented as the covariance matrix of model uncertainty and sensor noise;

[0021] When the estimated value x of the external positioning data is obtained, the state of the intelligent robot is updated. The measurement value of the external positioning system is set to Z k , the measurement function is h(X k ), calculate the measurement residual and the covariance matrix of the measurement residuals Among them, H k It is expressed as the Jacobian matrix of the measurement function h with respect to the state vector X, It is expressed as the transpose of the Jacobian matrix of the measurement function h with respect to the state vector X, R k Expressed as the covariance matrix of the measurement noise;

[0022] Finally calculate the Kalman gain The updated state vector is The updated covariance matrix is in Expressed as the prediction covariance matrix in Kalman filtering, is represented as the predicted state vector in the Kalman filter, and I is represented as the identity matrix.

[0023] Furthermore, in step 1, the target detection algorithm may be YOLO or FasterR-CNN.

[0024] Furthermore, in the step 2, the optical flow field is a two-dimensional vector field, which represents the movement speed and direction of each pixel in the image, and the temporal pooling of the regional optical flow feature vectors of several frames in the video data can be maximum pooling or average pooling.

[0025] Furthermore, in step 3, both w1 and w2 are calculated by using the soft-max function, specifically: where s(·) is represented as the dot product function.

[0026] Furthermore, in the step 4, vt in the velocity update formula and the position update formula represents the velocity vector at the movement time t, xt represents the position vector at the movement time t, and at represents the acceleration vector at the movement time t.

[0027] Furthermore, in step 5, for the feature point f i , find the point f closest to it in the point cloud data at another time j , if their distance is less than the set threshold, it is considered that (fi , f j ) is a pair of matching feature points.

[0028] Furthermore, in step five, when estimating the pose, the point cloud data is fused into the global map. For each new point cloud scan, it is converted to the global coordinate system according to the current estimated pose and then added to the global map to continuously update and improve the map.

[0029] Furthermore, in step six, transposition is to interchange rows and columns of the matrix.

[0030] Furthermore, in step six, by continuously repeating the prediction and update steps, the data of the internal and external sensors are integrated to obtain a more accurate estimation of the position and posture of the intelligent robot.

[0031] Further, a control system applied to an intelligent robot control method based on deep learning includes: an image data feature analysis module, a video data feature analysis module, a visual feature vector analysis module, an internal sensor data analysis module, an external positioning data analysis module and a posture state control analysis module;

[0032] The image data feature analysis module is used to extract and analyze the image data features to obtain the image feature vector;

[0033] The video data feature analysis module is used to extract and analyze the features of the video data to obtain the video feature vector;

[0034] The visual feature vector analysis module is used to perform comprehensive calculation and analysis on the image feature vector and the video feature vector to obtain a comprehensive visual feature vector;

[0035] The internal sensor data analysis module is used to calculate and analyze the internal sensor data of the intelligent robot to obtain the rate update formula and the position update formula;

[0036] The external positioning data analysis module is used to calculate and analyze the external positioning data of the intelligent robot to obtain the position and posture of the intelligent robot at each movement moment;

[0037] The posture state control analysis module obtains more accurate position and posture estimation of the intelligent robot by continuously repeating the prediction and updating steps and fusing the data of internal and external sensors.

[0038] Beneficial effects of the present invention:

[0039] 1. The present invention can accurately identify the target object in the image by adopting the target detection algorithm based on deep learning such as YOLO or FasterR-CNN. These algorithms have been trained with a large amount of data and have strong recognition ability for targets of different types and in different scenes, which helps the robot to quickly locate and identify relevant targets in complex environments; at the same time, by dividing the image into several regions and calculating the average, maximum and minimum values ​​of the pixels in each region to form a feature vector, this method can capture the characteristics of different local regions of the image. The image feature vector obtained by splicing the feature vectors of these regions comprehensively describes the content of the image and provides rich information for subsequent control decisions;

[0040] In addition, the optical flow algorithm is used to calculate the optical flow field for each frame in the video data, which can well reflect the movement speed and direction of the objects in the video. This is very important for robots to understand dynamic scenes and track moving targets. For example, when the robot follows a moving target or avoids a dynamic obstacle, the optical flow feature provides key motion clues. The optical flow field is divided into regions and the mean and standard deviation of the optical flow vector in the region are calculated to form a regional optical flow feature vector. Then, the video feature vector is obtained through time pooling (maximum pooling or average pooling). It can effectively compress the features in the video data while retaining important motion information, reduce the amount of data processing, and improve computing efficiency.

[0041] 2. The image feature vector and the video feature vector are fused through the attention mechanism module and the shared fully connected layer to obtain a comprehensive visual feature vector, which can integrate static image information and dynamic video information, so that the robot has a more comprehensive and accurate visual perception of the environment. For example, when grabbing an object that has both static appearance features and is in motion, the comprehensive visual features can better assist the robot in operation; the weight coefficients w1 and w2 are calculated using the soft-max function, and the weights can be adaptively allocated according to the actual situation of the image feature mapping vector and the video feature mapping vector. In this way, in different scenarios, the robot can make more reasonable use of image and video features. For example, in static scenes, the weight of image features may be higher; in dynamic scenes, the weight of video features will be more prominent, thereby optimizing the robot's control strategy. By deeply mining the features in image and video data and effectively fusing them, the intelligent robot based on deep learning can perform better in environmental perception, target recognition and tracking, and operational decision-making, thereby improving the robot's autonomy and adaptability in complex environments.

[0042] 3. In terms of vision, the present invention recognizes images through a target detection algorithm based on deep learning, and further analyzes the pixel features of the image area to obtain an image feature vector, which can help the robot quickly locate and identify targets in complex environments; uses an optical flow algorithm to process video data to obtain a video feature vector, which can effectively capture object motion information and compress features to reduce data processing volume; fuses image and video feature vectors through an attention mechanism and a shared fully connected layer, and adaptively assigns weights, which can enable the robot to have a more comprehensive and accurate visual perception of the environment and assist in operational decision-making. In terms of sensor data processing, the quaternion combined with the angular velocity data of the inertial measurement unit can accurately obtain the machine The robot's posture and acceleration data combined with the initial conditions can provide relatively accurate position and velocity information in a short period of time, which is helpful for instant motion control and preliminary positioning; using lidar point cloud data, through voxel filtering, feature point extraction and matching, pose graph construction and optimization and other operations, high-precision positioning and map construction can be achieved; based on the Kalman filter to fuse internal and external sensor data, and comprehensively consider the position, posture and velocity information, it can give full play to the advantages of each sensor, effectively deal with noise and uncertainty, and continuously optimize the robot's position and posture estimation, achieve high-precision positioning, posture control and motion planning, and improve the robot's autonomy, adaptability and task execution capabilities in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further described below in conjunction with the accompanying drawings.

[0044] Figure 1 This is a flow chart of an intelligent robot control method based on deep learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present invention.

[0046] As shown in the present invention and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0047] Although the present invention has made various references to certain modules in the system according to an embodiment of the present invention, any number of different modules can be used and run on a user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0048] The present invention uses a flow chart to illustrate the operations performed by the system according to an embodiment of the present invention. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0049] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0050] Embodiment 1:

[0051] See also Figure 1 As shown, the intelligent robot control method based on deep learning includes the following steps:

[0052] Step 1: Extract image data features: Use a deep learning-based target detection algorithm to identify the target object in the image (the target detection algorithm can be YOLO or FasterR-CNN), then input the identified image into the visual feature analysis module, divide the input image into several regions, and calculate the average, maximum and minimum values ​​of the pixels in each region. The average, maximum and minimum values ​​of the pixels in each region form a feature vector for each region. Finally, the feature vectors of several regions are spliced ​​to obtain an image feature vector.

[0053] Step 2: Extract video data features: Use the optical flow algorithm to calculate the optical flow field for each frame in the video data, divide the optical flow field into several regions, calculate the mean and standard deviation of the optical flow vector in each region, and obtain a regional optical flow feature vector. Finally, perform temporal pooling on the regional optical flow feature vectors of several frames in the video data to obtain a video feature vector.

[0054] It should be noted that the optical flow field is a two-dimensional vector field, which represents the movement speed and direction of each pixel in the image. The temporal pooling of the regional optical flow feature vectors of several frames in the video data can be maximum pooling or average pooling.

[0055] Step 3: Fusion of visual feature vectors: By constructing an attention mechanism module, inputting the image feature vector and the video feature vector into the attention mechanism module, mapping the image feature vector and the video feature vector to an intermediate representation space through a shared fully connected layer, obtaining the image feature mapping vector and the video feature mapping vector, which are denoted as FI and FV respectively, and finally calculating the comprehensive visual feature vector ST through the formula ST = w1*FI+w2*FV, where w1 and w2 represent the weight coefficients corresponding to the image feature mapping vector and the video feature mapping vector respectively;

[0056] It should be noted that both w1 and w2 are calculated by using the soft-max function, specifically: where s(·) is represented as the dot product function.

[0057] In a specific embodiment, the present invention can accurately identify the target object in the image by adopting a target detection algorithm based on deep learning such as YOLO or FasterR-CNN. These algorithms have been trained with a large amount of data and have strong recognition capabilities for targets of different types and in different scenarios, which helps the robot to quickly locate and identify relevant targets in complex environments; at the same time, by dividing the image into several regions and calculating the average, maximum and minimum values ​​of the pixels in each region to form a feature vector, this method can capture the characteristics of different local regions of the image, and the image feature vector obtained by splicing these regional feature vectors comprehensively describes the content of the image, providing rich information for subsequent control decisions;

[0058] In addition, the optical flow field is calculated for each frame in the video data through the optical flow algorithm, which can well reflect the movement speed and direction of the objects in the video. This is very important for robots to understand dynamic scenes and track moving targets. For example, when the robot follows a moving target or avoids a dynamic obstacle, the optical flow feature provides key motion clues. The optical flow field is divided into regions and the mean and standard deviation of the optical flow vector in the region are calculated to form a regional optical flow feature vector. Then, the video feature vector is obtained through time pooling (maximum pooling or average pooling). It can effectively compress the features in the video data while retaining important motion information, reduce the amount of data processing, and improve computing efficiency.

[0059] Finally, the image feature vector and the video feature vector are fused through the attention mechanism module and the shared fully connected layer to obtain a comprehensive visual feature vector, which can integrate static image information and dynamic video information, so that the robot has a more comprehensive and accurate visual perception of the environment. For example, when grasping an object that has both static appearance features and is in motion, the comprehensive visual features can better assist the robot in operation; the weight coefficients w1 and w2 are calculated using the soft-max function, and the weights can be adaptively allocated according to the actual situation of the image feature mapping vector and the video feature mapping vector. In this way, in different scenarios, the robot can make more reasonable use of image and video features. For example, in static scenes, the weight of image features may be higher; in dynamic scenes, the weight of video features will be more prominent, thereby optimizing the robot's control strategy. By deeply mining the features in image and video data and effectively fusing them, the intelligent robot based on deep learning can perform better in environmental perception, target recognition and tracking, and operational decision-making, thereby improving the robot's autonomy and adaptability in complex environments.

[0060] Step 4: Internal sensor data analysis: Let the quaternion corresponding to the initial posture of the intelligent robot be q 0 =(q 0w ,q 0x ,q 0y ,q 0z ), the angular velocity data provided by the inertial measurement unit is ω=(ω x ,ω y ,ω z ), the sampling period is Δt, where ω x ,ω y ,ω z They are expressed as angular velocities around the x, y, and z axes respectively;

[0061] By formula Calculate the modulus of the angular velocity, and then use the formula

[0062] Calculate the quaternion increment Δq, and finally update the formula based on the quaternion Calculate the quaternion vector q of the intelligent robot's posture at each motion moment t+1 ;

[0063] Assume the acceleration provided by the inertial measurement unit is a = (a x , a y , a z ), initial velocity v 0 =(v 0x , v 0y , v 0z ), the initial position is x 0 =(x 0x , x0y , x 0z ), therefore, the velocity update formula is v t+1 =vt+atΔt, the position update formula is x t+1 =xt+vtΔt;

[0064] It should be noted that, in the above speed update formula and position update formula, vt represents the velocity vector at the movement time t, xt represents the position vector at the movement time t, and at represents the acceleration vector at the movement time t.

[0065] Step 5: External positioning data analysis: Assume that the point cloud data obtained by the laser radar of the intelligent robot each time is P = {p 1 , p 2 ,…,p n}, where P i =(x i ,y i , z i ) is represented as the three-dimensional coordinates of the i-th point, and the voxel filtering method is used to remove noise points. Then the size of the voxel grid is determined, and the point cloud space is divided into a number of cubes with a side length of l, namely voxels. At the same time, the centroid of the point in each voxel is calculated as the representative point of the voxel;

[0066] Normal vector estimation and curvature calculation are used to extract feature points from the preprocessed point cloud data. i By calculating the covariance matrix of the points in the spherical neighborhood with a radius of r, the covariance matrix is ​​decomposed by eigenvalues. The eigenvector corresponding to the minimum eigenvalue is the point P i The normal vector n i , and according to the neighborhood point to point P i The curvature c is calculated based on the distance and the change of the normal vector i , if the curvature is greater than the set threshold, point P i Marked as feature points, let the extracted feature point set be J = {f 1 , f 2 , …, f m}, in the point cloud data at each motion moment, the corresponding feature point pairs are found through the nearest neighbor search algorithm;

[0067] It should be noted that for the feature point f i , find the point f closest to it in the point cloud data at another time j , if their distance is less than the set threshold, it is considered that (f i , f j ) is a pair of matching feature points.

[0068] The pose graph is constructed based on the matched feature point pairs. The nodes in the pose graph represent the poses of the intelligent robot at each motion moment. Assuming that in a two-dimensional plane, x i =(x ix , x iy ,θ i ), where x ix and x iy are expressed as position coordinates, θ i Expressed as attitude angle;

[0069] The constraint relationship between the poses is represented by the edge, and the constraint is calculated by matching feature point pairs. Suppose two matching feature point pairs (f i , f j ) in two different positions x i and x j The coordinate transformation relationship under T is ij , define the error function of the pose graph Where C represents the set of all constraint edges, It is expressed as the coordinate transformation relationship obtained by feature matching, T ij (x i , x j ) is represented by the coordinate transformation relationship calculated according to the current posture;

[0070] By using the Gauss-Newton method to minimize the error function E(x), we first calculate the Jacobian matrix J of the error function with respect to the pose variable x, and then use the iterative formula Δx = (J T J) -1 J T e updates the pose estimate x, where e is represented as the error vector; while estimating the pose, the point cloud data is fused into the global map. For each new point cloud scan, it is converted to the global coordinate system according to the current estimated pose, and then added to the global map to continuously update and improve the map;

[0071] Step 6. Posture state control analysis: Assume that the state vector of the intelligent robot after fusion is in is represented as the transpose of the position vector, Expressed as the transpose of the attitude vector, Expressed as the transpose of the velocity vector;

[0072] It should be noted that transposition is to swap the rows and columns of a matrix.

[0073] The posture state is predicted based on the data of the internal sensors, the position and velocity are predicted through the integral formula based on acceleration, and the posture state is predicted through the quaternion update formula based on angular velocity;

[0074] Assume the state transfer function is f(X k ,u k ), where X k It is represented as the state vector at the k-th motion moment, u k is represented by the control input obtained through the inertial measurement unit, then the predicted state vector is At the same time, the covariance matrix of the predicted state is calculated According to Kalman filter theory, where F k It is expressed as the Jacobian matrix of the state transfer function f with respect to the state vector X, P k It is represented as the covariance matrix of the state vector at k motion moments, It is represented as the transpose of the covariance matrix of the state vector at k motion moments, Q k It is represented as the covariance matrix of model uncertainty and sensor noise;

[0075] When the estimated value x of the external positioning data is obtained, the state of the intelligent robot is updated. The measurement value of the external positioning system is set to Z k , the measurement function is h(X k ), calculate the measurement residual and the covariance matrix of the measurement residuals Among them, H k It is expressed as the Jacobian matrix of the measurement function h with respect to the state vector X, It is expressed as the transpose of the Jacobian matrix of the measurement function h with respect to the state vector X, R k Expressed as the covariance matrix of the measurement noise;

[0076] Finally calculate the Kalman gain The updated state vector is The updated covariance matrix is in Expressed as the prediction covariance matrix in Kalman filtering, It is represented as the predicted state vector in Kalman filtering, and I is represented as the identity matrix;

[0077] By continuously repeating the prediction and updating steps, the data from internal and external sensors can be effectively fused to obtain a more accurate estimation of the position and posture of the intelligent robot.

[0078] In a specific embodiment, the present invention calculates the posture of the robot by combining the quaternion method with the angular velocity data of the inertial measurement unit. The quaternion has the advantages of avoiding universal joint lock, high computational efficiency and good numerical stability, and can accurately represent the posture change of the robot in three-dimensional space. The quaternion increment is calculated using the angular velocity data and the sampling period, and the posture quaternion vector at each motion moment is obtained by the quaternion update formula. This method can track the posture of the robot in real time, which is very important for tasks that require precise posture control (such as robotic arm operation, flying robot control, etc.);

[0079] Next, based on the acceleration data provided by the inertial measurement unit, combined with the initial speed and position, the speed and position are calculated through an integral formula. Although the inertial measurement unit has errors, it can provide relatively accurate position and speed information in a short time, which is helpful for the robot's instant motion control and preliminary positioning;

[0080] By using the point cloud data of the laser radar, the noise points are removed through voxel filtering, which reduces the redundancy and interference of the data and improves the efficiency and accuracy of subsequent processing. The feature points are extracted by normal vector estimation and curvature calculation, and the feature point pairs are found through the nearest neighbor search algorithm. This method can accurately extract and match the key features in the point cloud data, providing a reliable basis for the positioning and map construction of the robot. By constructing a pose graph and minimizing the error based on the Gauss-Newton method, the pose of the robot at each movement moment can be accurately estimated. This graph optimization-based method can effectively deal with the uncertainty and noise in the laser radar data and obtain high-precision positioning results. While estimating the pose, the point cloud data is integrated into the global map, which can continuously update and improve the map, so that the robot has a more comprehensive and accurate understanding of the environment, which is helpful for path planning and navigation.

[0081] Finally, the fused state vector is defined, and the position, attitude and speed information are comprehensively considered, which can comprehensively describe the robot's motion state and provide a rich data basis for control analysis. By fusing internal and external sensor data based on the Kalman filter method, the real-time nature of internal sensors and the high precision of external sensors can be fully utilized. Kalman filtering can effectively deal with sensor noise and uncertainty, and continuously update and optimize the robot's position and attitude estimation, thereby achieving more precise control. In the prediction step, the position and attitude state is predicted based on the internal sensor data, which can provide a preliminary estimate when there is no external positioning data. In the update step, the prediction results are corrected using external positioning data to ensure the accuracy and reliability of the robot's state estimation. Through in-depth analysis and effective fusion of internal and external sensor data, the intelligent robot can achieve high-precision positioning, attitude control and motion planning in complex environments, thereby improving the robot's autonomy, adaptability and task execution capabilities.

[0082] In summary, in terms of vision, the present invention recognizes images through a target detection algorithm based on deep learning, and further analyzes the pixel features of the image area to obtain an image feature vector, which can help the robot quickly locate and identify targets in complex environments; the optical flow algorithm is used to process video data to obtain a video feature vector, which can effectively capture object motion information and compress features to reduce data processing volume; the image and video feature vectors are fused through an attention mechanism and a shared fully connected layer, and weights are adaptively assigned, which can enable the robot to have a more comprehensive and accurate visual perception of the environment and assist in operational decision-making. In terms of sensor data processing, the quaternion combined with the angular velocity data of the inertial measurement unit can be accurately obtained The robot's posture and acceleration data combined with the initial conditions can provide relatively accurate position and velocity information in a short period of time, which is helpful for instant motion control and preliminary positioning; using lidar point cloud data, through voxel filtering, feature point extraction and matching, pose graph construction and optimization and other operations, high-precision positioning and map construction can be achieved; based on the Kalman filter, internal and external sensor data are fused, and the position, posture and velocity information are comprehensively considered. It can give full play to the advantages of each sensor, effectively deal with noise and uncertainty, and continuously optimize the robot's position and posture estimation, achieve high-precision positioning, posture control and motion planning, and improve the robot's autonomy, adaptability and task execution capabilities in complex environments.

[0083] Embodiment 2:

[0084] Specifically, the present embodiment also discloses an intelligent robot control system based on deep learning, which is applied to the intelligent robot control method based on deep learning proposed in the above embodiment, including: an image data feature analysis module, a video data feature analysis module, a visual feature vector analysis module, an internal sensor data analysis module, an external positioning data analysis module and a posture state control analysis module;

[0085] The image data feature analysis module is used to extract and analyze the image data features to obtain the image feature vector;

[0086] The video data feature analysis module is used to extract and analyze the features of the video data to obtain the video feature vector;

[0087] The visual feature vector analysis module is used to perform comprehensive calculation and analysis on the image feature vector and the video feature vector to obtain a comprehensive visual feature vector;

[0088] The internal sensor data analysis module is used to calculate and analyze the internal sensor data of the intelligent robot to obtain the rate update formula and the position update formula;

[0089] The external positioning data analysis module is used to calculate and analyze the external positioning data of the intelligent robot to obtain the position and posture of the intelligent robot at each movement moment;

[0090] The posture state control analysis module obtains more accurate position and posture estimation of the intelligent robot by continuously repeating the prediction and updating steps and fusing the data of internal and external sensors.

[0091] The above formulas are all dimensionless and numerical calculations. The formula is a formula that is obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The size of the coefficient is to quantify each parameter to obtain a specific value. Regarding the size of the coefficient, it is acceptable as long as it does not affect the proportional relationship between the parameter and the quantified value.

[0092] In addition, those skilled in the art will appreciate that various aspects of the present invention may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present invention may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of the present invention may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0093] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0094] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. An intelligent robot control method based on deep learning, characterized in that: The following steps are involved: Step 1: Extract image data features; Step 2: Extract video data features; Step 3: Visual feature vector fusion; Step 4: Internal sensor data analysis: Assume that the quaternion corresponding to the initial posture of the intelligent robot is q0=(q 0w ,q 0x ,q 0y ,q 0z ), the angular velocity data provided by the inertial measurement unit is ω=(ω x ,ω y ,ω z ), the sampling period is Δt, where ω x ,ω y ,ω z They are expressed as angular velocities around the x, y, and z axes respectively; By formula Calculate the modulus of the angular velocity, and then use the formula Calculate the quaternion increment Δq, and finally update the formula based on the quaternion Calculate the quaternion vector q of the intelligent robot's posture at each motion moment t+1 ; Assume the acceleration provided by the inertial measurement unit is a = (a x , a y , a z ), initial velocity v0=(v 0x , v 0y , v 0z ), the initial position is x0=(x 0x , x 0y , x 0z ), therefore, the velocity update formula is v t+1 =vt+atΔt, the position update formula is x t+1 =xt+vtΔt; Step 5: External positioning data analysis; Step 6: Posture state control analysis.

2. The intelligent robot control method based on deep learning according to claim 1, characterized in that: The step one is specifically as follows: by adopting a target detection algorithm based on deep learning to perform image recognition on the target object in the image, then inputting the recognized image into a visual feature analysis module, dividing the input image into several regions, and calculating the average value, maximum value and minimum value of the pixels in each region at the same time, the feature vector of each region is composed of the average value, maximum value and minimum value of the pixels in each region, and finally the feature vectors of several regions are spliced ​​to obtain an image feature vector. The target detection algorithm can be YOLO or FasterR-CNN.

3. The intelligent robot control method based on deep learning according to claim 1, characterized in that: The step 2 is specifically as follows: by using the optical flow algorithm to calculate the optical flow field for each frame in the video data, the optical flow field is divided into several regions, the mean and standard deviation of the optical flow vector are calculated in each region to obtain a regional optical flow feature vector, and finally the regional optical flow feature vectors of several frames in the video data are temporally pooled to obtain a video feature vector; the optical flow field is a two-dimensional vector field, which represents the movement speed and direction of each pixel in the image, and the temporal pooling of the regional optical flow feature vectors of several frames in the video data can be maximum pooling or average pooling.

4. The intelligent robot control method based on deep learning according to claim 1, characterized in that: The step three is specifically as follows: by constructing an attention mechanism module, inputting the image feature vector and the video feature vector into the attention mechanism module, mapping the image feature vector and the video feature vector to an intermediate representation space through a shared fully connected layer, obtaining the image feature mapping vector and the video feature mapping vector, which are respectively denoted as FI and FV, and finally calculating the comprehensive visual feature vector ST by the formula ST=w1*FI+w2*FV, wherein w1 and w2 respectively represent the weight coefficients corresponding to the image feature mapping vector and the video feature mapping vector, and both w1 and w2 are calculated by using the soft-max function, specifically: where s(·) is represented as the dot product function.

5. The intelligent robot control method based on deep learning according to claim 1, characterized in that: In the step 4, vt in the speed update formula and the position update formula represents the velocity vector at the movement time t, xt represents the position vector at the movement time t, and at represents the acceleration vector at the movement time t.

6. The intelligent robot control method based on deep learning according to claim 1, characterized in that: The specific step 5 is: suppose the point cloud data obtained by the laser radar of the intelligent robot each time is P = {p1, p2, ..., p n }, where P i =(x i ,y i , z i ) is represented as the three-dimensional coordinates of the i-th point, and the voxel filtering method is used to remove noise points. Then the size of the voxel grid is determined, and the point cloud space is divided into a number of cubes with a side length of l, namely voxels. At the same time, the centroid of the point in each voxel is calculated as the representative point of the voxel; Normal vector estimation and curvature calculation are used to extract feature points from the preprocessed point cloud data. i By calculating the covariance matrix of the points in the spherical neighborhood with a radius of r, the covariance matrix is ​​decomposed by eigenvalues. The eigenvector corresponding to the minimum eigenvalue is the point P i The normal vector n i , and according to the neighborhood point to point P i The curvature c is calculated based on the distance and the change of the normal vector i , if the curvature is greater than the set threshold, point P i Marked as feature points, let the extracted feature point set be J = {f1, f2, ..., f m }, in the point cloud data at each motion moment, the corresponding feature point pairs are found through the nearest neighbor search algorithm; The pose graph is constructed based on the matched feature point pairs. The nodes in the pose graph represent the poses of the intelligent robot at each motion moment. Assuming that in a two-dimensional plane, x i =(x ix , x iy ,θ i ), where x ix and x iy are expressed as position coordinates, θ i It is expressed as attitude angle; The constraint relationship between the poses is represented by the edge, and the constraint is calculated by matching feature point pairs. Suppose two matching feature point pairs (f i , f j ) in two different positions x i and x j The coordinate transformation relationship under T is ij , define the error function of the pose graph Where C represents the set of all constraint edges, It is expressed as the coordinate transformation relationship obtained by feature matching, T ij (x i , x j ) is represented by the coordinate transformation relationship calculated according to the current posture; By using the Gauss-Newton method to minimize the error function E(x), we first calculate the Jacobian matrix J of the error function with respect to the pose variable x, and then use the iterative formula Δx = (J T J) -1 J T e updates the pose estimate x, where e is represented as the error vector. For feature point f i , find the point f closest to it in the point cloud data at another time j , if their distance is less than the set threshold, it is considered that (f i , f j ) is a pair of matching feature points.

7. The intelligent robot control method based on deep learning according to claim 6, characterized in that: In step 5, when estimating the pose, the point cloud data is fused into the global map. For each new point cloud scan, it is converted to the global coordinate system according to the current estimated pose and then added to the global map to continuously update and improve the map.

8. The intelligent robot control method based on deep learning according to claim 1, characterized in that: The specific step six is: suppose the state vector of the intelligent robot after fusion is in is represented as the transpose of the position vector, Expressed as the transpose of the attitude vector, Expressed as the transpose of the velocity vector; The posture state is predicted based on the data of the internal sensors, the position and velocity are predicted through the integral formula based on acceleration, and the posture state is predicted through the quaternion update formula based on angular velocity; Assume the state transfer function is f(X k ,u k ), where X k It is represented as the state vector at the k-th motion moment, u k is represented as the control input obtained through the inertial measurement unit, then the predicted state vector is At the same time, the covariance matrix of the predicted state is calculated According to Kalman filter theory, where F k It is expressed as the Jacobian matrix of the state transfer function f with respect to the state vector X, P k It is represented as the covariance matrix of the state vector at k motion moments, It is represented as the transpose of the covariance matrix of the state vector at k motion moments, Q k It is represented as the covariance matrix of model uncertainty and sensor noise; When the estimated value x of the external positioning data is obtained, the state of the intelligent robot is updated. The measurement value of the external positioning system is set to Z k , the measurement function is h(X k ), calculate the measurement residual and the covariance matrix of the measurement residuals Among them, H k It is expressed as the Jacobian matrix of the measurement function h with respect to the state vector X, It is expressed as the transpose of the Jacobian matrix of the measurement function h with respect to the state vector X, R k Expressed as the covariance matrix of the measurement noise; Finally calculate the Kalman gain The updated state vector is The updated covariance matrix is in Expressed as the prediction covariance matrix in Kalman filtering, It is represented as the predicted state vector in Kalman filtering, I is represented as the identity matrix; transposition is to interchange the rows and columns of the matrix.

9. The intelligent robot control method based on deep learning according to claim 8, characterized in that: In step six, by continuously repeating the prediction and update steps, the data of internal and external sensors are integrated to obtain a more accurate estimation of the position and posture of the intelligent robot.

10. A control system applied to the deep learning-based intelligent robot control method according to any one of claims 1 to 9, characterized in that: include: Image data feature analysis module, video data feature analysis module, visual feature vector analysis module, internal sensor data analysis module, external positioning data analysis module and posture state control analysis module; The image data feature analysis module is used to extract and analyze the image data features to obtain the image feature vector; The video data feature analysis module is used to extract and analyze the features of the video data to obtain the video feature vector; The visual feature vector analysis module is used to perform comprehensive calculation and analysis on the image feature vector and the video feature vector to obtain a comprehensive visual feature vector; The internal sensor data analysis module is used to calculate and analyze the internal sensor data of the intelligent robot to obtain the rate update formula and the position update formula; The external positioning data analysis module is used to calculate and analyze the external positioning data of the intelligent robot to obtain the position and posture of the intelligent robot at each movement moment; The posture state control analysis module obtains more accurate position and posture estimation of the intelligent robot by fusing the data of internal and external sensors through continuous repetition of prediction and update steps.

Images