Mobile robot steering control method and system based on human body center of gravity detection

Abstract

The present application relates to a kind of mobile robot steering control method and system based on human body center of gravity detection, method includes: the depth image and color image of the person being followed are acquired;Key point information of human body is extracted from the color image using lightweight OpenPose algorithm, the key point information includes key point coordinates;Based on the coordinates of each key point and coefficient, the position of human body center of gravity is obtained;With the region surrounded by shoulder and hip key point as selected region, the depth value of selected region is extracted from the depth image, and the man-machine distance is calculated according to the depth value;Based on the position of human body center of gravity and the man-machine distance, the drift angle of pedestrian is calculated;Based on the drift angle of pedestrian, the mobile robot is controlled to steer.The present application is based on lightweight OpenPose algorithm to measure distance, and improve human body center of gravity calculation method, the drift angle of pedestrian is calculated, and the accuracy and stability of mobile robot steering control are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile robots, in particular to a mobile robot steering control method and system based on human body center of gravity detection. BACKGROUND

[0002] With the rapid development of the information age, in recent years, smart phones, intelligent monitoring, notebook computers and other electronic products have rapidly popularized, and the data such as text information, voice information, images and videos brought by them also show explosive growth. Among them, image data is the most important part of human understanding of knowledge through vision. In the face of increasing image data, manual image processing and analysis is costly and inefficient, and has strong subjective consciousness in the processing process, resulting in omissions in the result analysis, and the processing efficiency and accuracy are far from that of computers. Computer vision technology simulates human visual sense to analyze image data like humans. Typical computer vision technologies include target segmentation, target tracking, target detection, target classification, etc. How to use computer vision technology for rapid image processing is the main task of current computer vision research.

[0003] Human pose estimation is an important task in computer vision, aiming to locate the positions of key points in images or videos and connect them into corresponding human poses, which is the basis and premise of advanced vision tasks such as behavior recognition, pedestrian re-identification and pedestrian detection. This task can be used as the basis for other computer vision tasks. Early human pose estimation research methods cannot be applied in actual scenarios, and the rapid development of deep learning and convolutional neural networks has promoted the research of human pose estimation, making it widely used in intelligent monitoring, human-computer interaction, motion analysis and other fields. The combination of human pose estimation and neural networks solves the complex tasks that traditional image recognition cannot handle, and with the significant improvement of hardware devices such as GPU computing power, not only can human pose estimation in static images be realized, but also human pose estimation in videos can be implemented. Currently, both single-person human pose estimation and multi-person pose estimation algorithms rely on human target detection to achieve human pose estimation through human detection boxes. However, the detection box will change due to the motion amplitude of the person to be detected, which leads to inconsistent size of the detection box, easily leading to errors in human key point detection. Therefore, it is necessary to improve the original human pose estimation algorithm to achieve accurate human pose estimation tasks.

[0004] In the process of following pedestrians, mobile robots need to use the pedestrian yaw angle for steering control. The traditional method calculates the pedestrian yaw angle by using the center of the detection box. Affected by the posture of the pedestrian, this method cannot accurately represent the center of gravity of the pedestrian, thereby causing steering errors of the mobile robot. Therefore, in the process of following pedestrians, a method that can solve the steering angle of the mobile robot is needed to solve the problems faced in current real life. SUMMARY

[0005] The technical problem solved by the present application is to overcome the defects of the prior art, provide a mobile robot steering control method based on human body center of gravity detection, which is based on light OpenPose algorithm for ranging, and improves the human body center of gravity calculation method, calculates the yaw angle of the pedestrian, and improves the accuracy and stability of the mobile robot steering control.

[0006] To solve the above technical problems, the technical scheme of the present application is: a mobile robot steering control method based on human body center of gravity detection, comprising:

[0007] Obtain the depth image and color image of the followed pedestrian;

[0008] Extract human key point information from the color image using a light OpenPose algorithm, the key point information including key point coordinates;

[0009] Based on the coordinates and coefficients of each key point, the human body center of gravity position is obtained;

[0010] Take the area surrounded by shoulder and hip joint key points as the selected area, extract the depth value of the selected area from the depth image, and calculate the human-machine distance according to the depth value;

[0011] Based on the human body center of gravity position and the human-machine distance, the pedestrian yaw angle is calculated;

[0012] Based on the pedestrian yaw angle, the mobile robot is controlled to steer.

[0013] Further, the human key point information extracted from the color image using the light OpenPose algorithm comprises:

[0014] The human key point information extracted from the color image using the light OpenPose algorithm comprises:

[0015] The MobileNet network extracts features from the color image to obtain a feature map F1 with the same size as the color image, and the calculation formula is:

[0016] F1=MNT(X,W mnt )

[0017] Where X represents the input color image, MNT() represents the MobileNet network with a hole convolution, W mnt represents a learnable parameter matrix;

[0018] The CPM module obtains a feature map F2 by performing convolution and pooling operations on the input feature map F1 in multiple stages, and fuses the feature map F2 of the current stage with the feature map F1 of the previous stage to output a feature map F3 containing all key point position information.

[0019] In the initial stage, a 3x3 convolution layer is used to perform preliminary feature extraction on the feature map F3, and then a 1x1 convolution is used to perform channel mixing to generate a key point heat map branch S 1 and a partial affinity field branch L 1 The calculation formula is:

[0020] S 1 =ρ 1 (F4)

[0021] L 1 =φ 1 (F4)

[0022] Where F4 is obtained by performing preliminary feature extraction on the feature map F3 using a 3x3 convolution layer in the initial stage, and represents the initial stage key point heat map prediction network, represents the initial stage partial affinity field prediction network. 1 1

[0023] The feature map F3 and the initial stage key point heat map branch S 1 and the partial affinity field branch L 1 are outputted and spliced to obtain a spliced feature F c The calculation formula is:

[0024] F c =Cat(F3,S 1 ,L 1 )

[0025] Where Cat() represents the splicing operation.

[0026] The spliced feature F c is inputted into the refinement stage network to generate key point information, and the calculation formula is:

[0027] S 2 =ρ 2 (F c )

[0028] L 2 =φ 2 (F c )

[0029] Where represents the refinement stage key point heat map prediction network, and represents the refinement stage partial affinity field prediction network. 2 2 ​​​​

[0030] The feature map F3 and the refined stage key point heat map branch S 2 and the partial affinity field branch L 2 The output fusion of the partial affinity field branch L

[0031] Further, each key point is respectively a head center, a shoulder connecting line midpoint, a left shoulder, a left hand elbow, a left hand wrist, a right shoulder, a right hand elbow, a right hand wrist, a left hip, a left knee, a left ankle, a right hip, a right knee, a right ankle, and a two-hip connecting line center.

[0032] Further, the calculating the human-robot distance according to the depth value comprises:

[0033] P oi represents a depth value of an i-th pixel in the selected region, and the human-robot distance E o is calculated according to the following formula:

[0034]

[0035] wherein n represents a total amount of depth pixels in the selected region.

[0036] Further, the calculating the pedestrian yaw angle based on the human body center position and the human-robot distance comprises:

[0037] O'-x'y' is defined as an image pixel coordinate system, a coordinate origin of which is a center of an imaging plane, an x' axis of which is parallel to a horizontal direction of the image and a positive direction of which is leftward, and a y' axis of which is parallel to a vertical direction of the image and a positive direction of which is downward; O-xyz is a camera coordinate system, a coordinate origin of which is a center of a camera lens, an x axis and a y axis of which are respectively parallel to an x' axis and a y' axis of the image pixel coordinate system, and a z axis of which is a camera optical axis and is connected with the image pixel coordinate system; P(P x ,P y ) is a center of gravity of a followed pedestrian, lines parallel to the x' axis of which intersect the y' axis at a point Q and a point A on a left edge of the image, respectively, and QO and AO are connected, and an included angle between QO and AO is ζ, and a calculation formula is as follows:

[0038]

[0039]

[0040] An included angle between PO and QO is a pedestrian yaw angle The formula is as follows:

[0041]

[0042]

[0043] wherein the human-robot distance is taken as a length of OP.

[0044] The application also provides a mobile robot turning control system based on human body center detection, comprising:

[0045] An acquisition module is configured to acquire a depth image and a color image of a followed pedestrian;

[0046] A key point detection module is configured to extract human body key point information from the color image by using a lightweight OpenPose algorithm, wherein the key point information comprises key point coordinates; and a region surrounded by shoulder and hip joint key points is regarded as a selected region.

[0047] An extraction module is configured to extract a depth value of the selected region from the depth image.

[0048] A calculation module is configured to calculate a human body center position based on the coordinates of each key point and a coefficient, calculate a human-robot distance based on the depth value, and calculate a pedestrian yaw angle based on the human body center position and the human-robot distance.

[0049] A control module is configured to perform turning control on the mobile robot based on the pedestrian yaw angle.

[0050] After the above technical solution is adopted, the application has the following beneficial effects:

[0051] 1. The application performs distance measurement based on a lightweight OpenPose algorithm, improves a human body center calculation method, and performs pedestrian yaw angle calculation, thereby improving the accuracy and stability of robot turning control. Finally, the algorithm is applied to a mobile robot platform to achieve automatic following effect. In the following process, the relative pose of the target center and the mobile robot jointly determines the motion state.

[0052] 2. The application adopts a multiplication coefficient method to calculate the human body center position, which can calculate the human body center position more quickly and has high stability. The yaw angle calculation method based on the pedestrian detection frame center is easily affected by the limb action, and the target deformation and center deviation problems occur, thereby leading to poor algorithm robustness. Compared with the yaw angle calculation method based on the pedestrian detection frame center, the yaw angle curve based on the human body center of the application is relatively smooth. Considering that the center of gravity can accurately reflect the human body motion trend and has high robustness to the limb action, the yaw angle calculation method based on the human body center proposed by the application can provide the stability of pedestrian following. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A flowchart of the mobile robot turning control method based on human body center detection of the application;

[0054] Figure 2 A network structure diagram of the lightweight OpenPose algorithm of the application;

[0055] Figure 3 The algorithm structure for acquiring key point information and human-robot distance by using a lightweight OpenPose algorithm is used in the application.

[0056] Figure 4 The human key point constructed in the application;

[0057] Figure 5 The yaw angle model is used in the application. DETAILED DESCRIPTION

[0058] In order to make the content of the application more easily understood, the application is further described in detail below according to specific embodiments and in combination with the accompanying drawings.

[0059] Embodiment one

[0060] As shown in Figure 1 , 2 , 3, 4, 5, a mobile robot turning control method based on human center of gravity detection comprises:

[0061] S1, acquiring a depth image and a color image of a followed pedestrian;

[0062] S2, acquiring human key point information from the color image by using a lightweight OpenPose algorithm, wherein the key point information comprises key point coordinates;

[0063] S3, obtaining a human center of gravity position based on the coordinates of each key point and a coefficient; taking a region surrounded by shoulder and hip joint key points as a selected region, acquiring a depth value of the selected region from the depth image, and calculating a human-robot distance according to the depth value;

[0064] S4, calculating a pedestrian yaw angle based on the human center of gravity position and the human-robot distance;

[0065] S5, performing turning control on the mobile robot based on the pedestrian yaw angle.

[0066] The lightweight OpenPose is a network based on a bottom-up structure design, as shown in Figure 2As shown, the forward inference speed of the lightweight OpenPose will not be affected by the number of people in the image. The lightweight OpenPose first extracts the features in the image by a backbone network, and then sends the features into the CPM network. The initialization network is composed of a partial affinity field (PAF) network and a keypoint heat map network, the former is used to encode the key points into 2D vectors, which contains the position and direction information of the limbs, and the latter is used to represent the confidence of each key point, which matches the corresponding key point pairs in different human bodies. The lightweight OpenPose takes image input and outputs the key points of pedestrians. The difference between the lightweight OpenPose and the OpenPose is that the MobileNet is used as the backbone network, and the dilated convolution is added in the convolution process to improve the receptive field of the backbone network.

[0067] The CPM network is composed of a fully convolutional network for further extracting target body position and direction information to estimate key point information. In the initial stage, the convolution operation is used as a shared feature extraction network, and then divided into two parts of partial affinity field and key points, respectively, to extract the partial affinity field and the partial confidence map. In order to further reduce the spatial and temporal complexity of the network, the refinement stage is realized by multiple convolutions. In the model training process, the loss functions of the initial stage and the refinement stage are:

[0068]

[0069]

[0070] wherein, and respectively represent the loss functions of the keypoint heat map branch in the initial stage and the refinement stage, and respectively represent the loss functions of the partial affinity field branch in the initial stage and the refinement stage, represents the real keypoint confidence map, represents the real affinity field vector, p represents the key point, and w represents the mask. The overall loss function of the lightweight OpenPose network is as follows:

[0071]

[0072] The lightweight OpenPose algorithm does not require a cumbersome preprocessing process, and reduces the key point detection error caused by distributed training, so the lightweight OpenPose is selected as the key point detection algorithm in this paper, which directly extracts the target human skeleton from the color image in real time.

[0073] In one embodiment, as shown in Figure 3 , the human key point information is extracted from the color image by using the lightweight OpenPose algorithm, including:

[0074] S21, the MobileNet network extracts features from the color image to obtain a feature map F1 of the same size as the color image, and the calculation formula is:

[0075] F1=MNT(X,W mnt )

[0076] wherein X represents the input color image, MNT() represents the MobileNet network with a hole convolution, W mnt represents a learnable parameter matrix;

[0077] S22, the CPM module obtains a feature map F2 by performing convolution and pooling operations on the input feature map F1 through multiple stages, and fuses the feature map F2 of the current stage with the feature map F1 of the previous stage to improve the accuracy of pose estimation. Finally, the CPM module outputs a feature map F3 containing all key point position information;

[0078] S23, in the initial stage, a 3x3 convolution layer is used to perform preliminary feature extraction on the feature map F3, and the 3x3 convolution kernel size is relatively small, but has a large receptive field, which can effectively capture local features, and then a 1x1 convolution is used for channel mixing to generate a key point heat map branch S 1 and a partial affinity field branch L 1 , and the calculation formula is:

[0079] S 1 =ρ 1 (F4)

[0080] L 1 =φ 1 (F4)

[0081] wherein F4 is obtained by performing preliminary feature extraction on the feature map F3 through a 3x3 convolution layer, ρ 1 () represents the initial stage key point heat map prediction network, and φ 1 () represents the initial stage partial affinity field prediction network;

[0082] S24, the feature map F3 and the initial stage key point heat map branch S 1 and the partial affinity field branch L 1 are outputted and spliced to obtain a spliced feature F c , and the calculation formula is:

[0083] F c =Cat(F3,S 1 ,L 1 )

[0084] wherein Cat() represents a splicing operation;

[0085] S25, splicing feature F c The key point information is generated by feeding into the refinement stage network, and the calculation formula is:

[0086] S 2 = p 2 (F c )

[0087] L 2 = f 2 (F c )

[0088] Wherein, p 2 () represents the key point heat map prediction network of the refinement stage, and f 2 () represents the partial affinity field prediction network of the refinement stage.

[0089] S26, the outputs of the feature map F3 and the key point heat map branch S 2 and the partial affinity field branch L 2 are fused to obtain the key point information.

[0090] According to the human kinematics principle and the key point information, the chest region of the human body is relatively complete. At the same time, in order to avoid the influence of background pixels on the ranging accuracy, the embodiment selects the part surrounded by the shoulder and hip joint key points as the selected region, and extracts the depth value from the depth image guided by this region, so as to perform key point ranging.

[0091] In one embodiment, the human-machine distance is calculated according to the depth value, comprising:

[0092] Define P oi represents the depth value of the i-th pixel in the selected region, and the formula of the human-machine distance E o is:

[0093]

[0094] Wherein, n represents the total amount of depth pixels in the selected region.

[0095] In the traditional way, there are two methods for the human body model to calculate the center of gravity, which are analysis method and synthesis method, and the analysis method is the most widely used. The analysis method calculates the center of gravity of the human body according to the center of gravity coordinate formula and the relationship between the weight of each part of the body and the center position of each part. This requires inputting the center position coordinates of each part into the computer, and calculating the center of gravity of each part through the center position coordinates of each part. The synthesis method is to calculate the center of gravity by torque synthesis. The theoretical basis of the human body center of gravity calculation method based on torque synthesis is the theorem of torque: in any plane, the torque of the resultant force is the algebraic sum of the torque of each component force, and the calculation formula of the torque of the resultant force is as follows:

[0096] F·L = F1·L1 + F2·L2 + F3·L3 + … + F n ·L n =∑F i ·L i

[0097] Where F represents the resultant force, F i represents the i-th component force, and L represents the perpendicular distance from the resultant force line to the center of rotation, L i represents the perpendicular distance from the i-th component force line to the center of rotation. In the image plane, the human body center of gravity position is calculated by the skeleton information, and the center of gravity position is represented by longitudinal and transverse coordinates, and the calculation method is as follows:

[0098] G·X = G1·X1 + G2·X2 + G3·X3 + … + G n ·X n =∑G i ·X i

[0099] G·Y = G1·Y1 + G2·Y2 + G3·Y3 + … + G n ·Y n =∑G i ·Y i

[0100] Where G represents the weight of the human body, G i represents the weight of the i-th body skeleton, X and Y represent the longitudinal and transverse coordinates of the center of gravity of the human body, X i and Y i represent the center of gravity coordinates of the i-th skeleton. Given the weight and center of gravity coordinates of each skeleton, the center of gravity coordinates of the human body are:

[0101]

[0102]

[0103] The above traditional human body center of gravity calculation method needs to know the weight and center of gravity coordinates of each limb, and the center of gravity coordinates can be obtained by calculating the midpoint from the lightweight OpenPose output key point coordinates. However, this method has the problems of being unable to accurately obtain the weight of each limb and being complicated in calculating the center of gravity coordinates of the limbs.

[0104] In one embodiment, in combination with the key point information obtained by the lightweight OpenPose, the multiplication coefficient method can more quickly calculate the human body center of gravity position (X, Y), and the formula is:

[0105] X =∑k i ·x i

[0106] Y =∑k i ·y i

[0107] wherein (x i , y i ) is the coordinate of the i-th key point, k i is the coefficient of the i-th key point.

[0108] Compared with the method of calculating the center of gravity of the human body based on the synthesis of the moment, the multiplication coefficient method uses the multiplication of the key point coordinates and the coefficient to avoid calculating the weight and the center of gravity of the limb, simplifies the calculation process of the center of gravity, and is conducive to obtaining the position of the center of gravity of the human body in real time.

[0109] In one embodiment, the constructed human key point model is as shown in Figure 4 , and the construction process is as follows:

[0110] (1) In the process of following the pedestrian, the pedestrian faces away from the camera, so this embodiment selects the center of gravity of the double-ear connecting line as the head center of gravity.

[0111] (2) In the process of following the pedestrian, the palm weight cannot be detected, according to the biological principle of the human body, the palm is in a curved state during walking, so this embodiment adds the hand center of gravity coefficient and the wrist coefficient as a new wrist coefficient.

[0112] (3) OpenPose cannot detect the heel, and the ankle is used instead.

[0113] (4) Lightweight OpenPose cannot detect the tail of the spine, so this embodiment uses the two hip center points instead.

[0114] According to the proportion of the weight of the human body occupied by the area where each key point is located, each key point in this paper is respectively the head center of gravity, the midpoint of the shoulder connecting line, the left shoulder, the left elbow, the left wrist, the right shoulder, the right elbow, the right wrist, the left hip, the left knee, the left ankle, the right hip, the right knee, the right ankle and the center of the two hip connecting lines, and the coefficients are 0.0306, 0.1393, 0.0356, 0.0580, 0.0372, 0.0356, 0.0580, 0.0372, 0.1083, 0.063, 0.019, 0.1083, 0.063, 0.019 and 0.1879, respectively.

[0115] Generally, according to the proportion of the weight of the human body occupied by each key point, the coefficients of each key point are distributed according to the proportion of the body weight occupied by each key point of the human body.

[0116] In this embodiment, the key points use the lightweight OpenPose model, which only detects 15 key points, and the detection speed is obviously improved under the condition of losing some accuracy.

[0117] The specific formula for calculating the position (X, Y) of the center of gravity of the human body is:

[0118]

[0119]

[0120] wherein, k i represents the human key point coefficient after building the model, x i and y i represent the key point coordinates of the lightweight OpenPose detection.

[0121] The change of the human body gravity center coordinates is the embodiment of the human body motion trend. Therefore, the embodiment selects the yaw angle in the process of following the pedestrian to calculate the human body gravity center. The yaw angle represents the angle deviation of the human body gravity center and the image gravity center in the world coordinate system, as shown in Figure 5 In the process of following the pedestrian, the pedestrian motion and the robot motion jointly determine the size of the yaw angle. The larger the yaw angle is, the farther the pedestrian gravity center is from the image gravity center, which indicates that the pedestrian is turning. The smaller the yaw angle is, the closer the pedestrian gravity center is to the image center, which indicates that the pedestrian is walking in a straight line or keeping still.

[0122] The embodiment proposes that the calculation method of the human body gravity center adopts the multiplication coefficient method to calculate the human body gravity center position more quickly, has high stability, avoids the problem that the yaw angle calculation method based on the pedestrian detection frame center is easily affected by the limb action, target deformation and center deviation, thereby causing poor algorithm robustness, and compared with the yaw angle calculation method based on the pedestrian detection frame center, the yaw angle curve based on the human body gravity center of the embodiment is relatively smooth. Considering that the gravity center can more accurately reflect the human body motion trend and has high robustness to the limb action, the yaw angle calculation method based on the human body gravity center proposed in the embodiment can provide the stability of the pedestrian following.

[0123] In one embodiment, based on the human body gravity center position and the human-robot distance, the pedestrian yaw angle is calculated, comprising:

[0124] Defining O'-x'y' is the image pixel coordinate system, the coordinate origin is the center of the imaging plane, the x' axis is parallel to the horizontal direction of the image and the positive direction is to the left, and the y' axis is parallel to the vertical direction of the image and the positive direction is downward; O-xyz is the camera coordinate system, the coordinate origin is the center of the camera lens, the x axis and the y axis are respectively parallel to the x' axis and the y' axis of the image pixel coordinate system, and the z axis is the camera optical axis, which is connected with the image pixel coordinate system, P(P x ,P y ) is the gravity center of the followed pedestrian, which is parallel to the x' axis and intersects the y' axis at Q point and the left edge of the image at A point, and QO and AO are connected, then the included angle of QO and AO is ζ, and the calculation formula is as follows:

[0125]

[0126]

[0127] The angle between PO and QO is the pedestrian yaw angle The formula is:

[0128]

[0129]

[0130] wherein the human-robot distance is taken as the length of OP.

[0131] Embodiment Two

[0132] A mobile robot steering control system based on human body center of gravity detection, comprising:

[0133] An acquisition module for acquiring a depth image and a color image of a followed pedestrian;

[0134] A key point detection module for extracting human body key point information from the color image by using a lightweight OpenPose algorithm, the key point information comprising key point coordinates; wherein a region surrounded by shoulder and hip joint key points is taken as a selected region;

[0135] An extraction module for extracting a depth value of the selected region from the depth image;

[0136] A calculation module for calculating a human body center of gravity position based on the coordinates of each key point and a coefficient, calculating a human-robot distance according to the depth value, and calculating a pedestrian yaw angle based on the human body center of gravity position and the human-robot distance;

[0137] A control module for performing steering control on the mobile robot based on the pedestrian yaw angle.

[0138] In the present embodiment, the acquisition module adopts a Le Vision LeTMC-520 RGB-D camera, and the Le Vision LeTMC-520 RGB-D camera is taken as a basis for environmental perception. The depth information of a photographed object is acquired by using a structured light 3D imaging technology, and a color (RGB) image is output at the same time. The structured light depth camera emits laser speckles with certain structural features to the outside world, receives the distorted speckle images reflected back through an infrared camera, and thus calculates the image depth. The Le Vision depth camera can output color images and depth images in real time, the imaging size is 640*480, the output frame rate is 30 FPS, the interface protocol is USB2.0, supports Linux system, and meets the development requirements of the present text.

[0139] The mobile robot communicates with the host computer through a USB serial port, acquires color images and depth images, obtains the position and pose information of the pedestrian using ranging and yaw angle algorithms, and sends the information to an STM32 single-chip microcomputer. After receiving the distance and yaw angle of the pedestrian, the single-chip microcomputer decomposes the information through PID, sends the control command to a direct current motor, and drives the robot to follow the pedestrian in front.

[0140] The software platform of the embodiment is built based on a robot operating system (ROS), specifically, a ROS Melodic architecture of an Ubuntu 18.04 system is used, and the ROS mobile platform robot supports six control modes, including APP remote control, PS2 wired handle, ROS control, model remote control, CAN control and serial port control. The control modes are displayed in the lower left corner of an OLED display screen, and the ROS control mode is enabled by default. After the initialization of the car is completed, the car will move forward a distance to detect whether the movement of the car is problematic, and if there is a problem, the car will be prohibited from being controlled. The above algorithm is transplanted on the ROS robot car to realize the function of pedestrian following.

[0141] With the above ideal embodiments according to the present application as inspiration, through the above description, relevant personnel can make various changes and modifications without deviating from the technical idea of the present application. The technical scope of the present application is not limited to the content in the specification, and must be determined according to the scope of claims.

Claims

1.A mobile robot steering control method based on human body center detection, comprising: obtaining a depth image and a color image of a person being followed; extracting human key point information from the color image using a lightweight OpenPose algorithm, the key point information including key point coordinates; obtaining a human body center position based on the coordinates of each key point and a coefficient; extracting a depth value of a selected region from the depth image, the selected region being surrounded by shoulder and hip joint key points, and calculating a human-robot distance based on the depth value; calculating a pedestrian yaw angle based on the human body center position and the human-robot distance; controlling the steering of the mobile robot based on the pedestrian yaw angle; wherein the calculation of the pedestrian yaw angle based on the human body center position and the human-robot distance comprises: 2.The mobile robot steering control method based on human body center detection according to claim 1, wherein the extraction of the human key point information from the color image using the lightweight OpenPose algorithm comprises: Definitions is the image pixel coordinate system, with the origin at the center of the imaging plane, is the horizontal axis of the image pixel coordinate system, with the positive direction to the left, is the vertical axis of the image pixel coordinate system, with the positive direction downward; is the camera coordinate system, with the origin at the center of the camera lens, is the horizontal axis of the camera coordinate system, is the vertical axis of the camera coordinate system, is the horizontal axis of the camera coordinate system, is the vertical axis of the camera coordinate system, is the optical axis of the camera, connected to the image pixel coordinate system, is the center of gravity of the followed pedestrian, parallel to the axis, is the intersection of the axis and the left edge of the image, is the intersection of the axis and the right edge of the image, is the intersection of the axis and the top edge of the image, is the intersection of the axis and the bottom edge of the image, the angle between the and the is , and the calculation formula is as follows: ； ； and the angle between the line and the pedestrian yaw angle , the formula is: ； ； wherein the human-machine distance is used as the length of the line. a CPM module obtains a feature map F2 by convolution and pooling operations in multiple stages from an input feature map F1, and fuses the feature map F2 of the current stage with the feature map F1 of the previous stage to output a feature map F3 containing all key point position information. 3.The mobile robot steering control method based on human body center detection according to claim 1, wherein each key point is a head center, a midpoint of a shoulder connecting line, a left shoulder, a left elbow, a left wrist, a right shoulder, a right elbow, a right wrist, a left hip, a left knee, a left ankle, a right hip, a right knee, a right ankle, and a center of a line connecting the two hips. The MobileNet network extracts features of the color image to obtain a feature map with the same size as the color image The calculation formula is: ； wherein, represents an input color image, represents a MobileNet network with a dilated convolution, represents a learnable parameter matrix; 4.The mobile robot steering control method based on human body center detection according to claim 1, wherein the calculation of the human-robot distance based on the depth value comprises: In the initial stage, a 3x3 convolution layer is used to perform preliminary feature extraction on the feature map F3, and then a convolution of is used to perform channel mixing to generate a key point heat map branch and a partial affinity field branch , and the calculation formula is: ； ； Wherein, F4 is a preliminary feature extraction of the feature map F3 by a 3x3 convolutional layer in the initial stage, denotes an initial stage key point heat map prediction network, denotes an initial stage partial affinity field prediction network; feature map 3 with initial stage keypoint heat map branch and partial affinity field branch output concatenation, resulting in concatenated features the calculation formula is: ； wherein represents a splicing operation; Stitching features The input is fed into a refinement stage network to generate keypoint information, computed by the formula: ； ； wherein, denotes a refinement stage keypoint heat map prediction network, denotes a refinement stage partial affinity field prediction network; feature map 3 with the refinement stage keypoint heatmap branch and the partial affinity field branch The outputs of the branches are fused to obtain the keypoint information. 5.A mobile robot steering control system based on human body center detection, comprising: an obtaining module for obtaining a depth image and a color image of a person being followed; a key point detection module for extracting human key point information from the color image using a lightweight OpenPose algorithm, the key point information including key point coordinates; wherein a region surrounded by shoulder and hip joint key points is used as a selected region; an extraction module for extracting a depth value of the selected region from the depth image; Definitions Let D(i) represent the depth value of the i-th pixel in the selected region, then the human-machine distance is given by the formula: ； wherein, represents the total amount of depth pixels of the selected region. a calculation module for calculating a human body center position based on the coordinates of each key point and a coefficient, and calculating a human-robot distance based on the depth value, and calculating a pedestrian yaw angle based on the human body center position and the human-robot distance; a control module for controlling the steering of the mobile robot based on the pedestrian yaw angle; wherein the calculation of the pedestrian yaw angle based on the human body center position and the human-robot distance comprises: ​ ​ ​ ​ ​ Definitions is the image pixel coordinate system, with the origin at the center of the imaging plane, is the horizontal axis of the image pixel coordinate system, with the positive direction to the left, is the vertical axis of the image pixel coordinate system, with the positive direction downward; is the camera coordinate system, with the origin at the center of the camera lens, is the horizontal axis of the camera coordinate system, is the vertical axis of the camera coordinate system, respectively corresponding to the horizontal and vertical axes of the image pixel coordinate system, is the horizontal axis of the camera coordinate system, is the vertical axis of the camera coordinate system, parallel to the vertical axis of the image pixel coordinate system, is the optical axis of the camera, connecting the origin of the camera coordinate system to the origin of the image pixel coordinate system, is the center of gravity of the followed pedestrian, parallel to the horizontal axis of the camera coordinate system, is the line connecting the origin of the camera coordinate system to the center of gravity of the followed pedestrian, is the intersection of the line and the horizontal axis of the image pixel coordinate system, is the intersection of the line and the left edge of the image, is the intersection of the line and the right edge of the image, is the intersection of the line and the top edge of the image, is the intersection of the line and the bottom edge of the image, is the angle between the line and the horizontal axis of the image pixel coordinate system, is the angle between the line and the vertical axis of the image pixel coordinate system, is the angle between the line and the horizontal axis of the image pixel coordinate system, calculated as follows: ； ； and The included angle is the pedestrian yaw angle. The formula is: ； ； wherein the human-machine distance is used as a length of the line.

Images