A dynamic target tracking method for a legged robot

Abstract

The application discloses a kind of foot type robot dynamic target tracking method.First, the image and video in the process of walking of the pedestrian wearing mask, not wearing mask and not wearing mask correctly are collected, and the data set for the detection of the mask wearing of the pedestrian is built.Second, the target recognition module based on recurrent neural network is used to determine whether the pedestrian wears a mask, and the face position coordinates of the pedestrian not wearing a mask in the current frame are obtained.Third, the residual network is used as the reference network to extract the face semantic features of the pedestrian not wearing a mask in the current image frame, and predict the face semantic features of the pedestrian in the next image frame.Finally, a target tracking module based on twin network is designed, and the correlation coefficient between the face position semantic feature mapping of the pedestrian in the current frame and the next frame is calculated to track the pedestrian not wearing a mask, thereby realizing the tracking of the dynamic target by the foot type robot.The application can be applied to the foot type robot inspection process under the condition of epidemic prevention.

Description

Technical Field

[0001] This invention relates to the field of target detection, specifically to a method for dynamic target tracking of a legged robot. Background Technology

[0002] Using legged robots for security inspections in densely populated areas to accurately identify and track important targets has significant application value and practical significance.

[0003] Vision-based dynamic target tracking is a fundamental research problem in computer vision and is being widely applied in many fields such as smart monitoring, industrial inspection, and human-computer interaction. It has significant research and application value, but it still faces many challenges in complex scenarios, such as blurred images caused by vibration of the vision sensor, mutual occlusion of targets, significant changes in ambient lighting, and background interference.

[0004] In recent years, in-depth research into deep learning algorithms has further improved environmental perception performance, even reaching or surpassing human recognition accuracy in tasks such as target classification. Consequently, many deep learning-based target tracking methods have been proposed. Compared to traditional methods, deep learning-based dynamic target tracking methods have attracted widespread attention from researchers due to their superior accuracy and robustness.

[0005] Large variations in target scale and aspect ratio are very common in dynamic target tracking. Traditional methods first identify dynamic targets from input video image frames, generate bounding boxes of the target's location, determine the target type, and compare it with the next frame of the video. Tracking is then achieved based on the correlation coefficient between the two frames. However, during dynamic target tracking, the target scale and aspect ratio change with the movement of the target or the visual sensor, resulting in inaccurate hyperparameters of the generated bounding box containing the target. Therefore, it is difficult to effectively match the same target in two consecutive frames, making it hard to determine the specific location of the target in the next frame. Tracking performance is also poor when target movement or visual sensor vibration causes image blurring.

[0006] In recent years, a series of deep learning-based dynamic target tracking methods have been proposed. For example, Bertinetto et al. proposed using a Siamese Network for dynamic target recognition at the European Conference on Computer Vision. This network includes recognition and regression branches, which are used to obtain the semantic features of the target in the next frame and calculate the correlation coefficient between the semantic features of the target in the current frame and the next frame, respectively. The network then matches the same target in the two consecutive frames based on the obtained correlation coefficient. Experiments have shown that this method achieves superior dynamic target tracking performance, demonstrating the excellent performance of deep learning algorithms in dynamic target tracking. However, these methods cannot effectively analyze the semantic features of dynamic targets in multiple frames of the input video, making accurate tracking of dynamic targets difficult. Further research revealed that there is currently no deep learning-based dynamic target tracking method for legged robots that addresses image blurring caused by target motion or visual sensor vibration. Summary of the Invention

[0007] To address the issue of low accuracy in dynamic target tracking under motion blur in legged robots, this invention proposes a novel method for dynamic target tracking in legged robots. This method integrates semantic feature information of dynamic targets from multiple frames of the input video, enabling the identification and tracking of dynamic targets even when the input video is significantly affected by motion blur, thereby improving the accuracy of dynamic target identification and tracking in legged robots.

[0008] This invention provides a dynamic target tracking method for a legged robot, which includes two phases: training and testing.

[0009] The training phase is implemented by a target recognition module based on a recurrent neural network, a target semantic feature extraction module based on a residual network, and a target tracking module based on a Siamese network.

[0010] The target semantic feature extraction module based on residual networks uses the ResNet-50 model as the baseline network and removes the downsampling operations in the last two layers. The stride of the 4th and 5th convolutional blocks is set to 1. To reduce computational burden, a 1×1 convolutional block is added at the end, reducing the number of output feature channels to 256. Only the features of the 7×7 region at the center of the template branch are used, which still captures the entire target region. This module includes a template branch and a search branch, used to extract target semantic features in the current image frame and predict target semantic features in the next image frame, respectively.

[0011] The target tracking module based on Siamese networks includes an identification branch and a regression branch, which are used to obtain the target semantic feature map in the next image frame and calculate the correlation coefficient between the target semantic feature map in the current image frame and the next image frame, respectively. Each branch uses a deep cross-correlation layer to combine feature maps.

[0012] Testing Phase: After training, the legged robot's recurrent neural network-based target recognition module is used to determine whether pedestrians are wearing masks, and a Siamese network-based target tracking module is used to track pedestrians not wearing masks. This method includes the following steps:

[0013] Step 1: Prepare the training dataset.

[0014] Step 2: Input the videos from the training dataset into a recurrent neural network. The recurrent neural network obtains adjacent frame sequences (f) that are labeled with information on whether pedestrians are wearing masks and their facial bounding box information. t-1 ,f t ), t=2,…T-1, where, f t This represents the t-th frame image, which is marked with information on whether a pedestrian is wearing a mask and their facial bounding box information. The information on whether a pedestrian is wearing a mask is used to determine whether the pedestrian is wearing a mask correctly, not wearing a mask correctly, or not wearing a mask. The bounding box information is used to obtain the coordinates of the pedestrian's facial position.

[0015] Step 3: Combine the adjacent frame sequences (f) obtained by the recurrent neural network t-1 f t Input the target semantic feature extraction module based on the residual network to obtain the pedestrian facial semantic feature information in the t-th frame and t+1-th frame sequence. The pedestrian facial semantic feature information in the t-th frame image extracted from the template branch can be represented as: The pedestrian facial semantic feature information from the (t+1)th frame image predicted from the search branch can be represented as follows: The output feature size of the template branch is 127×127, and the output feature size of the search branch is 255×255.

[0016] Step 4: Calculate the loss function, where the overall loss function expression is updated as follows:

[0017]

[0018] Where λ1 and λ2 are hyperparameters, which define the weights of each branch in the overall loss function, respectively, λ1 = 1 and λ2 = 1.

[0019] L c Let represent the loss for recognizing whether a pedestrian is wearing a mask in frame t. A cross-entropy-based loss function is used, and its expression is as follows:

[0020]

[0021] Where y represents the confidence value of whether a pedestrian is wearing a mask in the t-th frame image obtained from the recurrent neural network. This represents the confidence value of whether a pedestrian is wearing a mask in the (t+1)th frame of the image predicted using a residual network.

[0022] L r Let represent the regression loss of the semantic features of pedestrian faces in frame t. A loss function based on intersection-union ratio (IU) is used, and its expression is as follows:

[0023] L r =1-IoU

[0024] Where IoU represents the intersection-union ratio of the predicted value of the pedestrian's face bounding box in the t-th frame image and the actual value of the pedestrian's face bounding box in the (t+1)-th frame image.

[0025] The semantic features of pedestrian faces extracted by the residual network and Inputting into the Siamese network-based target tracking module, semantic feature maps of pedestrian face locations in frame t and frame (t+1) are extracted using recognition and regression branches, respectively, and are represented as follows:

[0026] To match the same pedestrian face in frames t and t+1, it is necessary to calculate the semantic feature mapping of the pedestrian face between the two frames:

[0027]

[0028]

[0029] Where * denotes the convolution operation, P c Represents the identification mapping, P r This represents a regression mapping.

[0030] To match the same pedestrian face in frames t and t+1, we can use the recognition mapping P c and regression mapping P r Calculate the prediction coefficient P between the same pedestrian face positions in two frames. s Its expression is as follows:

[0031] P s =(1-ω)P c +ωP r

[0032] Where ω represents the hyperparameter, and here ω = 0.6.

[0033] To predict the bounding box coordinates of a pedestrian's face in the (t+1)th frame of the image, we can use the recognition mapping P c and regression mapping P r Perform calculations, assuming and Let $\mathbf$ and $\mathbf$ represent the coordinates of the top-left and bottom-right corners of the bounding box containing the pedestrian's face in the (t+1)th image. These coordinates can be obtained using the following expressions. and

[0034]

[0035]

[0036]

[0037]

[0038] in and They represent the regression mapping P respectively. r The predicted value, (P) i ,P j (P) represents the coordinates of the area within the bounded box of a pedestrian's face that can be used to determine if a pedestrian is not wearing a mask. i ,P j The following expression can be used for calculation:

[0039]

[0040] Where g w g h , Let represent the width, height, and center point coordinates of the bounding box containing the pedestrian's face in frame t, respectively. When the bounding box coordinates of the pedestrian's face in frame t+1 are obtained... and And the prediction coefficient P s At the highest value, it can be determined that the position has the highest correlation with the selected pedestrian face position in the t-th frame image, thus achieving the matching of the same pedestrian face position between two frames. This process is repeated for the entire training video.

[0041] After training, the proposed tracking method was deployed on a legged robot. A target recognition module based on a recurrent neural network was used to determine whether pedestrians were wearing masks. For pedestrians not wearing masks, a target semantic feature extraction module based on a residual network was used to extract the pedestrian's facial semantic features in the current image frame and predict the pedestrian's facial semantic features in the next image frame. A target tracking module based on a Siamese network was used to calculate and match the correlation coefficient between the semantic feature mappings of the pedestrian's facial position in the current frame and the next frame, thereby enabling the legged robot to track dynamic targets. Attached Figure Description

[0042] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0043] Figure 1 This is a flowchart of the dynamic target tracking method for legged robots in this invention.

[0044] Figure 2 This is a schematic diagram of the legged robot dynamic target tracking method in this invention.

[0045] Figure 3 This is a diagram illustrating the implementation effect of the present invention in a real-world campus inspection process using a legged robot. Detailed Implementation

[0046] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0047] A flowchart of a dynamic target tracking method for a legged robot is shown below. Figure 1 As shown, a schematic diagram of the method is as follows. Figure 2 As shown, the process includes two phases: training and testing. The training phase is implemented by a target recognition module based on a recurrent neural network, a target semantic feature extraction module based on a residual network, and a target tracking module based on a Siamese network. The testing phase involves using the recurrent neural network-based target recognition module mounted on the legged robot to determine whether pedestrians are wearing masks, and then using the Siamese network-based target tracking module to track pedestrians who are not wearing masks.

[0048] The target semantic feature extraction module based on residual networks uses the ResNet-50 model as the baseline network and removes the downsampling operations in the last two layers. The stride of the 4th and 5th convolutional blocks is set to 1. To reduce computational burden, a 1×1 convolutional block is added at the end, reducing the number of output feature channels to 256. Only the features of the 7×7 region at the center of the template branch are used, which still captures the entire target region. This module includes a template branch and a search branch, used to extract target semantic features in the current image frame and predict target semantic features in the next image frame, respectively.

[0049] The target tracking module based on Siamese networks includes an identification branch and a regression branch, which are used to obtain the target semantic feature map in the next image frame and calculate the correlation coefficient between the target semantic feature map in the current image frame and the next image frame, respectively. Each branch uses a deep cross-correlation layer to combine feature maps.

[0050] Specifically as follows:

[0051] Step 1: Use the RealSense depth camera mounted on a legged robot to collect continuous image sequences and video data of pedestrians walking in multiple different scenarios (such as schools, hospitals, parks, etc.) wearing masks, not wearing masks, and not wearing masks correctly. Complete the dataset production, which includes training set, validation set, and test set.

[0052] Step 2: Input the videos from the training dataset into a recurrent neural network. The recurrent neural network obtains adjacent frame sequences (f) that are labeled with information on whether pedestrians are wearing masks and their facial bounding box information. t-1 ,f t ), t=2,…T-1, where, f t This represents the t-th frame image, which is marked with information on whether a pedestrian is wearing a mask and their facial bounding box information. The information on whether a pedestrian is wearing a mask is used to determine whether the pedestrian is wearing a mask correctly, not wearing a mask correctly, or not wearing a mask. The bounding box information is used to obtain the coordinates of the pedestrian's facial position.

[0053] Step 3: Combine the adjacent frame sequences (f) obtained by the recurrent neural network t-1 f t Input the target semantic feature extraction module based on the residual network to obtain the pedestrian facial semantic feature information in the t-th frame and t+1-th frame sequence. The pedestrian facial semantic feature information in the t-th frame image extracted from the template branch can be represented as: The pedestrian facial semantic feature information from the (t+1)th frame image predicted from the search branch can be represented as follows: The output feature size of the template branch is 127×127, and the output feature size of the search branch is 255×255.

[0054] Step 4: Calculate the loss function, where the overall loss function expression is updated as follows:

[0055]

[0056] Where λ1 and λ2 are hyperparameters, which define the weights of each branch in the overall loss function, respectively, λ1 = 1 and λ2 = 1.

[0057] L c Let represent the loss for recognizing whether a pedestrian is wearing a mask in frame t. A cross-entropy-based loss function is used, and its expression is as follows:

[0058]

[0059] Where y represents the confidence value of whether a pedestrian is wearing a mask in the t-th frame image obtained from the recurrent neural network. This represents the confidence value of whether a pedestrian is wearing a mask in the (t+1)th frame of the image predicted using a residual network.

[0060] L r Let represent the regression loss of the semantic features of pedestrian faces in frame t. A loss function based on intersection-union ratio (IU) is used, and its expression is as follows:

[0061] L r =1-IoU

[0062] Where IoU represents the intersection-union ratio of the predicted value of the pedestrian's face bounding box in the t-th frame image and the actual value of the pedestrian's face bounding box in the (t+1)-th frame image.

[0063] The semantic features of pedestrian faces extracted by the residual network and Inputting into the Siamese network-based target tracking module, semantic feature maps of pedestrian face locations in frame t and frame (t+1) are extracted using recognition and regression branches, respectively, and are represented as follows:

[0064] To match the same pedestrian face in frames t and t+1, it is necessary to calculate the semantic feature mapping of the pedestrian face between the two frames:

[0065]

[0066]

[0067] Where * denotes the convolution operation, P c Represents the identification mapping, P r This represents a regression mapping.

[0068] To match the same pedestrian face in frames t and t+1, we can use the recognition mapping Pc and regression mapping P r Calculate the prediction coefficient P between the same pedestrian face positions in two frames. s Its expression is as follows:

[0069] P s =(1-ω)P c +ωP r

[0070] Where ω represents the hyperparameter, and here ω = 0.6.

[0071] To predict the bounding box coordinates of a pedestrian's face in the (t+1)th frame of the image, we can use the recognition mapping P c and regression mapping P r Perform calculations, assuming and Let $\mathbf$ and $\mathbf$ represent the coordinates of the top-left and bottom-right corners of the bounding box containing the pedestrian's face in the (t+1)th image. These coordinates can be obtained using the following expressions. and

[0072]

[0073]

[0074]

[0075]

[0076] in and They represent the regression mapping P respectively. r The predicted value, (P) i ,P j (P) represents the coordinates of the area within the bounded box of a pedestrian's face that can be used to determine if a pedestrian is not wearing a mask. i ,P j The following expression can be used for calculation:

[0077]

[0078] Where g w g h , Let represent the width, height, and center point coordinates of the bounding box containing the pedestrian's face in frame t, respectively. When the bounding box coordinates of the pedestrian's face in frame t+1 are obtained... and And the prediction coefficient P sAt the highest value, it can be determined that the position has the highest correlation with the selected pedestrian face position in the t-th frame image, thus achieving the matching of the same pedestrian face position between two frames. This process is repeated for the entire training video.

[0079] After training, the proposed tracking method was deployed on a legged robot. A target recognition module based on a recurrent neural network was used to determine whether pedestrians were wearing masks. For pedestrians not wearing masks, a target semantic feature extraction module based on a residual network was used to extract the pedestrian's facial semantic features in the current image frame and predict the pedestrian's facial semantic features in the next image frame. A target tracking module based on a Siamese network was used to calculate and match the correlation coefficient between the semantic feature mappings of the pedestrian's facial position in the current frame and the next frame, thereby enabling the legged robot to track dynamic targets.

[0080] The proposed legged robot dynamic target tracking algorithm was trained using the publicly available datasets MOT17, KITTI, and COCO. The server used was equipped with an Intel(R) Xeon(R) CPU E5-2620v4@2.10GHz×16, a quad-core NVIDIA GeForce GTX 2080 graphics card, 128GB of RAM, and Ubuntu 16.04 operating system. The algorithm was implemented using Python 3.6 and the PyTorch deep learning framework. The training hyperparameters were: Batchsize = 16, Epochs = 20, Learningrate = 5×10-1. -3 Weightdecay = 1 × 10 -4 , Momentum=0.9, Optimizer=StochasticGradientDescent(SGD).

[0081] The legged robot used in this invention is the Unitree A1 quadruped robot, equipped with an NVIDIA TX2 processor and a RealSense D455 depth camera. This camera achieves a maximum frame rate of 60 FPS for image acquisition and a video resolution of 640*480. The implementation effect of the legged robot during actual campus inspection is shown in the following diagram. Figure 3 As shown.

[0082] The advantage of this method is that the legged robot can accurately determine whether pedestrians affected by motion fuzziness are wearing masks and can track pedestrians who are not wearing masks. It has high robustness and can detect whether pedestrians affected by motion fuzziness are wearing masks in different scenarios (campus, hospital, park, etc.), thereby achieving accurate tracking of pedestrians who are not wearing masks with a tracking accuracy of ≥70%.

[0083] To simplify the explanation of this method, the above figures and text are described as a series of steps. However, it should be understood and understood that these methods are not restricted by the order of operations, because some actions may occur in different orders when implemented according to one or more steps, but those skilled in the art can understand the principle behind the actions.

[0084] Although illustrative specific embodiments of the invention have been described step by step to enable those skilled in the art to understand them, the invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various modifications or variations within the scope of the claims, as long as such variations are within the spirit and scope of the invention as defined and determined by the appended claims.

Claims

1. A method for dynamic target tracking of a legged robot, characterized in that: It includes two phases: training and testing. The training phase is implemented by a target recognition module based on a recurrent neural network, a target semantic feature extraction module based on a residual network, and a target tracking module based on a Siamese network. The target semantic feature extraction module based on residual networks uses the ResNet-50 model as the baseline network and removes the downsampling operations of the last two layers. The stride of the 4th and 5th convolutional blocks is set to 1. A 1×1 convolutional block is added at the end to reduce the number of output feature channels to 256, and only the features of the 7×7 region in the center of the template branch are used. This module includes a template branch and a search branch, which are used to extract the target semantic features in the current image frame and predict the target semantic features in the next image frame, respectively. The target tracking module based on Siamese network includes an identification branch and a regression branch, which are used to obtain the target semantic feature map in the next image frame and calculate the correlation coefficient between the target semantic feature map of the current image frame and the next image frame, respectively. Each branch uses a deep cross-correlation layer to combine feature maps. Testing Phase: After training, the target recognition module based on recurrent neural networks on the legged robot is used to determine whether pedestrians are wearing masks. For pedestrians who are not wearing masks, the target semantic feature extraction module based on residual networks is used to extract the facial semantic features of the pedestrians in the current image frame and predict the facial semantic features of the pedestrians in the next image frame. The target tracking module based on Siamese networks is used to calculate the correlation coefficient between the semantic feature mapping of the pedestrian's facial position in the current frame and the next frame and match them, thereby realizing the tracking of dynamic targets by the legged robot.

2. The core of the legged robot dynamic target tracking method according to claim 1 lies in the following training process: Step 1: Prepare the training dataset; Step 2: Input the videos from the training dataset into a recurrent neural network. The recurrent neural network obtains adjacent frame sequences (f) that are labeled with information on whether pedestrians are wearing masks and their facial bounding box information. t-1 ,f t ), t=2,…T-1, where, f t This represents the t-th frame image, which is marked with information on whether a pedestrian is wearing a mask and their facial bounding box information. The information on whether a pedestrian is wearing a mask is used to determine whether the pedestrian is wearing a mask correctly, not wearing a mask correctly, or not wearing a mask. The bounding box information is used to obtain the coordinates of the pedestrian's facial position. Step 3: Combine the adjacent frame sequences (f) obtained by the recurrent neural network t-1 f t Input the target semantic feature extraction module based on the residual network to obtain the pedestrian facial semantic feature information in the sequence of frame t and frame t+1. The pedestrian facial semantic feature information in the image of frame t extracted from the template branch is represented as follows: The pedestrian facial semantic feature information from the (t+1)th frame image predicted from the search branch is represented as follows: The output feature size of the template branch is 127×127, and the output feature size of the search branch is 255×255; Step 4: Calculate the loss function, where the overall loss function expression is updated as follows: Where λ1 and λ2 are hyperparameters, λ1 = 1, λ2 = 1; L c The loss for recognizing whether a pedestrian is wearing a mask in frame t is represented by a cross-entropy-based loss function, expressed as follows: Where y represents the confidence value of whether a pedestrian is wearing a mask in the t-th frame image obtained from the recurrent neural network. This represents the confidence value of whether a pedestrian is wearing a mask in the (t+1)th frame of the image predicted using a residual network. L r Let represent the regression loss of the semantic features of pedestrian faces in frame t. A loss function based on intersection-union ratio (IU) is used, and its expression is as follows: L r =1-IoU Wherein, IoU represents the intersection-union ratio of the predicted value of the pedestrian's face bounding box in the t-th frame image and the actual value of the pedestrian's face bounding box in the (t+1)-th frame image; The semantic features of pedestrian faces extracted by the residual network and Inputting into the Siamese network-based target tracking module, semantic feature maps of pedestrian face locations in frame t and frame t+1 are extracted using recognition and regression branches, respectively, and are represented as follows: To match the same pedestrian face in frames t and (+1) of the image, it is necessary to calculate the semantic feature mapping of the pedestrian face between the two frames: Where * denotes the convolution operation, P c Represents the identification mapping, P r Represents a regression mapping; To match the same pedestrian face in frames t and t+1, we can use the recognition mapping P c and regression mapping P r Calculate the prediction coefficient P between the same pedestrian face positions in two frames. s Its expression is as follows: P s =(1-ω)P c +ωP r Where ω represents the hyperparameter, and here ω = 0.6; To predict the bounding box coordinates of the pedestrian face location in the (t+1)th frame image, based on the recognition mapping P... c and regression mapping P r Perform calculations, assuming and The coordinates of the top-left and bottom-right corners of the bounding box representing the pedestrian's face position in the (t+1)th image are respectively obtained using the following expressions. and in and Representing regression mapping P respectively r The predicted value, (P) i ,P j (P) represents the coordinates of the area within the bounded box of a pedestrian's face that can be used to determine if a pedestrian is not wearing a mask. i ,P j The calculation is performed using the following expression: Where g w g h , These represent the true values ​​of the width, height, and center point coordinates of the bounding box of the pedestrian's face in the t-th frame image; when the bounding box coordinates of the pedestrian's face in the (t+1)-th frame image are obtained... and And the prediction coefficient P s At the highest point, it can be determined that the position has the highest correlation with the selected pedestrian face position in the t-th frame image, and at this time, the matching of the same pedestrian face position between the two frames is achieved; and so on, traversing the training video to complete the process.

Images