Mobile robot target following method based on improved YOLOX

Abstract

This invention discloses a target following method for mobile robots based on an improved YOLOX network, mainly comprising three parts: target detection, target tracking, and target following. First, based on the YOLOX network, an improved YOLOX network is adopted as the backbone network within its framework to improve the real-time performance of target detection in complex scenes. Then, an improved Kalman filter is used to obtain the target tracking status, and data association is used for target matching. When target occlusion is determined by depth histogram, depth probability information constraints and maximum a posteriori probability are used for matching and tracking to ensure stable target tracking by the robot under occlusion conditions. Finally, a target following algorithm based on visual servo control is employed. When the tracked target is lost, re-identification features are introduced to actively search for the target and achieve target following. This effectively improves the accuracy and speed of target detection, increases the target prediction and update rate, and ensures stable target following by the mobile robot.

Description

Technical Field

[0001] This invention belongs to the field of mobile robots, and in particular relates to a target following method for mobile robots based on an improved YOLOX. Background Technology

[0002] Robot target following algorithms can be broadly categorized into generative model methods and detection-tracking methods. Generative models primarily achieve following by constructing a target model. For example, some researchers use visual sensors to acquire the color and texture features of pedestrians. The robot then searches for matching regions within its field of view, integrates pedestrian and position / velocity information to build a model, and employs a generative target tracking algorithm to follow the pedestrian. However, these algorithms focus on the target itself, ignoring background information, and frequently result in tracking loss.

[0003] To simultaneously consider target and background information, detection-tracking methods have received increasing attention. These methods distinguish between the target and the background by constructing a classifier, and their tracking performance is generally superior to generative model methods. Some researchers have achieved stable tracking by combining fast discriminative scale-space switching correlation filtering and Kalman filtering algorithms. However, mobile robots are often affected by background clutter, lighting changes, target occlusion, and scale changes during following control, leading to target loss. Therefore, traditional detection-tracking methods are not suitable for target following in complex and variable scenes.

[0004] Deep learning-based target following algorithms for mobile robots possess advantages such as strong robustness. Some researchers have improved mobile robot tracking performance using TCB sampling strategies, but these haven't addressed issues like occlusion and pedestrian disappearance. Other researchers have proposed a deep learning-based target detector, incorporating Kalman filtering to predict target position and adding a re-identification module to handle occlusion; however, such algorithms require high-precision target detection results first. Given these issues, the JDE (Jointly Learns the Detector and Embedding model) detection model can be used to fuse the re-identification and detection branches, improving target detection accuracy. The YOLO series of algorithms is a class of one-stage bounding box target detection algorithms based on the JDE detection model, offering advantages such as high efficiency, flexibility, and good generalization performance.

[0005] The YOLO algorithm includes the YOLOv1-7 series and a series of object detection algorithms based on improved YOLO. Researchers proposed using YOLO for object detection, directly employing regression methods for bounding box detection and classification. A simple end-to-end network is used for coordinate regression and classification, significantly improving object detection speed. Subsequently, the YOLO network structure has been continuously optimized, becoming a mainstream algorithm in the field of object detection. Researchers introduced a ratio-aware mechanism, dynamically adjusting the input layer length and width hyperparameters of YOLOv3 to address the problem of large aspect ratio differences, effectively improving average tracking accuracy. Researchers introduced an improved YOLOv3 model, increasing the prediction scale from three to four and using additional feature maps to extract more details. However, YOLOv3 has poor object location recognition accuracy, performing poorly in complex scenes with densely distributed objects of varying sizes. YOLOv4 developed the Darknet53 object detection model, which has higher network input resolution, more network layer parameters, and higher computational complexity, but performs poorly in detecting small objects. To address this, YOLO-Z proposed a series of models at different scales to improve the performance of YOLOV5 in detecting small targets. Researchers proposed a single-stage (SSD) small target detection method, which improves the real-time performance of small target detection, but its two-stage target detector reduces target localization accuracy. YOLOV6 designed a more efficient backbone network and network layers, and YOLOV7 extended the efficient long-range attention network and incorporated a cascade-based model scaling method, all of which improved detection accuracy and inference efficiency to some extent. However, due to the lack of a re-identification branch, it cannot extract shallow features for subsequent tracking. YOLOX made a series of improvements based on the YOLO series. Its biggest difference from the YOLO series target detection algorithms is the use of an anchor-free detector. YOLOV1-YOLOV5, which used anchor-bound detectors, were not suitable for the JDE detection model because they could be detected by multiple anchor boxes simultaneously and had errors in the center of the detection box. Therefore, the anchor-free YOLOX target detection algorithm is more suitable for the JDE detection model.

[0006] The core issue in mobile robot target detection and tracking is the impact of complex scene interference on detection accuracy and tracking performance during movement. YOLOX, with its Darknet53 network backbone, offers high detection accuracy, but its large model size and slow inference speed make it unsuitable for real-time mobile robot tracking. While the feature extraction capability of the YOLOv5 network model increases with depth, the increasing number of downsampling iterations leads to gradient vanishing, significantly affecting detection accuracy. To improve detection accuracy, researchers have proposed the DeepSORT target tracking algorithm, which uses Kalman filtering to update target positions and associates them with the currently detected target. However, this algorithm does not address the target loss problem caused by occlusion during tracking. Researchers have proposed the PSR target tracking algorithm, which introduces depth information to evaluate tracking reliability and can actively detect lost targets. However, its use of correlation filtering for target tracking results in low robustness in complex scenes. Therefore, improving the network structure while incorporating depth information is a crucial method to explore for enhancing the detection and tracking performance of mobile robots. Summary of the Invention

[0007] To address the above technical problems, this invention provides a mobile robot target following method based on an improved YOLOX that can effectively improve robot detection and following performance.

[0008] The technical solution adopted by this invention to solve its technical problem is:

[0009] The mobile robot target following method based on the improved YOLOX includes the following steps:

[0010] Step S100: Obtain the video sequence and extract the RGB image and depth image from the video sequence;

[0011] Step S200: Input the RGB image into the target detection neural network to obtain the target detection result; the target detection neural network includes an improved YOLOX backbone network, network layers, target detection branch layers, and target re-identification branch layers. The improved YOLOX backbone network is used to extract features from the RGB image to obtain shallow features and deep features; the network layers perform convolution processing on both shallow and deep features to obtain re-identification branch feature maps and detection branch feature maps; the target detection branch layers are used to perform target, classification, and regression prediction based on the detection branch feature maps to obtain target detection branch results; the target re-identification branch layers are used to perform target re-identification based on the re-identification branch feature maps to obtain target re-identification results; and the target detection result is obtained based on the target detection branch results and the target re-identification results.

[0012] Step S300: Based on the target detection results and the improved Kalman filter, predict the state of the tracked target. Based on the state of the tracked target and the re-identification branch feature map, obtain the re-identification feature map of the predicted updated tracking box of the current detection frame. Calculate the cosine distance between the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box. Use the cosine distance and a preset distance threshold to determine whether the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box are related.

[0013] Step S400: If the association is unsuccessful, obtain the depth values ​​of the tracking target and all other targets between the previous frame and the current frame based on the depth image. Determine whether the tracking target is occluded based on the depth values ​​of the tracking target and all other targets between the previous frame and the current frame. If occlusion occurs, obtain the bounding box corresponding to the depth map of the detected target in the current frame and the bounding box corresponding to the depth map of the tracking target to obtain the overlap rate of the target boxes. Perform occlusion matching tracking based on the overlap rate of the target boxes and the preset objective function of the associated occluded target, and predict the occluded target in the current frame as the target tracking result.

[0014] Step S500: Select a target from the target tracking results as the following target, and use a target following algorithm based on visual servo control to keep the center point of the target box as the center point of the field of view. When the target disappears, the mobile robot actively searches according to the target's motion trajectory, re-identifies the target, and makes the mobile robot continue to follow the target.

[0015] Preferably, the improved YOLOX backbone network comprises seven stages, with the core of each stage being a MobileNetV2S network. The MobileNetV2S network includes a first 1×1 convolutional layer, a 3×3 convolutional layer, a 3×3 depthwise convolutional layer, a 1×1 projective convolutional layer, and a ReLU layer.

[0016] The first 1×1 convolutional layer uses a 1×1 convolutional kernel to expand the RGB feature map into feature map F. 1×1 ∈R H×W×C′ ;

[0017] 3×3 convolutional layers on feature map F 1×1 ∈R H×W×C′ The feature map F is obtained by performing depthwise separable convolution. 3×3 ∈R H ′×W′×C′ ;

[0018] Feature map F 3×3 ∈R H′×W′×C By splitting the feature map in half, the number of channels is reduced to half of the original number, resulting in feature map F. (3×3) / 2 ∈R H′×W′×(C′ / 2) and F′ (3×3) / 2 ∈RH′×W′×(C′ / 2) A 3×3 depth separable convolutional layer will convert the feature map F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) Depth-separable convolution is performed to obtain

[0019] Will With F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) The two are combined to obtain a new feature map F′ 3×3 ∈R H″′×W″′×C′ A 1×1 projective convolutional layer will generate a new feature map F′ 3×3 ∈R H″′×W″′×C′ Convolution yields the feature map F′∈R H″′×W″′×C′ The output feature map of each bottleneck layer is obtained through the ReLU layer. The output feature map F1 corresponding to the first stage is a shallow feature map, and the output feature map F7 corresponding to the seventh stage is a deep feature map.

[0020] Preferably, the target detection branch layer includes a decoupling head and target branches, classification branches, and regression branches that are all connected to the decoupling head. The network layers perform convolutional processing on deep features to obtain detection branch feature maps. The target detection branch layer is used to perform target, classification, and regression predictions based on the detection branch feature maps to obtain the target detection branch results, including:

[0021] Network layers for deep feature maps F7∈R 15×15×320 Convolution processing is performed to obtain the detection branch feature map. The first feature map is obtained after the detection branch feature map is decoupled by the decoupling head. Second feature map and the third feature map

[0022] The first feature map The data is sent to the target branch. Each feature point in the target branch represents the probability that the detected target within the corresponding prediction box belongs to the foreground. This determines whether the target is foreground or background, and the result of the target branch is obtained.

[0023] The second feature map The data is sent to the regression branch to predict the center coordinates and height and width (x, y, w, h) of the target bounding box, thus obtaining the regression branch result.

[0024] The third feature map The data is sent to the classification branch to predict and score the category to which the target belongs, thus obtaining the classification branch result.

[0025] The feature map is obtained by merging and summing the results of the target branch, regression branch, and classification branch. As a result of the object detection branch.

[0026] Preferably, the network layer performs convolution processing on the shallow features to obtain a re-identification branch feature map. The target re-identification branch layer is used to perform target re-identification based on the re-identification branch feature map to obtain the target re-identification result, including:

[0027] Network layer repetition identification branch feature map F1∈R 240×240×16 Convolution processing is performed to obtain the re-identification branch feature map.

[0028] Use a 3×3 convolution kernel to sequentially re-identify the branch feature map. Convolution yields feature maps.

[0029] A feature map with 128 channels is obtained through 128 sets of 1×1 convolutions. The shallow appearance features at the center point (x, y) of the corresponding target box are extracted from the feature map and used as the target re-identification result.

[0030] Preferably, step S200 further includes:

[0031] The loss value of the target detection branch layer is calculated based on the preset target detection loss function. The target detection branch layer is then updated in reverse based on the loss value to obtain the updated target detection branch layer.

[0032] The loss value of the target re-identification branch layer is calculated based on the preset re-identification loss function. The target re-identification branch layer is then updated in reverse based on the loss value of the target re-identification branch layer to obtain the updated target re-identification branch layer.

[0033] Preferably, the preset target detection loss function is as follows:

[0034] L detection =λ1L cls +λ2L obj +λ3L reg

[0035]

[0036]

[0037] L reg =1-IoU

[0038] Where λ1, λ2, and λ3 are balance coefficients, L cls L obj and L reg These are classification loss, target loss, and regression loss, respectively, N. obj Representative used to calculate L obj The loss function represents the total number of video frame targets, y. sThis represents the label of the test sample s, with foreground labeled 1 and background labeled 0. s N represents the probability that the test sample s is predicted as foreground, IoU represents the area overlap ratio between the predicted target bounding box and the ground truth target bounding box in the current frame, and N represents the probability that the test sample s is predicted as foreground. cls Representative used to calculate L cls The loss function is the total number of targets in the video frames, where M represents the number of categories, and y dc The sign function, if the true class of target d in the current video frame is equal to c, y dc If p is 1, then p is 0; otherwise, p is 0. dc This represents the predicted probability that target d in the current frame belongs to category c.

[0039] The preset re-identification loss function is as follows:

[0040]

[0041] in, The label encoding for the target ground truth bounding box is given by the feature distribution vector C = {c(b), b ∈ [1, B]}, N re-id B represents the total number of categories to which the current target belongs, and B is the ID number of all identities (IDs) in the training data.

[0042] Preferably, in step S300, the state of the target being tracked is predicted based on the target detection results and the improved Kalman filter, specifically as follows:

[0043] x t，i =f(x) t-1，i )+w t-1

[0044] z t，j =h(x t，i )+v t

[0045] Where, x t，i To detect the state of the target. β = (u, v, γ, h), where β represents the observed value of the target, (u, v) represents the center position of the bounding box, the aspect ratio is γ, the height is h, and the z-axis is h. t，j To track the state of the target, w t-1 and v t These are the process noise sequence and the measurement noise sequence, respectively.

[0046] In step S300, cosine distance and a preset distance threshold are used to determine whether the re-identification features of the current detection frame and the re-identification features of the predicted updated tracking box are related. Specifically:

[0047] When the cosine feature is less than or equal to the preset distance threshold, the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box are successfully associated; when the cosine feature is greater than the preset distance threshold, the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box are not successfully associated.

[0048] Preferably, in step S400, the depth values ​​of the tracking target and all other targets between the previous frame and the current frame are obtained based on the depth image, and the occlusion of the tracking target is determined based on the depth values ​​of the tracking target and all other targets between the previous frame and the current frame. Specifically:

[0049]

[0050] in, To track the occlusion changes of target j in frame t, It is the depth value of the tracked target j in frame t. This represents the sum of depth differences of all targets between frame t-1 and frame t. This represents the rate of change of depth value between frame t and frame (t-1). Represents less than w t -ξ t The sum of depth differences of all tracked targets;

[0051] The occlusion judgment criteria are as follows:

[0052]

[0053] Among them, T j This is the criterion for determining occlusion.

[0054] Preferably, in step S400, obtaining the bounding boxes corresponding to the depth maps of the detected target and the tracked target in the current frame to obtain the overlap rate of the target boxes specifically involves:

[0055] C = b(x) t，i )∩b(z t-1，j )>σ

[0056] Where C is the overlap rate of the target bounding box, σ is the preset overlap rate threshold, and b(x) t，i ) represents the target x detected in the current frame. t，i The bounding box corresponding to the depth map, b(z t-1，j To track target z t-1，j The bounding box corresponding to the depth map, if C is greater than σ, indicates that x t，i With z t-1，j Related matching;

[0057] In step S400, occlusion matching and tracking are performed based on the overlap rate of the target bounding box and the target function of the preset associated occluded target. The occluded target of the current frame is predicted as the target tracking result, specifically:

[0058]

[0059] Among them, S * For the target tracking result, P(z) t-1，j Let P(x) be the prior probability of all tracked targets. t，i |z t-1，j The conditional probability that the currently detected target belongs to the tracked target is calculated by the overlap rate between the detected target and the tracked target box in the previous frame.

[0060] Preferably, in step S500, if the target disappears, the mobile robot actively searches according to the target's movement trajectory, re-identifies the target, and enables the mobile robot to continue following the target, including:

[0061] If the target goes out of the mobile robot's field of vision during the tracking process, the mobile robot will retain the re-identification features and position information of the target in the last frame before it disappeared, and continue to move in the direction in which the target disappeared. If the target reappears in the field of vision, the robot will perform target detection and extract the re-identification features of the target and associate and match them with the re-identification features of the target in the last frame before it disappeared, thereby realizing the re-identification of the target and enabling the mobile robot to continue following the target.

[0062] The aforementioned mobile robot target following method based on the improved YOLOX network mainly comprises three parts: target detection, target tracking, and target following. First, based on the YOLOX network, the backbone network is an improved YOLOX network to enhance the real-time performance of target detection in complex scenes. Then, an improved Kalman filter is used to obtain the target tracking status, and data association is used for target matching. When target occlusion is detected using a depth histogram, depth probability information constraints and maximum a posteriori probability are used for matching and tracking to ensure stable target tracking even under occlusion conditions. Finally, a target following algorithm based on visual servo control is employed. When the tracked target is lost, re-identification features are introduced to actively search for the target and achieve target following. This effectively improves target detection accuracy and speed, increases target prediction and update rate, and ensures stable target following by the mobile robot. Attached Figure Description

[0063] Figure 1 A flowchart illustrating the mobile robot target following method based on the improved YOLOX provided by this invention;

[0064] Figure 2The overall flowchart of the mobile robot target following method based on the improved YOLOX provided by the present invention;

[0065] Figure 3 This is a depth histogram before and after occlusion in one embodiment of the present invention;

[0066] Figure 4 This is an image captured by a ZED camera in one embodiment of the present invention;

[0067] Figure 5 This is a diagram of a two-wheel differential drive model based on a ZED camera in one embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the target follower movement control section in one embodiment of the present invention. Detailed Implementation

[0069] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0070] In one embodiment, such as Figure 1 As shown, the mobile robot target following method based on the improved YOLOX includes the following steps:

[0071] Step S100: Obtain the video sequence and extract the RGB image and depth image from the video sequence.

[0072] Step S200: Input the RGB image into the target detection neural network to obtain the target detection result; the target detection neural network includes an improved YOLOX backbone network, network layers, target detection branch layers, and target re-identification branch layers. The improved YOLOX backbone network is used to extract features from the RGB image to obtain shallow features and deep features; the network layers perform convolution processing on both shallow and deep features to obtain re-identification branch feature maps and detection branch feature maps; the target detection branch layers are used to perform target, classification, and regression prediction based on the detection branch feature maps to obtain target detection branch results; the target re-identification branch layers are used to perform target re-identification based on the re-identification branch feature maps to obtain target re-identification results; and the target detection result is obtained based on the target detection branch results and the target re-identification results.

[0073] Specifically, target detection is a key issue in target following for mobile robots, and the accuracy of target detection largely determines the stability of the robot's following. This invention improves upon the YOLOX architecture, optimizing the network structure and loss function to enhance real-time detection performance. The backbone network uses the MobileNetV2S network, and the detection results are obtained through network layers, target detection branch layers, and target re-identification branch layers.

[0074] In one embodiment, the improved YOLOX backbone network comprises seven stages, with the core of each stage being a MobileNetV2S network. The MobileNetV2S network includes a first 1×1 convolutional layer, a 3×3 convolutional layer, a 3×3 depthwise convolutional layer, a 1×1 projective convolutional layer, and a ReLU layer.

[0075] The first 1×1 convolutional layer uses a 1×1 convolutional kernel to expand the RGB feature map into feature map F. 1×1 ∈R H×W×C′ ;

[0076] 3×3 convolutional layers on feature map F 1×1 ∈R H×W×C′ The feature map F is obtained by performing depthwise separable convolution. 3×3 ∈R H ′×W′×C′ ;

[0077] Feature map F 3×3 ∈R H′×W′×C′ By splitting the feature map in half, the number of channels is reduced to half of the original number, resulting in feature map F. (3x3) / 2 ∈R H′×W′×(C′ / 2) and F′ (3x3) / 2 ∈R H′×W′(C′ / 2) A 3×3 depth separable convolutional layer will convert the feature map F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) Depth-separable convolution is performed to obtain

[0078] Will With F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) The two are combined to obtain a new feature map F′ 3×3 ∈R H″′×W″′×C′ A 1×1 projective convolutional layer will generate a new feature map F′ 3×3 ∈R H″′×W″′×C′ Convolution yields the feature map F′∈R H″′×W″′×C″′ The output feature map of each bottleneck layer is obtained through the ReLU layer. The output feature map F1 corresponding to the first stage is a shallow feature map, and the output feature map F7 corresponding to the seventh stage is a deep feature map.

[0079] Specifically, this invention proposes the YOLOX-MobileNetV2S network (YOLOX-M2S), which uses the lightweight feature extraction network MobileNetV2S instead of the YOLOX backbone network. The core convolutional layer of this network is a depthwise separable convolutional layer, which can reduce the number of channels of the output feature map to half and then merge it with the feature map extracted by the original convolutional layer. Compared with MobileNetV2, which only uses a set of depthwise separable convolutions, this network can obtain more semantic information from the feature map.

[0080] On the YOLOX-M2S network, the network parameters were first trained using the COCO2017 training set, and then ported to a mobile robot platform for real-time detection. The COCO dataset is a large-scale dataset that can be used for image detection, containing more than 330K images (of which 220K are labeled images), covering 1.5 million objects and 80 object categories (object categories: pedestrians, cars, elephants, etc.) and 91 material categories (stuff categories: grass, walls, sky, etc.). Each image contains five sentences of description, and there are 250,000 pedestrians with keypoint annotations.

[0081] like Figure 2 As shown, the MobileNetV2S network divides object detection classification into 7 stages, with an input image resolution of H×W (H is the image height, W is the image width). Assume the input feature map is represented as F∈R. H×W×C Where H is the height, W is the width, and C is the number of channels, the core layer of each stage is the bottleneck layer.

[0082] The bottleneck layer in each stage includes the following steps: expanding the feature map to F using a 1×1 convolutional kernel. 1×1 ∈R H×W×C′ This significantly reduces the computational load; feature map F 1×1 ∈R H×W×C′ The feature map F is obtained by performing a 3×3 depthwise separable convolution. 3×3 ∈R H′×W′×C′ To obtain more semantic information, the feature map F 3×3 ∈R H′×W′×C′ By splitting the feature map in half, the number of channels is reduced to half of the original number, resulting in feature map F. (3×3) / 2 ∈R H′×W′×(C′ / 2) and F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) Then the feature map F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) Depth-separable convolution is performed to obtain Then With F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) The two are combined to obtain a new feature map F′ 3×3 ∈R H ″′×W″′×C′ Finally, the new feature map F′ 3×3 ∈R H″′×W″′×C′ The feature map F′∈R is obtained by further convolution using a projective convolutional layer with a 1×1 kernel. H″′×W″′×C′ This yields the output feature map for each bottleneck layer.

[0083] In the 7th stage of the MobileNetV2S network, the deep feature map F7∈R is obtained. 15×15×320 In the first stage, a shallow feature map F1∈R is obtained. 240×240×15 After passing through the network layers, the input feature maps of the target detection branch layer and the target re-identification branch layer are obtained.

[0084] In one embodiment, the target detection branch layer includes a decoupling head and target branches, classification branches, and regression branches that are all connected to the decoupling head. The network layers perform convolutional processing on deep features to obtain detection branch feature maps. The target detection branch layer is used to perform target, classification, and regression predictions based on the detection branch feature maps to obtain target detection branch results, including:

[0085] Network layers for deep feature maps F7∈R 15×15×320 Convolution processing is performed to obtain the detection branch feature map. The first feature map is obtained after the detection branch feature map is decoupled by the decoupling head. Second feature map and the third feature map

[0086] The first feature map The data is sent to the target branch. Each feature point in the target branch represents the probability that the detected target within the corresponding prediction box belongs to the foreground. This determines whether the target is foreground or background, and the result of the target branch is obtained.

[0087] The second feature map The data is sent to the regression branch to predict the center coordinates and height and width (x, y, w, h) of the target bounding box, thus obtaining the regression branch result.

[0088] The third feature map The data is sent to the classification branch to predict and score the category to which the target belongs, thus obtaining the classification branch result.

[0089] The feature map is obtained by merging and summing the results of the target branch, regression branch, and classification branch. As a result of the object detection branch.

[0090] Specifically, the detection branch feature map F7∈R output by the MobileNetV2S network 15×15×320 Feature maps are obtained after passing through network layers. After passing through the decoupling head, the detection branch is obtained, which includes three branches: target, classification, and regression. The feature map of the input target branch is... Each feature point in the branch represents the probability that the detected target within the corresponding predicted bounding box belongs to the foreground, thus determining whether the target is foreground or background. To stabilize the training process, accelerate convergence, and accurately locate the target, this branch estimates the continuous drift of each pixel relative to the target center, reducing the impact of downsampling. The anchor box size is used to estimate the height and width of the target anchor box at each location. The feature map input to the regression branch is... This branch predicts the center coordinates and height / width (x, y, w, h) of the bounding box; the feature map input to the classification branch is... This branch yields a predicted score for the target's category, such as whether the target belongs to pedestrian, vehicle, or animal, representing the probability of the target belonging to each category. Finally, the outputs of the three branches are combined and summed to obtain the feature map. This refers to the information from the target detection branch.

[0091] In one embodiment, the network layer performs convolution processing on shallow features to obtain a re-identification branch feature map. The target re-identification branch layer is used to perform target re-identification based on the re-identification branch feature map to obtain the target re-identification result, including:

[0092] Network layer repetition identification branch feature map F1∈R 240×240×16 Convolution processing is performed to obtain the re-identification branch feature map.

[0093] Use a 3×3 convolution kernel to sequentially re-identify the branch feature map. Convolution yields feature maps.

[0094] A feature map with 128 channels is obtained through 128 sets of 1×1 convolutions. The shallow appearance features at the center point (x, y) of the corresponding target box are extracted from the feature map and used as the target re-identification result.

[0095] Specifically, in order to complete target matching and recognition between consecutive video frames when the target disappears and reappears (i.e., target re-identification), a re-identification branch is added to the YOLOX-M2S network to extract shallow appearance features such as the target's color and texture as re-identification features.

[0096] In one embodiment, step S200 further includes:

[0097] The loss value of the target detection branch layer is calculated based on the preset target detection loss function. The target detection branch layer is then updated in reverse based on the loss value to obtain the updated target detection branch layer.

[0098] The loss value of the target re-identification branch layer is calculated based on the preset re-identification loss function. The target re-identification branch layer is then updated in reverse based on the loss value of the target re-identification branch layer to obtain the updated target re-identification branch layer.

[0099] In one embodiment, the preset target detection loss function is specifically:

[0100] L detection =λ1L cls +λ2L obj +λ3L reg

[0101]

[0102]

[0103] L reg =1-IoU

[0104] Where λ1, λ2, and λ3 are balance coefficients, L cls L obj and L reg These are classification loss, target loss, and regression loss, respectively, N. obj Representative used to calculate L obj The loss function represents the total number of video frame targets, y. s This represents the label of the test sample s, with foreground labeled 1 and background labeled 0. s N represents the probability that the test sample s is predicted as foreground, IoU represents the area overlap ratio between the predicted target bounding box and the ground truth target bounding box in the current frame, and N represents the probability that the test sample s is predicted as foreground. cls Representative used to calculate L cls The loss function is the total number of targets in the video frames, where M represents the number of categories, and y dc The sign function, if the true class of target d in the current video frame is equal to c, y dc If p is 1, then p is 0; otherwise, p is 0. dc This represents the predicted probability that target d in the current frame belongs to category c.

[0105] The preset re-identification loss function is as follows:

[0106]

[0107] in, The label encoding for the target ground truth bounding box is given by the feature distribution vector C = {c(b), b ∈ [1, B]}, N re-id B represents the total number of categories to which the current target belongs, and B is the ID number of all identities (IDs) in the training data.

[0108] Specifically, to measure the difference between the detected object information and the true object information, a loss function is further defined. The smaller the loss function value, the smaller the difference and the higher the accuracy of the trained model. Since the object detection branch in the MobilNetV2S network includes an object branch, a regression branch, and a classification branch, its corresponding loss function is derived from the object loss function L. obj Regression loss function L reg and classification loss function L cls It consists of three parts.

[0109] In object detection, it is necessary to first determine whether the predicted target belongs to the foreground or the background, and the target loss function L... obj The difference between the binary classification cross-entropy loss function and the true value is measured; L reg The regression loss function uses the Intersection over Union (IoU) loss function to measure the intersection-union ratio (area overlap) between the predicted detection box and the ground truth target box. The IoU index ranges from [0, 1], and the higher the area overlap, the higher the IoU index value. To evaluate the difference between the category to which the target belongs in the current video frame and the true value, the classification loss function uses the multi-class cross-entropy loss function to score the predicted category to which the target belongs.

[0110] A re-identification loss function is defined to evaluate the accuracy of the re-identified feature map; the smaller the value, the more accurate the re-identified feature map. Finally, the detection and re-identification loss functions are added together, and an uncertainty loss function is used to automatically balance the detection and re-identification loss functions, compared to using L alone. id Compared to the trained model, the training effect is improved while reducing computational complexity, and real-time requirements can be met.

[0111] Step S300: Based on the target detection results and the improved Kalman filter, predict the state of the tracked target. Based on the state of the tracked target and the re-identification branch feature map, obtain the re-identification feature map of the predicted updated tracking box of the current detection frame. Use cosine distance and a preset distance threshold to determine whether the re-identification features of the current detection frame and the re-identification features of the predicted updated tracking box are related.

[0112] Specifically, the target trajectory and tracking state are first initialized using the target bounding box detected in the first frame. Then, the target position in the next frame is predicted using an improved Kalman filter. Finally, the target tracking state is determined by data correlation between consecutive frames.

[0113] In the current video frame, suppose M targets i = 1, ..., M are detected at time t, and N targets j = 1, ..., N are tracked at time t. The detection and tracking results are updated in real time for each frame. Then the state of the i-th detected target at time t is x. t，i The state of the j-th tracked target is z t，j .

[0114] In one embodiment, step S300, which predicts the state of the tracked target based on the target detection result and the improved Kalman filter, specifically involves:

[0115] x t，i =f(x) t-1，i )+w t-1

[0116] z t，j =h(x t，i )+v t

[0117] Where, x t，i To detect the state of the target. β = {u, v, γ, h}, where β represents the observed value of the target, (u, v) represents the center position of the bounding box, the aspect ratio is γ, the height is h, and the z-axis is h. t，j To track the state of the target, w t-1 and v t These are the process noise sequence and the measurement noise sequence, respectively.

[0118] Specifically, assuming w t-1 and v t It is Gaussian white noise with a mean of 0 and variances Q. t and R t , i.e. w t-1 ~(0, Q) t V t ~(0, R) t ).

[0119] To explain in detail the process of improving Kalman filter prediction and update, the pseudocode for this part, Algorithm 1, is given below.

[0120]

[0121] In one embodiment, step S300 uses cosine distance and a preset distance threshold to determine whether the re-identification features of the current detection frame and the re-identification features of the predicted updated tracking box are related. Specifically:

[0122] When the cosine feature is less than or equal to the preset distance threshold, the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box are successfully associated; when the cosine feature is greater than the preset distance threshold, the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box are not successfully associated.

[0123] Specifically, an improved Kalman filter is used to obtain the predicted position z of the center point of target i in the previous frame in the current frame. t，j At the same time, through re-identification of feature maps The predicted center position is then used to obtain the predicted appearance features of the target from the previous frame in the current frame. During tracking, mobile robots may encounter situations such as occlusion and rapid movement. Cosine distance offers the advantage of rapid measurement. The cosine distance q(i,j) is used to determine the appearance feature vector corresponding to the center point of the current frame. The predicted appearance feature vector of the previous frame in the current frame Is it related?

[0124]

[0125] Among them, b i，j To correctly associate the trajectory set, the network parameters are trained on the training dataset to obtain the cosine distance, and compared with the cosine distance between the training set baseline to obtain the threshold λ. Then, in the above formula, when b... i，j If the value is less than the threshold λ, it indicates that the target i detected in the current frame is associated with the target j tracked in the previous frame, and the tracking is normal; when b i，j If the value is greater than the threshold λ, it indicates that the association was unsuccessful, and the process continues to determine whether the target is occluded or has disappeared.

[0126] Step S400: If the association is unsuccessful, obtain the depth values ​​of the tracking target and all other targets between the previous frame and the current frame based on the depth image. Determine whether the tracking target is occluded based on the depth values ​​of the tracking target and all other targets between the previous frame and the current frame. If occlusion occurs, obtain the bounding box corresponding to the depth map of the detected target in the current frame and the bounding box corresponding to the depth map of the tracking target to obtain the overlap rate of the target boxes. Perform occlusion matching tracking based on the overlap rate of the target boxes and the preset objective function of associating occluded targets, and predict the occluded target in the current frame as the target tracking result.

[0127] Specifically, when a target is occluded, its appearance changes significantly, leading to a reduction in target features and causing the mobile robot to lose track of the target. This invention proposes an effective occlusion handling mechanism. When occlusion is detected, depth probability is used to spatially constrain the area surrounding the target, and occlusion tracking is achieved through maximum a posteriori probability association matching.

[0128] (1) Occlusion detection

[0129] When multiple targets occlude each other, their RGB appearance is obscured, and different occluded targets can only be distinguished from depth information. The depth information acquired by the ZED camera is the depth information of the target closest to the camera among multiple occluded targets. Therefore, the position region of the target bounding box in the RGB image is mapped to the depth image and set as the depth occlusion region. If other targets are determined to enter this region, occlusion has occurred. The specific determination is as follows: Figure 3 As shown.

[0130] Before occlusion, the maximum peak value of the depth histogram for target 1 was 4000, and for target 2 it was 2500. After occlusion, the maximum peak value of the depth histogram in the occluded area was 2500, and the peak value of the depth histogram decreased from 4000 to 2500. Clearly, the depth histogram of the occluded area of ​​target 1 now shows a new rising peak value of 2500, which is smaller than the pre-occlusion peak value of 4000. Therefore, it is evident that the peak value of the depth histogram after occlusion is significantly reduced. Thus, the magnitude of the depth change can be used to determine whether occlusion has occurred. Let the mean depth change of the tracked target j between frame t-1 and frame t be approximated by a Gaussian distribution. Based on this, determine whether occlusion has occurred.

[0131] In one embodiment, step S400 involves obtaining the depth values ​​of the tracked target and all other targets between the previous and current frames based on the depth image, and determining whether the tracked target is occluded based on the depth values ​​of the tracked target and all other targets between the previous and current frames. Specifically:

[0132]

[0133] in, To track the occlusion changes of target j in frame t, It is the depth value of the tracked target j in frame t. This represents the sum of depth differences of all targets between frame t-1 and frame t. This represents the rate of change of depth value between frame t and frame (t-1). Represents less than w t -ξ t The sum of depth differences of all tracked targets;

[0134] The occlusion judgment criteria are as follows:

[0135]

[0136] Among them, T j This is the criterion for determining occlusion.

[0137] Specifically, This reflects the occlusion change of the tracked target j in frame t; a larger value indicates a higher probability of occlusion. When the target is not occluded, near T j T is close to 1; when the target is occluded, then T j Approaching 0, through T j The value is used to determine whether the target is occluded.

[0138] In one embodiment, step S400 involves obtaining the bounding boxes corresponding to the depth maps of the detected target and the tracked target in the current frame to obtain the overlap rate of the target boxes. Specifically:

[0139] C = b(x) t，i )∩b(z t-1，j )>σ

[0140] Where C is the overlap rate of the target bounding box, σ is the preset overlap rate threshold, and b(x) t，i ) represents the target x detected in the current frame. t，i The bounding box corresponding to the depth map, b(z t-1，j To track target z t-1，j The bounding box corresponding to the depth map, if C is greater than σ, indicates that x t，i With z t-1，j Related matching;

[0141] In step S400, occlusion matching and tracking are performed based on the overlap rate of the target bounding box and the target function of the preset associated occluded target. The occluded target of the current frame is predicted as the target tracking result, specifically:

[0142]

[0143] Among them, S * For the target tracking result, P(z) t-1，j Let P(x) be the prior probability of all tracked targets. t，i |z t-1，j The conditional probability that the currently detected target belongs to the tracked target is calculated by the overlap rate between the detected target and the tracked target box in the previous frame.

[0144] Step S500: Select a target from the target tracking results as the following target, and use a target following algorithm based on visual servo control to keep the center point of the target box as the center point of the field of view. If the target disappears, the mobile robot actively searches according to the target's motion trajectory, re-identifies the target, and makes the mobile robot continue to follow the target.

[0145] Specifically, due to the advantages of ZED cameras, such as high imaging resolution and the ability to acquire long-distance depth images, this invention uses a ZED camera as the vision sensor for a mobile robot, and its intrinsic parameters have been calibrated. Assuming that the lens distortion of the ZED camera is negligible, the camera's intrinsic parameters are represented using a pinhole model. The ZED camera imaging principle diagram is attached. Figure 4 As shown. The target coordinates in the image coordinate system Y are P = (x... cn y cn x cn ), z nIt is obtained from image coordinates and the inherent parameters of the ZED camera.

[0146] z n =f×b / (x l -x r )=f×b / d

[0147] Where f is the camera focal length, and b is the baseline of the left and right cameras, f and b are obtained through prior information or camera calibration. Due to epipolar constraints, the parallax d can be derived from the pixel x of the left camera. l With the right camera pixel x r The correspondence was calculated.

[0148] The algorithm of this invention simplifies the mobile robot platform into a two-wheel differential speed model based on a ZED camera, such as... Figure 5 As shown. Figure 5 The system includes the world coordinate system G, the robot coordinate system PR, the ZED camera coordinate system Z, and the image coordinate system Y. In the figure, C(x, y) is the center point of the mobile robot's motion, D is the distance between the two wheels, and θ is the orientation angle.

[0149] In the world coordinate system G, track the target position O. T With robot position O M The distance between them can be expressed as In the ZED camera coordinate system, the distance from the target to the robot It is based on the tracking target position O T And robot position O M It is obtained through the following formula:

[0150]

[0151] Where R(θ) Q θ C ) represents the rotation matrix from the world coordinate system Q to the camera coordinate system Z, and δd represents the distance between the mobile robot and the camera in the world coordinate system G.

[0152] The aforementioned tracking algorithm completes target tracking and obtains the depth information of the target tracking box. However, directly using the depth information of the tracking target box to calculate the distance between the robot and the target will introduce a lot of background information. Therefore, it is necessary to relocate the target center to find a suitable position for the tracking area and improve the robot's following accuracy.

[0153] The center of the target tracking box is set to This represents the coordinates of all pixels within the target tracking bounding box; the precise location of the region is... Internal relocation, using a circular matrix Get precise location The calculation formula is:

[0154]

[0155] exist maximum value The position (Δx, Δy) is the precise target tracking center (x). * y * ) and (x l y l Positional deviation between (x, y). Precise position of the tracking area (x, y). * y * The calculation is as follows:

[0156] (x * y * )=(x l y l )+(Δx, Δy)

[0157] Get the precise location (x) * y * After that, obtain (x) * y * Let f(t) be the coordinates of the four vertices of the central region bounding box. Calculate the average depth information f(t) corresponding to the center point and vertices, which represents the distance between the mobile robot and the target. Let f be the desired location of the mobile robot. * The error e(t) can be defined as:

[0158] e(t)=f(t)-f * (t)

[0159] The robot's control variable is X control =[U(t)=v t W(t) = w t ], v t w represents the linear velocity of the mobile robot. t The PID controller is designed to represent the angular velocity of the mobile robot.

[0160]

[0161] Where k P k I and k D λ is the PID coefficient, and λ is the adjustment factor.

[0162] In one embodiment, step S500, which states that if the target disappears, the mobile robot actively searches according to the target's trajectory, re-identifies the target, and enables the mobile robot to continue following the target, includes:

[0163] If the target goes out of the mobile robot's field of vision during the tracking process, the mobile robot will retain the re-identification features and position information of the target in the last frame before it disappeared, and continue to move in the direction in which the target disappeared. If the target reappears in the field of vision, the robot will perform target detection and extract the re-identification features of the target and associate and match them with the re-identification features of the target in the last frame before it disappeared, thereby realizing the re-identification of the target and enabling the mobile robot to continue following the target.

[0164] The block diagram of the target following control section is as follows: Figure 6 As shown, the error e(t) is the input of the PID controller, which controls the angular velocity and linear velocity of the mobile robot in real time. The mobile robot maintains a certain distance from the target and stably follows the target. During the following process, if the target goes out of the mobile robot's field of vision, the mobile robot will retain the re-identification features and position information of the target in the last frame before it disappeared, and continue to move in the direction in which the target disappeared. If the target reappears in the field of vision, target detection is performed, and the re-identification features of the target are extracted and associated with the re-identification features of the target in the last frame before it disappeared. This achieves target re-identification, which is equivalent to successful re-detection. Thus, it is possible to re-identify the target after it disappears and stably follow the target.

[0165] The aforementioned mobile robot target following method based on the improved YOLOX mainly consists of three parts: target detection, target tracking, and target following control. First, the YOLOX backbone network Darknet-53 is replaced with MobileNetV2S. Feature maps are obtained by inputting the acquired RGB video sequences into the trained MobileNetV2S network. Then, the re-identification loss function and detection loss function are used to train the re-identification branch and detection branch respectively, thus obtaining the target detection result. Next, an improved Kalman filter is used to obtain the tracking state, and target matching is achieved through trajectory association. Simultaneously, an occlusion discrimination mechanism is introduced; if the target is determined to be occluded, depth probability constraints are added for occluded target tracking and matching. Finally, an active search strategy based on visual servo control is used to complete the mobile robot target following.

[0166] In summary, YOLO-based mobile robot target following algorithms are robust and accurate, but they have weak adaptability and generalization capabilities in changing environments, and their operating speed is low. Traditional mobile robot target following algorithms are fast, but they are prone to target loss when the target undergoes deformation, scale changes, or severe occlusion. Therefore, to achieve stable target following for mobile robots in complex scenarios, this invention proposes an improved YOLOX mobile robot target following method. The main contributions are as follows:

[0167] (1) To improve the accuracy and speed of target detection, a target detection algorithm based on YOLOX-MobileNetV2S network (YOLOX-M2S) is proposed. Cross-entropy loss, regression loss and re-identification loss function are used to train the detection and re-identification branches together.

[0168] (2) To improve the target prediction and update rate, an improved Kalman filter is used to obtain the target tracking status. At the same time, an occlusion detection mechanism based on depth histogram is added, and inter-frame target matching is performed by using depth probability constraints to improve the occlusion tracking accuracy;

[0169] (3) During the target following process, an active search strategy based on visual servo control is proposed, and when the target disappears, re-identification features are introduced for tracking and following to ensure that the mobile robot can stably follow the target.

[0170] The above provides a detailed description of the mobile robot target following method based on the improved YOLOX provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A target-following method for mobile robots based on improved YOLOX, characterized in that, The method includes the following steps: Step S100: Obtain a video sequence, and obtain an RGB image and a depth image from the video sequence; Step S200: Input the RGB image into the target detection neural network to obtain the target detection result; the target detection neural network includes an improved YOLOX backbone network, network layers, target detection branch layers, and target re-identification branch layers. The improved YOLOX backbone network is used to extract features from the RGB image to obtain shallow features and deep features; the network layers perform convolution processing on both the shallow features and the deep features to obtain re-identification branch feature maps and detection branch feature maps; the target detection branch layers are used to perform target classification and regression prediction based on the detection branch feature maps to obtain target detection branch results; the target re-identification branch layers are used to perform target re-identification based on the re-identification branch feature maps to obtain target re-identification results; and the target detection result is obtained based on the target detection branch results and the target re-identification results; wherein, the improved YOLOX backbone network includes seven stages, the core of each stage being a MobileNetV2S network, the MobileNetV2S network including 1×1 convolutional layers, 3×3 convolutional layers, 3×3 depth convolutional layers, 1×1 projective convolutional layers, and ReLU layers. The 1×1 convolutional layer uses a 1×1 convolutional kernel to expand the RGB feature map into a feature map F. 1×1 ∈R H×W×C′ ; The 3×3 convolutional layer applies to the feature map F 1×1 ∈R H×W×C′ The feature map F is obtained by performing depthwise separable convolution. 3×3 ∈R H ′×W′×C′ ; The feature map F 3×3 R H′×W′×C′ By splitting the feature map in half, the number of channels is reduced to half of the original number, resulting in feature map F. (3×3) / 2 ∈R H′×W′×(C′ / 2) and F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) The 3×3 deep convolutional layer will convert the feature map F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) Depth-separable convolution is performed to obtain Will With F′ (3×3) / 2 ∈R H′×W′×(C′ / 2) Combining the two yields a new feature map F3×3∈R H″′×W″′×C′ The 1×1 projective convolutional layer will generate the new feature map F′ 3×3 ∈R H″′×W″′×C′ Convolution yields the feature map F′∈R H″′×W″′×C″ The output feature map of each bottleneck layer is obtained through the ReLU layer. The output feature map F1 corresponding to the first stage is a shallow feature map, and the output feature map F7 corresponding to the seventh stage is a deep feature map. Step S300: Based on the target detection result and the improved Kalman filter, predict the state of the tracked target. Based on the state of the tracked target and the re-identification branch feature map, obtain the re-identification feature map of the updated tracking box for the current detection frame. Calculate the cosine distance between the re-identification features of the current detection frame and the re-identification features of the updated tracking box. Use the cosine distance and a preset distance threshold to determine whether the re-identification features of the current detection frame and the re-identification features of the updated tracking box are related. Specifically, step S300, which predicts the state of the tracked target based on the target detection result and the improved Kalman filter, involves: x t,i =f(x t-1,i )+μ t-1 z t,j =h(x t,i )+v t Where, x t,i To detect the state of the target. β = (u,v,γ,h), where β represents the observed value of the target, (u,v) represents the center position of the bounding box with an aspect ratio of γ and a height of h, and z t,j To track the state of the target, μ t-1 and v t These are the process noise sequence and the measurement noise sequence, respectively. Step S400: If the association is unsuccessful, obtain the depth values ​​of the tracking target and all other targets between the previous frame and the current frame based on the depth image. Determine whether the tracking target is occluded based on the depth values ​​of the tracking target and all other targets between the previous frame and the current frame. If occlusion occurs, obtain the bounding box corresponding to the depth map of the detected target in the current frame and the bounding box corresponding to the depth map of the tracking target to obtain the overlap rate of the target boxes. Perform occlusion matching tracking based on the overlap rate of the target boxes and the preset objective function of the associated occluded target, and predict the occluded target in the current frame as the target tracking result. Step S500: Select a target from the target tracking results as the following target, and use a target following algorithm based on visual servo control to keep the center point of the following target box as the center point of the field of view. If the target disappears, the mobile robot actively searches according to the target's motion trajectory, re-identifies the target, and makes the mobile robot continue to follow the target.

2. The method according to claim 1, characterized in that, The target detection branch layer includes a decoupling head and target branches, classification branches, and regression branches connected to the decoupling head. The network layers perform convolution processing on the deep features to obtain detection branch feature maps. The target detection branch layer is used to perform target classification and regression prediction based on the detection branch feature maps to obtain target detection branch results, including: The network layer processes the deep feature map F7∈R 15×15×320 Convolution processing is performed to obtain the detection branch feature map. The detection branch feature map is processed by the decoupling head to obtain the first feature map. Second feature map and the third feature map The first feature map The data is sent to the target branch, where each feature point represents the probability that the detected target within the corresponding prediction box belongs to the foreground, thereby determining whether the target is foreground or background and obtaining the target branch result. The second feature map The regression branch is sent to predict the center coordinates and height and width (x, y, w, h) of the target box, and the regression branch result is obtained. The third feature map The data is sent to the classification branch to predict and score the category to which the target belongs, thus obtaining the classification branch result. The feature map is obtained by merging and adding the results of the target branch, the regression branch, and the classification branch. As a result of the object detection branch.

3. The method according to claim 2, characterized in that, The network layer performs convolution processing on the shallow features to obtain a re-identification branch feature map. The target re-identification branch layer is used to perform target re-identification based on the re-identification branch feature map to obtain the target re-identification result, including: The network layer features F1 ∈ R for identifying duplicate branches. 240×240×16 Convolution processing is performed to obtain the re-identification branch feature map. The 3×3 convolution kernel is sequentially applied to the re-identification branch feature map. Convolution yields feature maps. A feature map with 128 channels is obtained through 128 sets of 1×1 convolutions. The shallow appearance features at the center point (x, y) of the corresponding target box are extracted from the feature map and used as the target re-identification result.

4. The method according to claim 3, characterized in that Step S200 also includes: The loss value of the target detection branch layer is calculated according to a preset target detection loss function. The target detection branch layer is then updated in reverse based on the loss value to obtain the updated target detection branch layer. The loss value of the target re-identification branch layer is calculated according to the preset re-identification loss function, and the target re-identification branch layer is updated in reverse according to the loss value of the target re-identification branch layer to obtain the updated target re-identification branch layer.

5. The method according to claim 4, characterized in that, The preset target detection loss function is as follows: L detection =λ1L cls +λ2L obj +λ3L reg L reg =1-IoU Where λ1, λ2, and λ3 are balance coefficients, L cls L obj and L reg These are classification loss, target loss, and regression loss, respectively, N. obj Representative used to calculate L obj The loss function represents the total number of video frame targets, y. s This represents the label of the test sample s, with foreground labeled 1 and background labeled 0. s N represents the probability that the test sample s is predicted as foreground, IoU represents the area overlap ratio between the predicted target bounding box and the ground truth target bounding box in the current frame, and N represents the probability that the test sample s is predicted as foreground. cls Representative used to calculate L cls The loss function represents the total number of targets in the video frames, where M represents the number of categories, and y dc The sign function, if the true class of target d in the current video frame is equal to c, y dc If p is 1, then p is 0; otherwise, p is 0. dc The predicted probability that target d in the current frame belongs to category c; The preset re-identification loss function is as follows: in, The label encoding for the target ground truth bounding box is given by the feature distribution vector C = {c(b), b ∈ [1, B]}, N re-id B represents the total number of categories to which the current target belongs, and B is the ID of all identities in the training data.

6. The method according to claim 5, characterized in that, In step S300, cosine distance and a preset distance threshold are used to determine whether the re-identification features of the current detection frame and the re-identification features of the predicted updated tracking box are related. Specifically: When the cosine feature is less than or equal to the preset distance threshold, the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box are successfully associated; when the cosine feature is greater than the preset distance threshold, the re-identification feature of the current detection frame and the re-identification feature of the predicted updated tracking box are not successfully associated.

7. The method according to claim 6, characterized in that, In step S400, the depth values ​​of the tracking target and all other targets between the previous frame and the current frame are obtained based on the depth image. Based on these depth values, it is determined whether the tracking target is occluded. Specifically: in, To track the occlusion changes of target j in frame t, It is the depth value of the tracked target j in frame t. This represents the sum of depth differences of all targets between frame t-1 and frame t. This represents the rate of change of depth values ​​between frame t and frame (t-1). Represents less than w t -ξ t The sum of depth differences of all tracked targets; The occlusion judgment criteria are as follows: Among them, T j This is the criterion for determining occlusion.

8. The method according to claim 7, characterized in that, In step S400, the bounding boxes corresponding to the depth maps of the detected target and the tracked target in the current frame are obtained to calculate the overlap rate of the target boxes. Specifically: C=b(x t,i )∩b(z t-1,j )>σ Where C is the overlap rate of the target bounding box, σ is the preset overlap rate threshold, and b(x) t,i ) represents the target x detected in the current frame. t,i The bounding box corresponding to the depth map, b(z t-1,j To track target z t-1,j The bounding box corresponding to the depth map, if C is greater than σ, indicates that x t,i With z t-1,j Related matching; In step S400, occlusion matching and tracking are performed based on the overlap rate of the target bounding box and the target function of the preset associated occlusion target. The occlusion target of the current frame is predicted as the target tracking result. Specifically: Among them, S * For the target tracking result, P(z) t-1,j Let P(x) be the prior probability of all tracked targets. t,i ∣z t-1,j The conditional probability that the currently detected target belongs to the tracked target is calculated by the overlap rate between the detected target and the tracked target box in the previous frame.

9. The method according to claim 8, characterized in that, If the target disappears in step S500, the mobile robot actively searches according to the target's movement trajectory, re-identifies the target, and enables the mobile robot to continue following the target, including: If the target goes out of the mobile robot's field of vision during the tracking process, the mobile robot will retain the re-identification features and position information of the target in the last frame before it disappeared, and continue to move in the direction in which the target disappeared. If the target reappears in the field of vision, the robot will perform target detection and extract the re-identification features of the target and associate and match them with the re-identification features of the target in the last frame before it disappeared, thereby realizing the re-identification of the target and enabling the mobile robot to continue following the target.

Images