Robot following sensing method based on fine-grained pedestrian re-identification and related device

Abstract

The invention discloses a robot following sensing method based on fine-grained pedestrian re-identification and a related device. The method comprises the following steps: acquiring a video, and extracting a pixel mask area of each pedestrian in each frame of image to obtain a pedestrian mask; a unique tracking ID is allocated to each segmented pedestrian mask in continuous frames in the video, and each pedestrian mask allocated to the tracking ID is input into the trained fine-grained pedestrian re-identification model; the fine-grained pedestrian re-identification model comprises a feature extraction network, a global branch and a local branch, wherein the global branch and the local branch share two parameters; and extracting a feature vector through the trained fine-grained pedestrian re-identification model, performing feature comparison on the feature vector and a pedestrian feature vector pre-stored in an owner feature library, and returning a corresponding tracking ID as a retrieval result based on a comparison result. According to the invention, a more accurate pedestrian identification function and a following function can be realized.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot following perception method and related apparatus based on fine-grained pedestrian re-identification. Background Technology

[0002] With the rapid development of intelligent mobile platforms such as service robots, logistics robots, and personal companion robots, achieving stable and autonomous following of specific individuals by robots has become a key requirement. The core challenge of this task lies in the fact that robots need to overcome interference such as occlusion, changes in lighting, changes in target appearance, and changes in perspective in complex dynamic environments, and continuously and accurately perceive and track specific targets.

[0003] Traditional robot following solutions mainly rely on the following types of technologies, but all of them have obvious limitations:

[0004] Color / feature-based tracking methods, such as mean shift and correlation filtering algorithms, track targets using color histograms or manually designed features (such as HOG and SIFT). These methods are computationally simple, but are extremely sensitive to target occlusion, rapid movement, and changes in appearance; once the target is lost, it is difficult to re-identify it.

[0005] Beacon-based methods require target personnel to wear or carry RFID, UWB, color-coded markers, or QR codes. While stable, this method significantly sacrifices the naturalness and convenience of the user experience and lacks universality.

[0006] The detection and tracking paradigm utilizes deep learning object detectors (such as YOLO and SSD) to continuously detect the "human" category and combines them with trackers (such as Kalman filtering and SORT / DeepSORT) for data association. However, in densely populated or long-term occlusion scenarios, this method cannot distinguish between different pedestrians and is prone to target identity switching problems, meaning the robot may mistakenly follow other people.

[0007] To overcome the aforementioned limitations, researchers have recently introduced person re-identification technology into robot following systems, forming a more advanced perception paradigm. Person re-identification aims to solve the problem of matching pedestrians across different cameras and time periods, that is, determining whether pedestrians in different image or video sequences belong to the same person. Its technological advantages are:

[0008] Strong discriminative ability: It can learn to distinguish fine-grained visual features (such as clothing, body shape, etc.) of different pedestrians, rather than just recognizing the category of "person".

[0009] Robustness to occlusion and loss: When a target temporarily leaves the field of vision or reappears after being occluded, the pedestrian re-identification model can re-identify it based on appearance features, enabling the robot to resume tracking rather than permanently losing it.

[0010] No specific markers required: Enables seamless following of pedestrians in their natural state, resulting in a better user experience.

[0011] However, existing pedestrian re-identification schemes still face challenges:

[0012] Adaptability to drastic changes in appearance: When the target pedestrian changes clothes, carries large items, or the lighting conditions change drastically, the pedestrian re-identification model based on appearance features may fail.

[0013] Feature confusion and similarity interference: In scenarios where there are pedestrians with similar appearances, mismatches are likely to occur.

[0014] Poor performance in recognizing small targets at long distances: Due to low image resolution, lack of feature information, target occlusion, and drastic changes in pose, recognizing small targets at long distances is one of the core challenges faced by robot following algorithms based on pedestrian re-identification, and its performance will decline significantly.

[0015] Therefore, there is an urgent need in this field for an efficient and robust autonomous following algorithm that can operate stably on resource-constrained robotic platforms and effectively cope with the challenges of complex real-world scenarios. Summary of the Invention

[0016] To address the aforementioned technical issues, this invention proposes a robot following perception method and related apparatus based on fine-grained pedestrian re-identification. By optimizing pedestrian re-identification feature learning, the learned features include both the overall features of the pedestrian and discriminative local features, thereby further improving the performance and practicality of the robot following system.

[0017] To achieve the above objectives, the technical solution of the present invention is as follows:

[0018] A robot following perception method based on fine-grained pedestrian re-identification includes the following steps:

[0019] Acquire video, extract the pixel mask region of each pedestrian in each frame of image to obtain the pedestrian mask;

[0020] Assign a unique tracking ID to each segmented pedestrian mask within consecutive frames of the video, and input each pedestrian mask assigned to a tracking ID into a trained fine-grained pedestrian re-identification model; the fine-grained pedestrian re-identification model includes a feature extraction network and two parameters-shared global and local branches;

[0021] Feature vectors are extracted using the trained fine-grained pedestrian re-identification model. These feature vectors are then compared with pedestrian feature vectors pre-stored in the owner feature database. Based on the comparison results, the corresponding tracking ID is returned as the retrieval result.

[0022] Preferably, the step of extracting the pixel mask region of each pedestrian in each frame of the image to obtain the pedestrian mask includes the following steps:

[0023] The YOLOv8 network is used to identify each pedestrian in each frame of the image. The YOLOv8 network includes a backbone network, a neck network, and a head network. Multi-scale feature maps are extracted through the backbone network. The neck network performs upsampling and downsampling operations on the multi-scale features to obtain fused features. The detection head in the head network predicts the fused features to generate the coordinates of the target bounding box, the classification probability, and the confidence score of the target.

[0024] Based on the coordinates, classification probability, and target confidence of the target bounding box, the pedestrian and background in the target bounding box are segmented to obtain a pixel mask region that retains only the pedestrian.

[0025] Preferably, the ByteTrack multi-target tracking algorithm is used to assign a unique tracking ID to each segmented pedestrian mask in consecutive frames of the video.

[0026] Preferably, the training process of the fine-grained pedestrian re-identification model includes the following steps:

[0027] The features of the pedestrian mask are extracted using a feature extraction network to obtain a global feature map;

[0028] The global feature map is input into both the global branch and the local branch. In the global branch, the global feature map is input into a fully connected layer for classification prediction, and the global classification loss is calculated. In the local branch, the global feature map is aggregated along the channel dimension to obtain a global activation map. Multiple sliding windows are generated by operating on the global activation map using anchor boxes of different sizes. The average activation value of each sliding window is calculated and sorted, and non-maximum suppression is performed to obtain N local windows. The pixel regions corresponding to the N local windows on the pedestrian mask are input into a feature extraction network to extract features and then input into a fully connected layer for classification prediction, and the corresponding local classification loss is calculated. The global classification loss and the N local classification losses are summed to obtain the total loss. The model parameters are updated using the backpropagation algorithm until the model converges.

[0029] Preferably, the feature extraction network is the backbone network of ResNet-50.

[0030] Preferably, both the global classification loss and the local classification loss are cross-entropy loss.

[0031] Preferably, the step of comparing the feature vector with the pedestrian feature vector pre-stored in the owner feature database, and returning the corresponding tracking ID as the retrieval result based on the comparison result, includes the following steps:

[0032] The cosine similarity is calculated for each of the feature vectors and the pedestrian feature vectors pre-stored in the owner feature library.

[0033] Return the tracking ID corresponding to the highest cosine similarity.

[0034] Based on the above, the present invention also discloses a robot following perception device based on fine-grained pedestrian re-identification, comprising:

[0035] The pedestrian segmentation module is used to acquire video, extract the pixel mask region of each pedestrian in each frame of the image, and obtain the pedestrian mask;

[0036] The target tracking module is used to assign a unique tracking ID to each segmented pedestrian mask in consecutive frames of the video, and input each pedestrian mask assigned to the tracking ID into the trained fine-grained pedestrian re-identification model; the fine-grained pedestrian re-identification model includes a feature extraction network and two parameters-shared global and local branches;

[0037] The re-identification module is used to extract feature vectors through the trained fine-grained pedestrian re-identification model, compare the feature vectors with the pedestrian feature vectors pre-stored in the owner feature library, and return the corresponding tracking ID as the retrieval result based on the comparison result.

[0038] Based on the above, the present invention also discloses a robot, comprising:

[0039] body;

[0040] A control system communicating with the fuselage, the control system including a memory and a processor, the memory for storing a computer program; the processor for executing the computer program to implement any of the methods described above.

[0041] Based on the above, the present invention also discloses a readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above.

[0042] Based on the above technical solution, the beneficial effects of the present invention are:

[0043] 1. This invention can segment and guide pedestrian tracking and re-identification: The accurate target bounding box or mask provided by the pedestrian segmentation module provides a high-quality initial position for the tracker and eliminates background interference for the pedestrian re-identification module, thus improving the quality of feature extraction. Traditional pedestrian re-identification processes use object detection methods to extract the 2D bounding box of each pedestrian. However, due to changes in human posture or movement, the pixels within the 2D bounding box usually contain a large amount of background noise, which greatly affects the performance of the feature extraction model.

[0044] 2. This invention enables a shift from "coarse-grained localization" to "fine-grained pedestrian identification": 1) Strong robustness to occluded and partially visible targets: Traditional trackers or global pedestrian re-identification models experience a sharp performance drop when the target is partially occluded. Fine-grained pedestrian re-identification models, however, learn local features with attention mechanisms (e.g., extracting high-value local pixel regions), enabling them to identify targets based on a single visible local area (e.g., a hat, a backpack). This effectively solves the tracking interruption problem in occluded scenarios. 2) High resolution against similar interfering objects: In crowds, pedestrians with similar clothing colors and styles often appear. Traditional methods are prone to mis-tracking. Fine-grained pedestrian re-identification can focus on subtle differences (e.g., the texture and style of different clothing), thus accurately distinguishing different individuals with similar appearances. 3) Adaptability to changes in target appearance: When a target undergoes changes in posture (turning around, crouching), lighting, or distance, its global appearance changes drastically. Fine-grained pedestrian re-identification focuses on local features (such as shoe style and backpack shape) that are relatively stable and therefore not sensitive to such changes.

[0045] 3. This invention employs a multi-granularity feature hierarchical fusion and alignment strategy: by using a sliding window to extract local features from the global feature map and jointly calculating the global classification loss and local classification loss, it directly optimizes the core difficulties of pedestrian re-identification from both the feature extraction and loss function levels. 1) Achieving implicit, adaptive part alignment to alleviate pose changes and occlusion problems: Pedestrian images often exhibit significant pose changes (front, side, back), viewpoint differences, and partial occlusion. When directly comparing the global features of two images, severe misalignment of body parts, such as mismatching the head features of one image with the leg features of another, leads to distorted similarity calculations. Using a sliding window on the final convolutional feature map is equivalent to dense sampling in different regions of the original image. Each window focuses on a local region. This method does not rely on precise keypoint detection or part segmentation (which are inherently difficult and error-prone tasks). Instead, it ensures, through a "netting" approach, that some windows always cover well-aligned, visible key parts. During training, the local classification loss forces the model to make the features of the same pedestrian as similar as possible within the corresponding window, and the features of different pedestrians as different as possible. 2) Force the model to learn discriminative local nuances rather than over-relying on global appearance: Models are prone to "shortcut learning," that is, over-relying on the easiest-to-learn global statistical features, such as dominant clothing color. This leads to a sharp drop in performance in "clothing change" scenarios and an inability to distinguish between similar but different people. A single global classification loss function (such as classification loss) tends to optimize overall features. Introducing local classification loss (e.g., calculating a classification loss independently for each local feature generated by a sliding window) is equivalent to adding multiple "optimization objectives" to the model. To minimize all local classification losses simultaneously, the model must ensure that the features of every local region from head to toe are discriminative. It cannot just learn the feature of "red shirt" well; it must also capture nuances such as "unique print on the shirt," "pleat pattern on the pants," and "brand logo on the shoes." 3) Provide richer supervision signals to improve the structure of the feature space: On difficult samples (different pedestrians with extremely similar appearances, or images of the same pedestrian under different lighting and viewing angles), the feature boundaries learned by the model may be blurred, leading to misclassification. Using multiple sliding windows is equivalent to providing each training sample with one global classification loss and multiple local classification losses simultaneously. This significantly increases the quantity and diversity of the supervision signals provided by each training sample. The global classification loss is responsible for separating pedestrians of different identities at a macro level; while each local classification loss further refines the feature space within its focused local region. Ultimately, in the feature space, not only are the feature vectors of the entire sample well separated, but the corresponding features of each local component are also required to be separated. This makes the decision boundary clearer and more stable. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of a robot following perception method based on fine-grained pedestrian re-identification in one embodiment;

[0047] Figure 2 This is a schematic block diagram of a robot following perception method based on fine-grained pedestrian re-identification in one embodiment;

[0048] Figure 3 This is a schematic diagram of a pedestrian segmentation mask in one embodiment;

[0049] Figure 4 This is a schematic diagram of the local attention candidate module processing in one embodiment. Detailed Implementation

[0050] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0051] like Figures 1 to 4 As shown, this embodiment provides a robot following perception method based on fine-grained pedestrian re-identification, including the following steps:

[0052] Step 100: Acquire video, extract the pixel mask region of each pedestrian in each frame of the image, and obtain the pedestrian mask.

[0053] In this embodiment, an object segmentation branch is added to the object detection model. This allows it to not only perform object localization and classification but also predict a pixel-level mask for each detected pedestrian, accurately outlining their contours. The object detection model uses the YOLOv8 network, but is not limited to it; those skilled in the art can select other models for pedestrian recognition based on the specific circumstances.

[0054] Step 200: Assign a unique tracking ID to each segmented pedestrian mask in consecutive frames of the video, and input each pedestrian mask assigned to a tracking ID into the trained fine-grained pedestrian re-identification model; the fine-grained pedestrian re-identification model includes a feature extraction network and two parameters-shared global and local branches.

[0055] In this embodiment, the target tracking algorithm adopts the ByteTrack tracking scheme, which maximizes the information value of the detection results themselves through an extremely ingenious "secondary association" design, greatly improving the performance of multi-target tracking in complex scenarios without significantly increasing the computational cost.

[0056] Step 300: Extract feature vectors using the trained fine-grained pedestrian re-identification model, compare the feature vectors with the pedestrian feature vectors pre-stored in the owner feature library, and return the corresponding tracking ID as the retrieval result based on the comparison result.

[0057] In this embodiment, an attention mechanism is introduced during the training process of the fine-grained pedestrian re-identification model, enabling the feature extraction model to learn to focus on information-rich local areas while paying attention to the whole.

[0058] During inference, pedestrian segmentation is performed first, extracting the pixel mask region of each pedestrian from the input image, such as... Figure 3 Then, based on the target tracking algorithm, a unique tracking ID is assigned to each segmented pedestrian mask within consecutive frames. Subsequently, the pedestrian pixel region assigned to each tracking ID is input into the fine-grained pedestrian re-identification model to extract a normalized 2048-dimensional feature vector. This vector is then compared with the pedestrian feature vectors pre-stored in the feature library for retrieval. The cosine similarity between each pair of vectors is calculated, and the feature vector with the highest similarity is found. Its ID is returned as the retrieval result.

[0059] The robot can achieve more accurate pedestrian recognition and following functions by relying solely on visual image input without relying on other complex sensors, using the aforementioned pedestrian segmentation, pedestrian tracking, and fine-grained pedestrian re-identification.

[0060] In one embodiment of a robot following perception method based on fine-grained pedestrian re-identification, a specific process is also provided for extracting the pixel mask region of each pedestrian in each frame of image to obtain the pedestrian mask. This process includes the following steps:

[0061] The YOLOv8 network is used to identify each pedestrian in each frame of the image. The YOLOv8 network includes a backbone network, a neck network, and a head network. Multi-scale feature maps are extracted through the backbone network. The neck network performs upsampling and downsampling operations on the multi-scale features to obtain fused features. The detection head in the head network predicts the fused features to generate the coordinates of the target bounding box, the classification probability, and the confidence score of the target.

[0062] Based on the coordinates, classification probability, and target confidence of the target bounding box, the pedestrian and background in the target bounding box are segmented to obtain a pixel mask region that retains only the pedestrian.

[0063] This embodiment draws inspiration from the classic YOLOv8 network, which includes a backbone network responsible for extracting multi-level feature maps from the image. Then, the neck structure receives multi-level feature maps of different scales from the backbone network, performing upsampling and downsampling operations to effectively fuse deep, high-level semantic features (beneficial for classification) with shallow, fine-grained positional features (beneficial for localization). This allows the model to achieve good detection results for targets of different sizes. Finally, the head structure directly predicts the coordinates of the target bounding box, the classification probability, and the target's confidence level. The segmentation branch predicts a segmentation mask prototype and mask coefficients. During inference, the mask coefficients are multiplied by the mask prototype, and then upsampling is performed to obtain an accurate binary mask corresponding to the target region in the original image.

[0064] The precise bounding boxes or masks provided by pedestrian segmentation offer high-quality initialization locations for subsequent trackers and eliminate background interference for fine-grained pedestrian re-identification, thereby improving the quality of feature extraction.

[0065] In one embodiment of a robot following perception method based on fine-grained pedestrian re-identification, a specific process is also provided for assigning a unique tracking ID to each segmented pedestrian mask in consecutive frames of a video using the ByteTrack multi-target tracking algorithm. This process includes the following steps:

[0066] The first association involves matching high-resolution detection boxes with existing trajectories. The input consists of the tracking trajectory from the previous frame, which is in an "active" state, and high-confidence detection boxes (above 0.6) predicted by the object detection model. Then, using IoU as the similarity metric, the optimal match is performed using the Hungarian algorithm. Unmatched high-resolution detection boxes may represent new targets and will be used to create new trajectories. Trajectories that do not find a matching high-resolution detection box in the current frame are not immediately deleted but instead proceed to the second association.

[0067] The second association process involves matching low-scoring detection boxes (between 0.1 and 0.6) with "unmatched trajectories," also using the IoU + Hungarian algorithm. The matched low-scoring detection boxes are used to update the trajectory, successfully "recovering" the target whose score had decreased due to occlusion and maintaining trajectory continuity. Trajectories that fail to match in two consecutive association rounds may indicate that the target has disappeared. These are marked as "lost" and permanently deleted after a certain period (e.g., 30 frames).

[0068] The target tracking algorithm uses the ByteTrack scheme, which does not require complex appearance feature extraction, but only relies on the position information (IoU) of the detection box and the motion model (Kalman filtering). This makes it extremely fast and computationally inefficient.

[0069] In one embodiment of the robot following perception method based on fine-grained person re-identification, a training process for the fine-grained person re-identification model is also provided, which includes the following steps:

[0070] The features of the pedestrian mask are extracted using a feature extraction network to obtain a global feature map;

[0071] The global feature map is input into both the global branch and the local branch. In the global branch, the global feature map is input into a fully connected layer for classification prediction, and the global classification loss is calculated. In the local branch, the global feature map is aggregated along the channel dimension to obtain a global activation map. Multiple sliding windows are generated by operating on the global activation map using anchor boxes of different sizes. The average activation value of each sliding window is calculated and sorted, and non-maximum suppression is performed to obtain N local windows. The pixel regions corresponding to the N local windows on the pedestrian mask are input into a feature extraction network to extract features and then input into a fully connected layer for classification prediction, and the corresponding local classification loss is calculated. The global classification loss and the N local classification losses are summed to obtain the total loss. The model parameters are updated using the backpropagation algorithm until the model converges.

[0072] In this embodiment, the fine-grained pedestrian re-identification model includes a ResNet-50 backbone network as the feature extraction network, with two parameter-sharing branches: a global branch and a local branch. The global branch learns the overall features of the pedestrian pixel region. Then, based on the feature map of the target image, the local branch extracts multiple local windows with high discriminative power and low redundancy through a local attention candidate module. Within the local branch, local patches are cropped from the original image based on the local windows, and these patches are fed into the ResNet-50 feature extraction network to extract features, which are then processed through a fully connected layer for classification and prediction. This paradigm forces the entire model to focus on fine-grained features at different scales and locations.

[0073] During training, for each input pedestrian mask image, its resolution is scaled up to... The size, so that the shape of the input data is... Then, through the ResNet-50 feature extraction model, a result is obtained. global feature map The number of dimensions is The resolution is After performing global average pooling on this feature map, we obtain a... The feature vector can be used as a global representation of the current feature map. This feature vector is then input into a fully connected layer for classification prediction, and the global classification loss is calculated. The global classification loss uses cross-entropy loss, as shown in the following formula:

[0074]

[0075] In the formula, It is the last branch in the global branch. The class probabilities output by the layer. These are actual category labels.

[0076] By observing this global feature map, it can be found that high activation value regions on the map are usually key parts for pedestrian identification, such as the pedestrian's head, objects carried, and legs. Therefore, this application designs a local attention candidate module. First, along the channel dimension, the global feature map... Perform aggregation to obtain the global activation graph. The formula is as follows:

[0077]

[0078] in It is the number of channels. It is the first Feature maps for each channel. Given the global feature map. The resolution is Seven anchor point frames of different sizes are used, with dimensions of [missing information]. Then, the anchor point bounding boxes are slid one by one across the global feature map from top to bottom and from left to right, resulting in a set of local windows of different positions and sizes on the global feature map, such as... Figure 4 As shown. For each local window, its average activation value is calculated. The formula is as follows:

[0079]

[0080] in It is the first Local feature maps of each window It is within this window The activation value of the location. This refers to the height and width of this window. Then, based on... The values ​​are used to sort all local windows. A higher value indicates richer information within the window. Given that the activation values ​​of adjacent windows are relatively close, we used Non-Maximum Suppression (NMS) to remove adjacent and redundant local feature maps, ultimately retaining a fixed number of local windows. The pixel regions corresponding to these local windows in the original image were then scaled down to... The resolution is input into the ResNet-50 feature extraction model to obtain a The feature vectors are then input into a fully connected layer for classification prediction. The local classification loss can be calculated by comparing them with the actual target category. The local classification loss also uses cross-entropy loss, as shown in the following formula:

[0081]

[0082] in, It is the last local branch The class probabilities output by the layer correspond to the first... There are 10 windows in total. One window, These are actual category labels.

[0083] Total loss Defined as:

[0084]

[0085] During training, we used local branches and local feature maps to force the model to focus on information-rich local regions. However, during inference, to reduce computational cost, local branches are no longer needed; instead, the feature vector predicted by the global branch is directly extracted as the feature representation of the current pedestrian image. Then, the cosine similarity between this feature vector and feature vectors in the feature library is calculated one by one, and the tracking ID with the highest similarity is returned.

[0086] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0087] Based on the same inventive concept, this application also provides an apparatus for implementing the above-described robot following perception method based on fine-grained pedestrian re-identification. The solution provided by this apparatus is similar to the implementation described in the above method, and therefore will not be repeated here.

[0088] In one embodiment, a robot following perception device based on fine-grained pedestrian re-identification is also provided, comprising:

[0089] The pedestrian segmentation module is used to acquire video, extract the pixel mask region of each pedestrian in each frame of the image, and obtain the pedestrian mask;

[0090] The target tracking module is used to assign a unique tracking ID to each segmented pedestrian mask in consecutive frames of the video, and input each pedestrian mask assigned to the tracking ID into the trained fine-grained pedestrian re-identification model; the fine-grained pedestrian re-identification model includes a feature extraction network and two parameters-shared global and local branches;

[0091] The re-identification module is used to extract feature vectors through the trained fine-grained pedestrian re-identification model, compare the feature vectors with the pedestrian feature vectors pre-stored in the owner feature library, and return the corresponding tracking ID as the retrieval result based on the comparison result.

[0092] In the above embodiments, the various modules of the robot following perception device based on fine-grained pedestrian re-identification can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0093] In one embodiment, a robot is also provided, comprising:

[0094] body;

[0095] A control system communicating with the fuselage, the control system including a memory and a processor, the memory for storing computer programs; the processor for executing the computer programs to implement the methods as described in the above embodiments.

[0096] In one embodiment, a readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps as described in all the above method embodiments.

[0097] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0098] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A robot following perception method based on fine-grained pedestrian re-identification, characterized in that, Includes the following steps: Acquire video, extract the pixel mask region of each pedestrian in each frame of image to obtain the pedestrian mask; Assign a unique tracking ID to each segmented pedestrian mask within consecutive frames of the video, and input each pedestrian mask assigned to a tracking ID into a trained fine-grained pedestrian re-identification model; the fine-grained pedestrian re-identification model includes a feature extraction network and two parameters-shared global and local branches; Feature vectors are extracted using the trained fine-grained pedestrian re-identification model. These feature vectors are then compared with pedestrian feature vectors pre-stored in the owner feature database. Based on the comparison results, the corresponding tracking ID is returned as the retrieval result.

2. The robot following perception method based on fine-grained pedestrian re-identification according to claim 1, characterized in that, The step of extracting the pixel mask region of each pedestrian in each frame of the image to obtain the pedestrian mask includes the following steps: The YOLOv8 network is used to identify each pedestrian in each frame of the image. The YOLOv8 network includes a backbone network, a neck network, and a head network. Multi-scale feature maps are extracted through the backbone network. The neck network performs upsampling and downsampling operations on the multi-scale features to obtain fused features. The fused features are predicted by the detection head in the head network to generate the coordinates of the target bounding box, the classification probability, and the confidence level of the target; Based on the coordinates, classification probability, and target confidence of the target bounding box, the pedestrian and background in the target bounding box are segmented to obtain a pixel mask region that retains only the pedestrian.

3. The robot following perception method based on fine-grained pedestrian re-identification according to claim 1, characterized in that, The ByteTrack multi-target tracking algorithm is used to assign a unique tracking ID to each segmented pedestrian mask in consecutive frames of the video.

4. The robot following perception method based on fine-grained pedestrian re-identification according to claim 1, characterized in that, The training process of the fine-grained pedestrian re-identification model includes the following steps: The features of the pedestrian mask are extracted using a feature extraction network to obtain a global feature map; The global feature map is input into a global branch and a local branch respectively. In the global branch, the global feature map is input into a fully connected layer for classification prediction, and the global classification loss is calculated. In the local branch, the global feature map is aggregated along the channel dimension to obtain a global activation map. Multiple sliding windows are generated by operating on the global activation map using anchor boxes of different sizes. The average activation value of each sliding window is calculated and sorted, and non-maximum suppression is performed to obtain N local windows. The pixel regions corresponding to the N local windows on the pedestrian mask are input into a feature extraction network to extract features and then input into a fully connected layer for classification prediction, and the corresponding local classification loss is calculated. The total loss is obtained by summing the global classification loss and the N local classification losses. The model parameters are then updated using the backpropagation algorithm until the model converges.

5. The robot following perception method based on fine-grained pedestrian re-identification according to claim 4, characterized in that, The feature extraction network is the backbone network of ResNet-50.

6. The robot following perception method based on fine-grained pedestrian re-identification according to claim 4, characterized in that, Both the global classification loss and the local classification loss adopt the cross-entropy loss.

7. The robot following perception method based on fine-grained pedestrian re-identification according to claim 1, characterized in that, The step of comparing the feature vector with the pedestrian feature vector pre-stored in the owner feature database and returning the corresponding tracking ID as the retrieval result based on the comparison result includes the following steps: The cosine similarity is calculated for each of the feature vectors and the pedestrian feature vectors pre-stored in the owner feature library. Return the tracking ID corresponding to the highest cosine similarity.

8. A robot following perception device based on fine-grained pedestrian re-identification, characterized in that, include: The pedestrian segmentation module is used to acquire video, extract the pixel mask region of each pedestrian in each frame of the image, and obtain the pedestrian mask; The target tracking module is used to assign a unique tracking ID to each segmented pedestrian mask in consecutive frames of the video, and input each pedestrian mask assigned to the tracking ID into the trained fine-grained pedestrian re-identification model; the fine-grained pedestrian re-identification model includes a feature extraction network and two parameters-shared global and local branches; The re-identification module is used to extract feature vectors through the trained fine-grained pedestrian re-identification model, compare the feature vectors with the pedestrian feature vectors pre-stored in the owner feature library, and return the corresponding tracking ID as the retrieval result based on the comparison result.

9. A robot, characterized in that, include: body; A control system that communicates with the fuselage, the control system including a memory and a processor, the memory being used to store computer programs; The processor is configured to implement the method as described in any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images