A robotic summoning and following system and method

Abstract

This invention discloses a robot summoning and following system and method based on visual edge computing, belonging to the fields of edge computing and embodied intelligent robots. The system uses an RK3588S as the edge computing platform and an STM32 as the motion controller, employing a USB camera for visual perception to construct a localized closed loop of "detection-tracking-decision-execution". A lightweight YOLOv8-Pose improved LSCD detection head is used, and INT8 operation mode is employed for inference acceleration. Summoning and stopping posture interaction commands are realized through multi-frame determination of the angle between key points of the upper limbs, enabling autonomous selection and switching of the following target. Based on the ByteTrack algorithm, HSV-LBP features of the rigid body region of the torso are integrated to improve the robustness of re-identification in occlusion and target loss scenarios. A multi-threaded parallel architecture is adopted to decouple inference and post-processing, significantly improving hardware resource utilization and system real-time performance. Standardized serial port protocol communication supports coordinated switching between vision and LiDAR, and the PID algorithm is optimized to achieve smooth robot following. This invention requires no cloud or external positioning equipment, exhibits low latency response, stable tracking, and natural interaction in complex dynamic scenarios, and can be widely applied in industry, military, and service sectors.

Description

Technical Field

[0001] This invention relates to the field of edge computing, and more particularly to a robot summoning and following system and method based on visual edge computing, which determines the posture and position information of a pedestrian target by combining hardware and software, enabling a mobile robot to perform a following task. Background Technology

[0002] Embodied intelligence emphasizes that robots interact with their environment in real time through their own hardware (such as cameras and sensors), forming a closed-loop "perception-decision-execution" chain, enabling robots to dynamically adapt to complex scenarios and accurately respond to human needs. Pedestrian following, a typical application scenario of embodied intelligence, is widely applicable to service robots, assistive robots, and other fields. It requires robots to autonomously identify target pedestrians, track changes in position in real time, and adjust their own motion parameters to achieve unconstrained, low-interference, and highly synchronized following movements. This process places high demands on the real-time performance of data processing, the robustness of target detection, and the flexibility of system response. Currently, robot following methods rely more on UWB positioning devices or require fixed motion paths, limiting the flexibility and intelligence of robot following systems. Traditional local robot computing modes are often limited by hardware computing power, making it difficult to efficiently complete complex tasks such as target recognition. Cloud computing has computing power advantages and is often used for complex video data processing and analysis, but it is easily affected by network bandwidth limitations leading to data transmission delays and also poses privacy risks, making it difficult to meet the millisecond-level response requirements of following tasks. The main technical challenges faced by autonomous robot systems include insufficient responsiveness, limited edge computing hardware and software resources, and issues with the stability of target pedestrian recognition, triggering, and tracking in dynamic scenes.

[0003] Target detection and tracking technologies based on deep learning principles are enabling technologies for intelligent control of mobile robot summoning and following systems. Summoning and following methods based on human posture semantic understanding are less commonly used on mobile robot platforms. Rapidly locating pedestrian targets and outputting their bounding boxes, confidence scores, and key feature information helps in deterministic target selection, subsequent posture analysis, and tracking continuity. Single-stage target detection methods, primarily the YOLO series, have significant influence; however, deploying YOLO directly on edge nodes can easily lead to a decrease in detection frame rate due to insufficient computing power, causing delays in the following system's response. Target tracking technology associates the same pedestrian target in consecutive frames, updating the target's position, scale, and motion state information in real time, maintaining stable tracking even under scenarios involving brief target occlusion, posture changes, scale scaling, or background interference. The DeepSORT method uses Kalman filtering and appearance feature matching to achieve multi-target tracking and identity maintenance; however, its deep semantic feature extraction process is computationally expensive, and the model parameters are typically high, making it prone to competing with target detection algorithms for edge computing platform resources, thus delaying the wake-up and following action. The ByteTrack algorithm overcomes computational and real-time bottlenecks by efficiently utilizing motion information. It leverages the Kalman filter model to predict the target's motion state, generating candidate bounding boxes. The algorithm then directly matches the detected bounding boxes from the target detection algorithm with the predicted bounding boxes based on motion information, eliminating the need for complex deep semantic appearance feature extraction and matching processes. This motion-matching design significantly reduces the algorithm's computational overhead.

[0004] Threads, as the basic unit of processor scheduling and execution, rely on shared resources from processes, have low switching overhead, and can directly read and write shared variables for communication. However, the crash of a single thread will cause the entire process to exit. Single-core CPUs achieve macro-level concurrency and micro-level alternating execution through time-slice rotation, improving resource utilization during I / O wait times. Multi-core CPUs can schedule to different cores to achieve true parallelism, optimizing the computational performance of CPU-intensive tasks. To solve the race condition problem caused by multi-threaded resource sharing, synchronization and mutual exclusion mechanisms are needed to ensure the atomicity, visibility, and ordering of operations. Common mechanisms include mutexes, semaphores, condition variables, read-write locks, and atomic operations. Multithreading is suitable for I / O-intensive tasks, background asynchronous tasks, and multi-core parallel computing. For I / O-intensive tasks such as deep learning processing of video frame data, this invention achieves multi-threaded parallel processing by splitting tasks, improving computational resource utilization during thread wait intervals, facilitating resource sharing, and significantly reducing program response time.

[0005] By deploying data processing, storage, and decision-making all at the robot's edge nodes, such as embedded processors and edge computing modules, the robot system can directly receive multi-source data from visual sensors and other sources. Edge computing enables a localized closed loop for data acquisition, analysis, processing, and command output, significantly reducing transmission latency and network dependence. At the same time, targeted optimization of hardware computing power can greatly improve the computational efficiency of visual algorithms, enabling intelligent robots to achieve efficient environmental perception and real-time interaction with the dynamic environment. Furthermore, combining lightweight deep learning models with edge computing technology improves the performance of vision-based edge computing-driven follower robot systems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a robot summoning and following system and method based on visual edge computing. The technical solution adopted by this invention is as follows:

[0007] This system provides a vision-based, gesture-interactive pedestrian summoning and following robot intelligent system, comprising a Rockchip RK3588S edge computing platform, a USB camera, an STM32 microcontroller, motor and drive modules, and the robot itself. The Rockchip RK3588S edge computing device (Luban Cat 4) serves as the host computer of the intelligent system, responsible for tasks such as video pedestrian target detection, target tracking, following decision-making, and data information transmission. The STM32 microcontroller serves as the slave computer, responsible for real-time motion control and obstacle avoidance of the robot based on the data sent from the host computer.

[0008] The hardware part uses an STM32 microcontroller to control the motor speed and direction to realize the motion control of the main body of the car, and establishes data communication with the Luban Cat 4 platform through USART serial communication.

[0009] The software adopts a layered architecture, including a host computer vision processing module and a lower computer precision motion control module. The two parts achieve data interaction through a standardized serial communication protocol, and a multi-threaded scheduling mechanism is introduced to optimize the utilization of edge computing resources.

[0010] The host computer vision processing module includes an image acquisition and preprocessing unit, a lightweight pedestrian key point detection unit, a posture interaction and following decision unit, a multi-feature fusion target tracking and re-identification unit, and a multi-thread scheduling and data synchronization unit.

[0011] The image acquisition and preprocessing unit acquires images at a resolution of 1280×720 and a frame rate of 30fps in real time via a USB camera. It performs camera calibration, completes distortion correction based on the intrinsic parameter matrix to ensure geometric measurement accuracy, and improves the contrast of the image under complex lighting conditions through grayscale equalization operations, providing high-quality data support for subsequent pose detection and re-identification.

[0012] The lightweight pedestrian keypoint detection unit is based on the YOLOv8-Pose model and leverages the INT8 integer computation advantages of the edge computing platform's NPU to deploy a lightweight pedestrian pose recognition model. It utilizes a shared convolution and adaptive feature scaling to reduce the YOLOv8-Pose model detection head, employing a deeply compressed three-layer convolutional structure to maintain the receptive field while reducing the number of parameters. Group normalization (GN) replaces batch normalization (BN) to improve stability and robustness under small-batch data. A weight sharing and feature decoupling mechanism is established to separate bounding box regression from classification tasks, introducing distributed focus loss to improve the prediction accuracy of blurred boundary targets, thus balancing the trade-off between real-time performance and detection accuracy in edge computing scenarios. This unit outputs pedestrian target bounding boxes, confidence scores, and coordinate information for 17 keypoints.

[0013] The posture interaction following decision unit defines semantic information based on pedestrian posture detection results, which serves as the robot's command and control decision, enabling autonomous selection and switching of the followed target. Based on the human keypoints output by the YOLOv8-Pose model, it calculates the vector angle formed by the shoulder, elbow, and ear keypoints to determine the posture command. When a pedestrian raises their right arm, causing the vector angle to be less than the posture angle threshold Th_Agl for three consecutive frames or more, it is determined to be a robot summoning semantic, activating the robot to execute the following action, adjusting the tracking box corresponding to the detection box to the active state, and marking the corresponding tracking ID as the followed target. When the left arm raises, causing the vector angle to be less than the posture angle threshold Th_Agl for three consecutive frames or more, it is determined to be a termination semantic, stopping the following action, and adjusting the tracking box corresponding to the pedestrian detection box to the stopped state. Target data information is generated synchronously during state changes, and the serial communication module sends the corresponding followed target position information. Simultaneously, a tracking cleanup mechanism removes invalid target states, providing a decision-making basis for precise following control.

[0014] A multi-feature fusion target tracking and re-identification unit, based on the ByteTrack algorithm, incorporates an appearance feature re-identification and retrieval mechanism on top of Kalman filter prediction of target motion trajectories. To analyze pedestrian feature information in videos, a rigid-body and non-rigid-body region segmentation strategy is proposed, taking into account the differences in motion characteristics of different body parts during pedestrian movement. Human posture key points are divided into two types of regions based on motion rigidity features, providing structured feature support for subsequent multi-modal feature fusion and tracking association, simplifying and effectively describing complex, multi-degree-of-freedom human motion. By extracting color and texture features from the target rigid-body region, a human rigid-body feature vector is constructed. The combined features, last position, and loss time of activated targets are stored in real time, forming a lost target feature library. When a new target is detected, feature matching is performed using weighted Euclidean distance. If the matching degree exceeds a preset threshold, re-identification is completed and associated with the original tracking ID. A timeout cleanup mechanism is also implemented to remove expired lost targets to optimize resource usage. Finally, the unique ID, real-time coordinates, motion state, and re-identification matching results of the tracked target are output.

[0015] The multi-threaded scheduling and data synchronization unit aims to address the issues of idle hardware resources and delayed task response in single-threaded architectures, thereby improving the utilization of edge computing resources. It employs a multi-threaded parallel design architecture, using parallel processing flows for video reading, model inference, and result processing. The computationally intensive tasks of the YOLOv8pose model are split into four parallel inference threads responsible for executing YOLOv8pose object detection and pose estimation. Each thread runs independently, responsible for retrieving tasks from the task queue, performing inference, and saving results, significantly increasing the number of frames processed per unit time. In the result processing thread, inference results are retrieved from the result queue of the thread pool in order of frame ID, executing logic such as object tracking, trajectory re-identification and retrieval, pose determination of trajectory status, and serial port command transmission. Separating complex post-processing logic from the inference thread avoids the impact of time-consuming inference in a single thread on the decision-making process, ensuring the real-time performance of tracking and control commands. A mutex lock mechanism is used to achieve synchronized access to shared resources, ensuring data consistency. This design effectively improves the CPU utilization of the Rockchip RK3588S platform, avoids system lag caused by a single task blocking under a single thread, and ensures efficient collaborative operation of all modules of the robot system's host computer.

[0016] The main process of implementing a robot summoning and following system includes the following steps:

[0017] Step 1: System initialization, complete hardware connection and verification, software startup, LSCD-YOLOv8pose model loading, multi-thread pool initialization (preset number of threads) and parameter preset, complete serial port initialization and parameter configuration, laying the foundation for following tasks.

[0018] Step 2: Multi-threaded image acquisition and parallel inference. Start the image reading thread to acquire image frames in real time and assign unique frame IDs. Submit the frames to the inference task queue. The inference thread pool calls the LSCD-YOLOv8pose model in parallel to perform inference calculations, complete pedestrian detection and key point extraction, and store the inference results in the result queue. The result processing thread retrieves the inference results in order of frame ID. Combine the ByteTrack algorithm and Kalman filter to realize the correlation between target frames and simultaneously complete the LiDAR data acquisition and preprocessing.

[0019] Step 3: Posture interaction decision determines the following target, extracts key points of the pedestrian's upper limbs and calculates the included angle, determines the calling or termination semantics through a multi-frame verification mechanism, determines the following target and generates control commands, and uses a hash table to manage the trajectory activation state to avoid repeated triggering.

[0020] Step 4: Initiate the multi-feature fusion re-identification process. Extract color and texture features from the activated rigid body region of the following target, fuse multiple feature vectors and save them in the feature library, and maintain the feature libraries for active and lost targets. Under the condition of severe occlusion causing tracking disconnection, extract and fuse features from the rigid body region of newly appearing detection boxes of pedestrians, perform similarity matching with the feature vectors in the lost target feature library, and retrieve the target.

[0021] Step 5: Multi-threaded collaborative serial communication and vehicle motion control. The result processing thread sends control commands containing trajectory ID, target distance, and detection box position to the lower-level machine via serial port according to a preset frequency limit. After receiving the upper-level machine's STM32 microcontroller, the lower-level machine integrates the attitude perception decision command and motion control algorithm to adjust the vehicle's steering and speed to achieve smooth following.

[0022] Step 6: Multi-threaded resource management. Edge computing resource utilization is optimized through multi-threaded scheduling and data synchronization units. A multi-threaded parallel architecture is adopted to decouple model inference and post-processing tasks, splitting computationally intensive tasks for parallel execution. A synchronization and mutual exclusion mechanism ensures multi-threaded data consistency, reducing resource idleness and task blocking, and improving overall system efficiency. The result processing thread is responsible for cleaning up expired trajectory data, feature library data, counter data, and serial port transmission records according to a preset cycle, releasing memory resources. Simultaneously, the thread exit status is monitored; after all image frames have been processed, all threads are terminated, completing system cleanup.

[0023] Compared with the prior art, the advantages of the present invention are as follows:

[0024] A comprehensive optimization system for the "perception-decision-execution" chain, adapted to edge computing scenarios, was constructed. Through hardware architecture collaboration, lightweight improvements to the detection head, upgraded interaction modes, and optimized resource scheduling, this system systematically addresses the technical pain points of existing following robots under edge computing resource constraints, such as insufficient real-time performance, poor tracking robustness, awkward human-computer interaction, and low resource utilization. This aims to solve the problems of high parameter count in general detection models, poor human-computer interaction, easy target loss, and wasted hardware resources in traditional single-threaded computing modes under edge computing architectures. Specifically, this is reflected in the following aspects:

[0025] We construct a lightweight pose detection architecture adapted to edge computing. Based on YOLOv8-Pose, we design a lightweight LSCD detection head and combine it with NPU INT8 integer operations and an optimization strategy of replacing batch normalization with group normalization to achieve efficient real-time inference on edge platforms. While reducing the number of parameters, the detection accuracy is only slightly reduced, effectively solving the technical pain point of insufficient real-time performance of existing general detection models when deployed on edge nodes.

[0026] Interaction mode upgrade. The design incorporates multi-frame verification of the semantics of the joint angle of the human upper limbs, and executes summon verification to trigger following, realizing natural human-computer interaction without wearable devices. This replaces the traditional UWB positioning or fixed path following method, enabling flexible summon and follow commands, and improving the interaction method of traditional AGVs that can only complete actions based on line following or laser guidance.

[0027] Robustness optimization of target tracking decisions. The ByteTrack tracking scheme, which uses multi-feature fusion based on human body regions to re-identify targets, effectively solves the drift problem of existing tracking technologies in scenarios with target occlusion and brief loss, thus improving the robustness of target tracking.

[0028] Resource scheduling is made more efficient. The multi-threaded scheduling strategy for edge computing is optimized, decoupling the inference thread from the post-processing thread. Four-way parallel inference improves frame processing efficiency, increasing processor utilization from 35% to 75% in a single thread. The overall system frame rate is stabilized at 30fps, addressing the low resource utilization issue of existing single-threaded architectures. Closed-loop collaboration in the execution phase is achieved by building a closed-loop control system between the upper and lower computers. Low-latency linkage between visual decision-making and motion control is realized through a standardized serial port protocol. Combined with dynamic PID parameter adaptation and three-loop PID fusion control, the robot can complete tasks with high quality.

[0029] In summary, this invention achieves end-to-end hardware collaboration and algorithm optimization across the "perception-decision-execution" chain, enabling localization of the "detection-tracking-decision" process on the RK3588S edge platform. This results in a single-frame processing time of less than 50ms, ensuring continuous tracking. It provides a natural and convenient interaction method for robots to follow pedestrians, supports target selection and dynamic parameter adaptation in multi-person scenarios, and can be extended to various mobile robot following scenarios. It is highly versatile and possesses significant comprehensive advantages. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0031] Figure 1 This is a schematic diagram of the hardware architecture of the intelligent system for the following vehicle according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the lightweight processing architecture of the YOLOv8-Pose detection head according to an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram illustrating the definition range of the "rigid body region" for a pedestrian in a video, according to an embodiment of the present invention.

[0034] Figure 4 This is a schematic diagram of the following vehicle re-identification system according to an embodiment of the present invention;

[0035] Figure 5 This is a flowchart illustrating the multi-threaded architecture of the following vehicle according to an embodiment of the present invention. Detailed Implementation

[0036] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, outlines the specific implementation steps of a robot summoning and following method proposed in this invention:

[0037] Step 1: Initialize the system by completing hardware connection and verification, software startup, deep learning model loading, multi-thread pool initialization (preset number of threads) and parameter preset, and serial port initialization and parameter configuration. Start the Rockchip RK3588S edge computing platform, STM32 microcontroller, and USB camera sequentially. During startup, initialize each hardware unit and verify link connectivity simultaneously. The camera continuously acquires test images of a certain length (Fra_test∈[8, 15]) to verify image data integrity. Load the YOLOv8-Pose model and configure the RK3588S NPU's INT8 operation mode.

[0038] Step 2: Multi-threaded image acquisition and parallel inference. An image reading thread is started to acquire image frames in real time and assign unique frame IDs, submitting them to the inference task queue. The inference thread pool calls the LSCD-YOLOv8pose model in parallel to perform inference calculations, completing pedestrian detection and keypoint extraction, and storing the inference results in the result queue. Leveraging the INT8 computational acceleration advantage of the RK3588S NPU, the USB camera feed is used to perform inference detection on each frame. After inference, the bounding boxes, confidence scores, and coordinates of 17 keypoints for all pedestrian targets in the image are output. The detection box is sent to the ByteTrack tracker to perform the target tracking process. First, the inference output results are filtered and optimized, and the Kalman filter is initialized. Then, based on the state information of the effective target in the previous frame, the position and size of the candidate tracking box of the target in the current frame are predicted. The intersection-union ratio (IOU) between the tracking box and the high-confidence detection box is calculated. A preset IOU matching threshold is used to determine whether to generate inter-frame target association. A unique tracking ID is assigned to the successfully matched trajectory box. The unmatched trajectory box and the low-score detection box enter a second association matching. Similarly, a new trajectory identifier is assigned to improve the anti-occlusion capability.

[0039] Step 3: Perform posture interaction decision-making to determine the following target. Analyze the results of human skeleton keypoint detection and target tracking to generate posture interaction following decisions and control commands to complete the initial target tracking association. For each effectively tracked pedestrian target, symmetrically extract and construct human posture analysis triplet data and right arm keypoint triplet data tuple. rt Tuple of key data points for (right ear 4, right shoulder 6, right elbow 8) and left arm lt (Left ear 3, left shoulder 5, left elbow 7); Calculate the angle between the vectors connecting the key points of the triplet, i.e., the angle θ of the right arm. rt For vectors (Right shoulder → Right ear) and vector The angle between the right shoulder and right elbow, and the angle θ between the left arm and elbow. lt For vectors (Left shoulder → Left ear) and vector The angle between the left shoulder and left elbow; initiate a multi-frame verification mechanism to determine human posture semantics and avoid false triggers, such as the angle θ of a pedestrian's right arm. rt If for more than N consecutive frames in the range [3, 8] the angle between the pedestrian and the vehicle is less than the attitude angle threshold Th_Agl, then the host computer will record the pedestrian tuple. rtIf a target is identified as a valid keypoint, an "Activate Follow" command is generated, marking the tracking ID as the current target. A control command packet containing a "START" flag, tracking ID, target center x-coordinate, and actual target distance is generated. If the left arm angle is less than Th_Agl for N consecutive frames, a "Stop Follow" command is generated, clearing the current target marker and generating a control command packet containing a "STOP" flag and tracking ID. For valid tracking IDs of non-target targets, only the tracking state is maintained, and no control commands are generated. For tracking targets without valid keypoints, attitude determination is temporarily suspended, and the decision state of the previous frame is maintained.

[0040] Step 4: Initiate the multi-feature fusion re-identification process to solve the tracking interruption problem caused by target occlusion or temporary loss in dynamic scenes. For the activated target, determine the torso rigid body feature region Reg_rigid based on key points, that is, the smallest bounding rectangle formed by the key points of the left shoulder, right shoulder, left hip, and right hip. The HSV color histogram and LBP texture histogram in the feature region Reg_rigid are weighted and connected to generate a fused feature vector. After normalization, it is stored in the activated target feature library. The latest bounding box coordinates, tracking ID and timestamp of the target are recorded simultaneously. For detected targets in the current frame that do not have a matching tracking box, including newly detected targets or targets that are suspected to have been lost and then reappeared, repeat the above feature extraction process to generate a feature vector to be matched. Calculate the cosine similarity between the vector and the weighted fused feature vector in the lost target feature library. If the similarity value is greater than a preset threshold, it is determined to be the same target. Associate the current detection box with the original tracking ID and update the location information of that ID, and remove it from the lost feature library. If the similarity is less than the threshold, it is determined to be a new target. Assign a new tracking state maintenance mechanism. Mark the tracking target that has not had a matching detection box for N consecutive frames as "lost state" and move it into the lost target feature library. Perform feature vector and state information cleanup on targets in the lost feature library that have exceeded a preset time limit to release the memory resources of the edge computing platform.

[0041] Step 5: After receiving instructions from the host computer, the STM32 microcontroller integrates attitude perception decision instructions with motion control algorithms to achieve stable following of the vehicle. First, it receives the host computer instruction packet via interrupt. After verification, it parses the identifier bit, tracking ID, target position, and distance parameters. Invalid instructions are discarded, and valid instructions are updated to the following parameter buffer in shared memory. Then, it executes steering control, calculating the lateral error between the target center's horizontal coordinate and the image center. This error is input to a steering smoothing PID controller with multi-frame averaging filtering, dead-zone processing, and integral limiting, outputting the steering control quantity Move_Z and limiting the maximum steering speed. Simultaneously, it executes distance control, calculating the error between the actual target distance and the preset following distance. This error is input to a distance PID controller optimized with low-pass filtering, dead-zone processing, and differential limiting, outputting the forward / backward control quantity Move_X and limiting the maximum forward / backward speed. Move_X (forward / backward) and Move_Z (steering) are connected to the speed loop PI controller and steering loop PD controller respectively to obtain the final PWM values ​​for the left and right wheels. After limiting, these values ​​are sent to the motor drive module to control the motors and complete the vehicle following task.

[0042] Specifically, an asynchronous serial communication is implemented using a USART peripheral. At the hardware level, the physical link between the host computer and the lower-level control board is established via the TX / RX pins. To balance transmission rate and stability, the communication parameters are strictly aligned, with the baud rate set to 115200bps. The data format uses 8 data bits, 1 stop bit, and no parity bit. To achieve ordered transmission of multi-dimensional control parameters, a structured serial data frame protocol is used, defined as START, Track: %d, Class: %[^,], Dist: %d, Box: %d, %d, %d, %d, END\n, where START is the frame start flag and END\n is the frame end flag. The fields sequentially include the target tracking identifier ID, target category, distance calculated by monocular ranging, coordinates of the detection box center point, and width and height of the detection box. A target loss stop frame is set to inform the lower-level computer that a specific ID target has been lost, triggering a safe stopping logic. At the lower-level receiving end, a circular buffer is used to temporarily store the serial port received data. Circular storage and retrieval of data are managed through the receive index and start index to avoid buffer overflow. In the interrupt service function, each received byte is stored in the buffer. When the end-of-frame marker is detected, the complete data frame from the start index to the receive index is extracted, each field is parsed according to the protocol format, and the parsing result is stored in the structure array.

[0043] Step 6: Multi-threaded resource management. Edge computing resource utilization is optimized through multi-threaded scheduling and data synchronization units. A multi-threaded parallel architecture is adopted to decouple model inference from post-processing tasks, splitting computationally intensive tasks for parallel execution. A synchronization and mutual exclusion mechanism ensures multi-threaded data consistency, reducing resource idleness and task blocking, and improving overall system efficiency. The result processing thread is responsible for cleaning up expired trajectory data, feature library data, counter data, and serial port transmission records according to a preset cycle, releasing memory resources. Simultaneously, the thread exit status is monitored; after all image frames have been processed, all threads are terminated, completing system cleanup.

[0044] Specifically, the video reading thread is responsible for reading image frames from the camera, generating unique frame IDs, and then submitting the frame data to the task queue of the thread pool for processing by the subsequent inference thread. Separating video reading from subsequent inference and processing avoids overall efficiency degradation due to blocking. In the YOLOv8pose inference thread, four parallel inference threads complete computationally intensive inference tasks, responsible for executing YOLOv8pose object detection and pose estimation. Each thread runs independently, responsible for retrieving tasks from the task queue, performing inference, and saving results, significantly increasing the number of frames processed per unit time. In the result processing thread, inference results are retrieved from the result queue of the thread pool in order of frame ID, and the logic for target tracking, trajectory re-identification and retrieval, pose determination of trajectory status, and serial port command transmission is executed. Separating complex post-processing logic from the inference thread avoids the impact of time-consuming inference in a single thread on the decision-making process, ensuring the real-time performance of tracking and control commands.

[0045] The interactive robot following method based on visual edge computing provided in this application embodiment, such as... Figure 1The diagram shows the hardware architecture of the following vehicle intelligent system. After completing the hardware link construction, the system is initialized and configured, and hardware startup and software parameter presets are completed. The trained YOLOv8-Pose weight file is loaded using the Ultralytics framework, and ONNX export is performed. Dynamic axes are disabled to adapt to static inference at the edge. After optimization, the consistency of the model's input and output dimensions is verified, ensuring that the output still contains the coordinates and confidence information of 17 human keypoints. The format conversion is completed using the Rockchip RKNN-Toolkit2v1.4.0 toolkit. First, the hardware parameters of the RK3588S platform are configured, the optimized ONNX model is loaded, 1NT8 quantization calibration is performed, the quantization error compensation strategy is enabled, and the weights of the keypoint prediction branch are fine-tuned to ensure the accuracy of the quantized model. The RKNN model is then generated, and the Rockchip RK3588S edge computing platform, STM32 microcontroller, and USB camera are started sequentially. During startup, the initialization of each hardware unit and the connectivity verification of the link are completed simultaneously. The generated RKNN model file was transferred to the RK3588S edge computing platform. The model was loaded based on the RKNN RuntimeSDK provided by Rockchip, the NPU hardware acceleration channel was initialized, the inference was configured to INT8 operation mode, and the output tensor was parsed after the inference was completed. The key point coordinates, confidence scores and detection box information were extracted. Ten test images were continuously acquired from the camera to verify the integrity of the image data. The lightweight YOLOv8-Pose model was loaded, the INT8 operation mode of the RK3588S NPU was configured, and the model was loaded to test single-frame images.

[0046] By optimizing the target detection model structure, a balance is achieved between parameter quantity, computational cost, and detection accuracy on edge computing platforms. The lightweight detection model design and implementation revolve around shared convolutions and adaptive feature scaling. In the feature extraction stage, the YOLOv8-Pose backbone network and feature pyramid structure are used as shared basic feature layers to ensure the expressive power of basic features. Based on this, streamlined task-specific branches are built. Unlike the complete deep specialized network used in traditional detection heads, a three-layer convolutional structure with deep compression is designed for keypoint detection tasks. The number of feature channels is dynamically adjusted through an intelligent channel compression strategy, precisely controlling the number of feature channels in the middle layer to one-quarter of the number of input channels. At the same time, dimensionality adaptation ensures that the number of output channels meets the prediction dimensionality requirements of 17 human keypoints. The convolutional kernel combination adopts an optimized paradigm, deploying 1×1 convolutions, 3×3 convolutions, and 1×1 convolutions in sequence: the first 1×1 convolution is responsible for rapid adjustment of channel dimensions and initial dimensionality reduction, reducing subsequent computational overhead. The second 3×3 convolution focuses on fine extraction of spatial features, maintaining an effective receptive field of 9×9 to cover key areas of the human body. The third 1×1 convolution completes the final feature mapping and output transformation. This combination reduces computational complexity while ensuring the integrity of feature extraction.

[0047] To improve the model's stability in small-batch inference scenarios on edge platforms, a normalization optimization was implemented, replacing the original batch normalization (BN) of the YOLOv8-Pose detection head with group normalization (GN). Specifically, the input feature channels are divided into 8 independent groups. Within each group, the mean and variance are calculated and normalized. This effectively mitigates the impact of batch size variations on model performance, making it particularly suitable for small-batch scenarios with single-frame inference on edge devices. It also reduces noise interference and improves model convergence speed and prediction accuracy. The specific calculation process of group normalization is as follows:

[0048]

[0049] Where, x i It is the output of the previous layer, S i It is the set of feature regions used to calculate the mean and variance, m is the size of the set, and ∈ is a very small constant.

[0050] To optimize bounding box prediction accuracy, a weight-sharing and feature decoupling mechanism is implemented, abandoning the traditional multi-branch independent convolution paradigm. Bounding box regression and classification prediction tasks are separated into two independent branches, achieving specialized optimization based on the inherent differences between the tasks: the bounding box regression branch focuses on learning the geometric features of the target location, achieving accurate prediction of continuous spatial coordinates. The classification branch concentrates on processing semantic discrimination information, achieving accurate discrimination of discrete class probabilities, effectively avoiding gradient conflicts and inconsistent optimization directions between tasks, and improving the performance of each task. Regarding the optimization of bounding box prediction accuracy, a distributed focus loss function is introduced to replace the traditional direct regression of bounding box coordinates. This transforms coordinate prediction into a probability distribution learning process. By setting a regression maximum parameter, the model learns coordinate values ​​according to a discrete probability distribution rather than a single numerical value, better handling the uncertainty of blurred boundaries or partially occluded targets. By focusing on difficult samples and distribution edge regions, the accuracy and robustness of bounding box prediction are improved.

[0051] This embodiment proposes an interactive following strategy based on interactive posture semantic understanding and defines a set of clearly distinguishable "activation" and "deactivation" postures. Human "posture information" is used as a high-level command to determine the activation and termination states of the trajectory, integrating it into the trajectory state decision-making closed loop to complete the transmission and termination of serial port data, thereby realizing a paradigm shift from "passive following" to "active specification." A hash table data structure is built for trajectory activation state management. The hash table uses the trajectory ID (track_id) output by ByteTrack as the unique key and the trajectory activation state (Boolean value) as the mapping value. Leveraging the fast lookup characteristics of hash functions, it ensures the efficiency of state query and update in multi-target scenarios, providing data structure support for the accurate association between posture commands and trajectories.

[0052] For each frame of YOLOv8-Pose outputting keypoint sets of targets, the detection box parameters (x, y, w, h) of the corresponding target are first extracted. The coordinates of the center point of the detection box are calculated using the formula: x1 = x + w / 2 and y1 = y + h / 2. Simultaneously, all active trajectory lists output by ByteTrack are traversed, and the trajectory box parameters (x′, y′, w′, h′) corresponding to each trajectory are extracted. Similarly, the coordinates of the center point of the trajectory box are calculated: x2 = x′ + w′ / 2 and y2 = y′ + h′ / 2. The straight-line distance dst between the two center points is calculated using the Euclidean distance formula. Depending on the video frame resolution, a preset distance threshold TH_pxl is used, in pixels, which can be TH_plx ∈ [25, 35]. If dst < TH_plx, the keypoint set is determined to belong to the same target as the current trajectory. The keypoint set is then bound to the trajectory ID, achieving precise association between skeletal keypoints and target trajectories, laying the foundation for subsequent pose commands to trigger the corresponding trajectory state.

[0053] Employing a hash-based trajectory activation state management scheme, after ByteTrack first tracks a target and generates a new trajectory ID, the system, while traversing the list of active trajectories, first checks if the trajectory ID exists in the hash table. If not, it automatically initializes its state to false (inactive), completing the initial registration of the trajectory state. When a pedestrian raises their right arm, the system extracts the associated key points of the right arm (ear, shoulder, elbow) and calculates the joint angle using the vector dot product formula.

[0054] Where o = {5, 6}, p = {3, 4}, q = {7, 8}

[0055] The preset attitude angle threshold Th_Agl is a fixed acute angle, such as Th_Agl = 60°. If θ < Th_Agl degrees is detected for N consecutive frames ∈ [3, 8] and the hash table state corresponding to the current trajectory ID is false, the system interprets it as an "activate follow" instruction and immediately updates the state corresponding to the trajectory ID in the hash table to true, that is, it has been activated and the subsequent linkage process is triggered synchronously.

[0056] After the trajectory state is activated, the system calculates the actual distance to the target based on the bound detection box parameters, using a monocular ranging algorithm and relying on the camera intrinsic parameter matrix and the principle of similar triangles.

[0057]

[0058] The distance formula obtained after transformation is:

[0059]

[0060] The actual height of the target object is H. actual Its height in the image is h. image The camera focal length, f, is obtained after camera calibration, and the unit is pixels.

[0061] The calculated data will generate structured data containing trajectory ID, target distance, and the center position (x, y) of the detection box. This data is first marked as PENDING, and after verifying its integrity, it is marked as SENT. The data is then sent to the lower-level motion control layer via the USART serial port, triggering the motor drive to start following. Stopping the following operation is achieved by raising the left arm posture command. The system monitors in real time the vector angle formed by the key points of the left arm (ear, shoulder, elbow) corresponding to each trajectory. When the angle is less than Th_Agl, a counter is started to accumulate. Only when N consecutive frames meet the condition and the trajectory ID is true in the hash table is the stop logic executed: the key-value pair entry corresponding to the trajectory ID is deleted from the hash table, and a "stop following" command is sent to the upper-level computer. Upon receiving this command, the lower-level computer controls the motor PWM signal to decay to 0 frame by frame, achieving a smooth stop for the vehicle.

[0062] Analyze the stable feature information of the tracked pedestrian in the video. Figure 3 This is a schematic diagram of the rigid body region definition range for a pedestrian in an embodiment of the present invention. Based on the 17 human body key points output by YOLOv8-Pose, four points—left shoulder (5), right shoulder (6), left hip (11), and right hip (12)—are selected as the positioning references for the rigid body region. Since the area enclosed by these four points (the torso) only undergoes a small translation or rotation during the pedestrian's walking process, the internal structure is relatively fixed and the deformation amplitude is extremely small, which can preserve the target's appearance features to the greatest extent and avoid background interference. Specifically, the pixel coordinates (x5, y5), (x6, y6), (x11, y11), and (x12, y12) of the above four points are extracted from the pose detection results of each frame. By calculating the minimum x_min = min(x5, x6, x11, x12), the maximum x_max = max(x5, x6, x11, x12), the minimum y_min = min(y5, y6, y11, y12), and the maximum y_max = max(y5, y6, y11, y12) of the four points, the minimum bounding rectangle formed by these four points is determined as the rigid body region (ROI). The width of the minimum bounding rectangle is x_max - x_min, and the height is y_max - y_min. This region concentrates the main color and texture information of the pedestrian's clothing, laying the foundation for subsequent stable feature extraction.

[0063] To address interference from lighting variations and texture differences in the following scene, a fusion scheme of HSV color features and LBP texture features is adopted to characterize the features of the target pedestrian region. First, the extracted rigid region of interest (ROI) is preprocessed by converting the RGB image to the HSV color space. Discretization and binning are performed on the chroma and saturation channels. Considering the balance between computational efficiency and representational capability, 30 bins are set for the chroma channel and 32 bins for the saturation channel. The V channel, being significantly affected by lighting, is not included in feature construction, generating a 960-dimensional HSV color histogram. Simultaneously, the ROI region is grayscaled, and a 3×3 neighborhood LBP operator is used to encode the pixel neighborhood grayscale relationship, capturing the texture details of the clothing surface, generating a 256-dimensional LBP texture histogram. To eliminate the dimensional differences between the two features, the HSV and LBP histograms are normalized and mapped to the [0, 1] interval. Then, vector concatenation is used to form an initial fused feature vector with a length of 1216 dimensions. To improve feature robustness, the fused feature vectors are L2 normalized to obtain standardized appearance features for subsequent matching.

[0064] Building upon the ByteTrack model's existing advantage of two IOU motion associations (matching high-scoring detection boxes with active trajectories and low-scoring detection boxes with unmatched trajectories), appearance feature compensation is added, forming a "motion-first, appearance-filled" association logic. When a pedestrian following a trajectory is activated, the standardized fusion feature vector of its rigid body region is extracted in real time, and the trajectory ID, current position coordinates, and timestamp are recorded simultaneously and stored in the lost target feature library. When the trajectory is disconnected due to occlusion or other reasons (failure of motion association between the detection box and the trajectory), the trajectory is marked as "lost state," and its feature vector and last position information are continuously retained in the feature library. When the occlusion is removed and a new detection box appears or a new target appears in the image, the above rigid body region extraction and feature fusion process is performed on the unmatched detection box to obtain the feature vector to be matched, and its similarity with all historical feature vectors in the feature library is calculated.

[0065] The similarity calculation uses the Pearson correlation coefficient method. The similarity calculation method for both color and texture feature vectors uses the Pearson correlation coefficient method. Let the vectors of the two histograms be C = [c1, c2, ..., c...]. m ,] and D=[d1,d2,...,d m If , then the correlation coefficient is:

[0066]

[0067] In the formula, Let C be the mean of the histogram. The mean of histogram D is given.

[0068] The weighted fusion of color and texture features is as follows:

[0069] combined_sim=WT1*color_sim+WT2*lbp_sim

[0070] WT1 and WT2 are weight parameters, color_sim is the HSV color feature similarity, and lbp_sim is the LBP texture feature similarity. WT1 = 0.4 and WT2 = 0.6 are the optimal weight parameters determined through experimental debugging and optimization after multi-scene testing. The preset similarity threshold Th_Sml ranges from [0.5, 0.8]. When combined_sim ≥ Th_Sml, it is determined to be the same target, the active state of the original trajectory is restored, and its position and feature information are updated. If the similarity is lower than the threshold and the new detection box has a high score, the target is initialized as a new trajectory. To optimize resource consumption, the feature library adopts a timeout cleanup mechanism, removing trajectory features and information that have not been successfully matched for more than a continuous duration Duratioin ∈ [8, 12] seconds.

[0071] The multi-threaded architecture uses a scalable thread pool as its core scheduling hub. The video reading thread serves as the data input source. After starting, the thread first initializes the USB camera, configuring acquisition parameters of 1280×720 resolution and 30fps frame rate, and then loops through the acquisition process. To ensure data independence across multiple threads, the frame ID and frame data copy are encapsulated into a structured task and pushed to the task queue. After pushing, a condition variable wakes up the waiting inference thread. If the task queue is full, it blocks and waits until there is free space, ensuring a dynamic balance between data input and caching capacity. The four inference threads in the thread pool are the core of parallel computing, focusing on the inference execution of the YOLOv8-Pose model. Each thread independently completes model loading, task acquisition, inference calculation, and result encapsulation. After inference, the output pedestrian target bounding box, confidence score, coordinates of 17 human keypoints (including visibility confidence), and frame ID are encapsulated into a structured inference result. The confidence score filtering threshold can be set to 0.5. The structured inference result is then pushed to the result queue. After pushing, it enters a blocking and waiting state again until the next task is triggered. The thread pool has a built-in load balancing mechanism. The four inference threads obtain tasks in sequence to avoid long-term overload of any one thread. At the same time, an inference timeout threshold is set. Inference tasks that have not been completed within the timeout period are automatically marked as invalid and the corresponding thread is restarted to ensure the overall availability of the thread pool.

[0072] The result processing thread, as the core of post-processing and output, independently undertakes target tracking, re-identification, attitude decision-making, serial communication, and resource cleanup. It executes in parallel with the inference thread throughout the process without resource contention. After starting, the thread initializes components such as the ByteTrack tracker, re-identification feature library, and attitude decision counter, and then enters a loop processing flow. By obtaining the currently pending frame ID, it calls the thread pool's interface to accurately extract the inference result of the corresponding frame from the result queue. If the inference result corresponding to the frame ID is not ready, it blocks and waits using a condition variable until the inference thread pushes the result and wakes it up. After obtaining the result, post-processing is performed in the following sequence: "target tracking → re-identification retrieval → attitude angle calculation → following state determination → serial command encapsulation → resource cleanup." First, the ByteTrack algorithm is used to complete the inter-frame target association and assign a unique tracking ID. For unassociated potential lost targets, the HSV color histogram and LBP texture features of their rigid body region (the area enclosed by four points: left shoulder, right shoulder, left hip, and right hip) are extracted and matched with the lost target feature library for similarity. If the matching degree exceeds the set similarity threshold, re-identification is completed and the original tracking ID is associated. Then, the angle between key points of the upper limbs is calculated, and the "activate following" or "stop following" instruction is determined by a continuous N_frame frame verification mechanism. For activated following targets, control instructions containing tracking ID, target distance, and bounding box coordinates are encapsulated according to a standardized protocol and sent to the lower-level machine via serial port at fixed intervals of Interval = 200ms to avoid serial port buffer overflow caused by high-frequency transmission. Resource cleanup is performed every Num = 100 frames, deleting expired tracking IDs and lost target features that have not been retrieved within Duratioin ∈ [8, 12] and releasing memory resources.

[0073] To ensure the safety and efficiency of multi-threaded collaboration, the architecture employs multiple fine-grained synchronization mechanisms: Operations on both the task queue and result queue are protected by independent mutex locks, allowing only one thread to operate on the queue at a time, preventing data corruption or race conditions. Condition variables are bound to the queue state; the inference thread is only awakened when the task queue is not empty, and the result processing thread is only awakened when the result queue contains the current frame ID to be processed, reducing CPU idle time. Atomic variables ensure the temporal consistency of frame data; the frame IDs submitted by the reading thread are strictly incremented, and the processing thread processes them in order of frame ID, ensuring frame alignment for "acquisition-inference-processing" even if there are differences in the processing speed of the inference threads. Shared resources (such as the tracking target status mapping table and the lost target feature library) are read and written only by the result processing thread, fundamentally avoiding the risk of multi-threaded data races. Serial communication adopts a single-threaded exclusive mode; the result processing thread temporarily stores instructions in a buffer and sends them at fixed intervals, avoiding data loss or instruction corruption caused by concurrent writing to the serial port by multiple threads.

[0074] To achieve smooth and stable following of the target vehicle, a linear and angular velocity control strategy based on the PID control algorithm was designed. This study employs a PID control algorithm to construct linear and angular velocity controllers, with refined design of the proportional, integral, and derivative components to achieve smooth motion control. By calculating the deviation between the distance between the vehicle and the target and the desired distance, as well as the deviation between the target's horizontal coordinate and the horizontal coordinate of the center point of the image, the proportional term provides a rapid response, enabling the vehicle to quickly adjust its posture to center the target in the image. The integral term limits the steering from exceeding the desired value, while the derivative term reduces parameter tuning, implementing anti-saturation constraints and dynamic smoothing optimization. Combined with a safe distance constraint, this ensures the target remains centered in the image of the target vehicle, maintaining a safe distance between the target and the vehicle. The vehicle's speed and angular velocity are adjusted in real-time based on the calculated deviations, supporting smooth and stable following of the target by the two-wheeled vehicle.

[0075] Let the image pixel width be W, and the horizontal coordinate of the image's center point be W / 2. The horizontal coordinate of the target detection bounding box sent by the edge computing device is x_target. Here, e_x(t) > 0 indicates the target is to the right of the center of the field of view, and e_x(t) < 0 indicates the target is to the left of the center of the field of view. The controller's goal is to make e_x(t) approach 0 by steering. At time t, the longitudinal distance controller's goal is to maintain a preset distance D between the vehicle and the target. Let the current target distance sent by the host computer be d_target(t).

[0076]

[0077] e d (t)=d target (t)-D

[0078] In visual tracking, the lateral and distance deviations of the target center are easily affected by image noise and detection algorithm jitter, resulting in high-frequency components in the error signal. Furthermore, the control output of the PID controller may also experience a step change due to sudden error changes, causing drastic fluctuations in the motor drive signal and affecting motion stability. To achieve a smoother tracking control effect, this system incorporates a moving average filter and a first-order low-pass filter into the PID controller to reduce noise in the input deviation and smooth the output control signal, respectively.

[0079] Moving average filtering is based on the concept of a time window, retaining the error data of the most recent N frames and calculating the arithmetic mean within the window as the current effective deviation, thus smoothing high-frequency noise. The input x-axis error e... x (k) Perform moving average filtering to reduce noise interference and obtain the filtered error e. x，f (k). Similarly, for the input distance deviation e d(k) Perform moving average filtering to reduce noise interference and obtain the filtered error e. d，f (k).

[0080]

[0081] In the formula: N and M are the filter window sizes, and k is the sampling time. Subsequently, the filtered error is input to the PID controller. Its incremental PID control output u ω (k) and u v (k) are respectively:

[0082]

[0083]

[0084] Where: K P1 and K d1 These are the proportional and derivative coefficients of the angular velocity control PID, K P2 and K d2 These are the proportional and derivative coefficients of the angular velocity control PID, K i This refers to the integral term coefficient of the angular velocity control PID. Finally, to make the final steering command smoother and prevent mechanical shock, the controller output u is adjusted. ω (k) Perform a first-order low-pass filter to obtain the final forward control and steering control values, respectively:

[0085] Move_Z(k)=(1-α)*Move_Z(k-1)+αu ω (k)

[0086] Move_X(k)=(1-β)*Move_X(k-1)+βu v (k)

[0087] In the formula: α and β are filter coefficients, and the control quantity Move_Z will be used as the target angular velocity of the bottom-level steering loop. Move_X will be used as the target linear velocity of the bottom-level velocity loop.

[0088] The lidar data acquisition uses an 8-bit data bit and a 1-bit stop bit serial port configuration. The lidar outputs environmental information in fixed-format frames, with each frame containing a frame header, length identifier, rotation speed, angle range, distances to 16 points, and a CRC checksum. Data is stitched together byte-by-byte by a state machine matching the frame structure identifier. Accumulated checksums and the CRC value at the frame tail are used to filter valid data and avoid transmission errors. For preprocessing, a "two-point mean fusion" strategy is used to address instantaneous noise. The 16 original scan points in each frame are grouped pairwise and averaged to obtain 8 fusion point distance data. The fusion point angles are then obtained by evenly distributing the distances between the fusion points and the frame start / end angles. A 225-point data buffer is constructed, and the fused data is stored sequentially. Once full, it is synchronized to the global data array to form a stable data source. The 225 fusion points in the global data array are traversed to eliminate invalid ranging values ​​and distant interference points. Isolated environmental noise is filtered by statistically counting the number of potential target points (the threshold is experimentally adjusted). The core target is locked using the "minimum distance criterion." The minimum distance is initialized to the maximum value of a 16-bit unsigned integer, and the minimum distance and corresponding angle are updated in real time during traversal. To simplify direction determination, the original angles from 0 to 360° are mapped to polar coordinates from -180° to 180° (positive angles correspond to the right side of the vehicle, negative angles to the left side, and 0° to the front). When the angle is greater than 0, the vehicle turns right; when it is less than 0, it turns left. An angle de-jitter mechanism is designed, storing historical valid angles in static variables and recording the number of consecutive fluctuations to ensure tracking stability.

[0089] The vision and LiDAR collaborative switching strategy uses the LiDAR distance threshold as the core trigger condition to achieve seamless transition between the two modes. When the target is within the effective visual ranging range, vision tracking is prioritized using YOLOv8-Pose and ByteTrack. When the tracking distance is too close, causing the monocular camera to fail to capture complete body key points, the system triggers a switch to LiDAR tracking mode. During the switching process, data consistency verification is performed: using the vehicle's geometric center as the origin of a unified coordinate system, the 2D angle and distance information of the pedestrian detection box output by the vision system are mapped to the 3D scanning angle of the LiDAR point cloud cluster center. Real-time matching and verification ensure that the target's spatial orientation is consistent before and after the switch. During the transition phase, parallel monitoring of dual-sensor data is maintained, tracking errors are fed back in real time, and the transition coefficient is dynamically adjusted to reduce distance and angle fluctuations and achieve a smooth switch.

Claims

1. A robot summoning and following system, characterized in that, The system comprises hardware and software components. The hardware includes a Rockchip RK3588S edge computing platform, a USB camera, an STM32 microcontroller, a motor and drive module, and the robot car itself. The Rockchip RK3588S edge computing platform acts as the host computer, responsible for video pedestrian target detection, target tracking, and issuing decision commands. The STM32 microcontroller acts as the slave computer, responsible for real-time control of the robot's movement, obstacle avoidance, and low-level motion planning. The USB camera is used for real-time image acquisition, and the motor and drive module receives commands from the slave computer to drive the robot car's movement. The host computer and slave computer establish data interaction through USART serial communication. The software adopts a layered architecture, including an upper-computer vision processing module and a lower-computer precision motion control module. The upper-computer vision processing module includes an image acquisition and preprocessing unit, a lightweight pedestrian key point detection unit, a posture interaction following decision unit, a multi-feature fusion target tracking and re-identification unit, and a multi-thread scheduling and data synchronization unit. The lower-computer precision motion control module uses a PID control algorithm to achieve smooth following and speed and steering adjustment of the vehicle.

2. The robot summoning and following system according to claim 1, characterized in that, The lightweight pedestrian keypoint detection unit employs the LSCD-YOLOv8pose lightweight pose detection model. This LSCD-YOLOv8pose model relies on the NPU of the host computer's edge computing platform and uses the INT8 operation mode to accelerate inference, ensuring that the model's inference time meets the real-time following requirements of the vehicle. The model uses a lightweight LSCD detection head designed with shared convolution and adaptive feature scaling, reducing the number of model parameters and computational cost while maintaining detection accuracy, thus adapting to the deployment and inference needs of the vehicle's edge computing device. Specific design improvements are as follows: The LSCD lightweight detection head constructs streamlined task-specific branches on top of a shared basic feature layer. The keypoint branch adopts a deep-compressed three-layer convolutional structure. Through an intelligent channel compression strategy, the number of intermediate feature channels is dynamically adjusted to one-quarter of the number of input channels, while meeting the model output dimension requirements. The three-layer convolutional structure uses an optimized combination of convolutional kernels, sequentially using 1×1 convolution to adjust the channel dimension, 3×3 convolution to extract spatial features, and 1×1 convolution to map the output. This reduces the number of model parameters and computational complexity while ensuring the integrity of feature extraction and an effective receptive field. The LSCD lightweight detection head uses group normalization (GN) instead of the original batch normalization (BN) of YOLOv8-pose. It divides the input feature channels into multiple independent groups, calculates the mean and variance within each group, and performs normalization. The calculation formula is as follows: where x i is the output feature of the last layer, S i is the set of feature regions for computing mean and variance, m is the size of the set, and ε is a small constant for avoiding zero denominator; this design effectively reduces the impact of batch size changes on model performance, adapts to single-frame and small-batch inference scenarios of edge devices, reduces noise interference, and improves the convergence speed, prediction accuracy, and generalization robustness of the model. The LSCD lightweight detection head establishes a weight-sharing and feature decoupling mechanism, abandoning the traditional multi-branch independent convolution paradigm. It separates bounding box regression and classification prediction tasks into two independent branches, optimizing based on the essential differences between the two tasks. The bounding box regression branch focuses on learning the geometric features of the target location, achieving accurate prediction of continuous spatial coordinates; the classification branch concentrates on processing semantic discrimination information, achieving accurate discrimination of discrete class probabilities. A distributed focus loss function is introduced in the bounding box prediction stage, replacing the traditional direct regression of bounding box coordinates. This transforms coordinate prediction into a probability distribution learning process, and by setting a regression maximum parameter, the model learns coordinate values. Discrete probability distributions, rather than single numerical values, can better handle uncertainties such as blurred boundaries and partially occluded targets. By focusing on difficult samples and distribution edge regions, the accuracy and robustness of bounding box prediction are further improved. Each detection layer is configured with an independent and learnable scale factor, replacing the traditional fixed scale mechanism. Each feature pyramid level adaptively adjusts the feature response intensity of that level through dedicated scaling parameters. The predefined anchor point scheme is abandoned, and a dynamic anchor point generation strategy is adopted. The optimal anchor point distribution is calculated in real time according to the actual input image size, solving the matching problem between fixed anchor points and variable input sizes, and avoiding the defects of insufficient anchor point coverage or excessive redundancy.

3. The robot summoning and following system according to claim 1, characterized in that, The posture interaction following decision unit triggers the following command based on the angle between key points of the upper limbs, without the need for external devices. It also solves the problem that the key point set output by the YOLOv8-Pose model and the trajectory set output by the ByteTrack model are independent, ensuring that the action trigger corresponds to the correct target trajectory. The specific mechanism is as follows: An interactive following strategy based on posture semantic understanding is proposed, and a set of "activation" and "deactivation" postures with clear distinctions is defined. Human "posture information" is used as a high-level command to decide the activation and termination states of the trajectory, which is integrated into the trajectory state decision-making closed loop to complete the transmission and termination of serial port data, thereby realizing the paradigm shift from "passive following" to "active specification". A hash table data structure is built for trajectory activation state management. The hash table uses the trajectory ID (track_id) output by ByteTrack as the unique key and the boolean value of the trajectory activation state as the mapping value. Relying on the fast lookup characteristics of the hash function, the efficiency of state query and update in multi-target scenarios is ensured, and data structure support is provided for the accurate association between posture commands and trajectories. For each set of target key points output by YOLOv8-Pose for each frame, first extract the detection box parameters (x, y, w, h) of the corresponding target, and calculate the center coordinates of the detection box, that is, the abscissa of the center point of the detection box x1 = x + w / 2, and the ordinate y1 = y + h / 2; synchronously traverse all the active track lists output by ByteTrack, extract the track box parameters (x′, y′, w′, h′) corresponding to each track, and similarly calculate the center coordinates of the track box x2 = x′ + w′ / 2, y2 = y′ + h′ / 2; according to the Euclidean distance formula, calculate the straight-line distance dst between the above two center points, and preset the distance threshold TH_pxl according to the different video frame resolutions. If dst < TH_plx, it is determined that the set of key points and the current track belong to the same target, and the set of key points is bound to the track ID to achieve the precise association of the skeletal key points and the target track; Adopt a hash-based track activation status management scheme. After ByteTrack first tracks a target and generates a new track ID, when the system traverses the active track list, it will first query whether the track ID exists in the hash table. If it does not exist, its status will be automatically initialized to false, that is, not activated, to complete the initial registration of the track status; when it is detected that a pedestrian raises the right arm, extract the key points associated with the right arm (ear, shoulder, elbow), and calculate the joint angle θ through the vector dot product formula Where o = {5, 6}, p = {3, 4}, q = {7, 8} Preset the pose angle threshold Th_Agl, which takes a fixed acute angle; if it is detected that θ < Th_Agl degrees for N_frame consecutive frames and the status of the hash table corresponding to the current track ID is false, the system interprets it as an "activation follow" instruction to summon the robot, and immediately updates the status corresponding to the track ID in the hash table to true, that is, activated, and synchronously triggers the subsequent linkage process; correspondingly, the stop follow operation is achieved through the pose instruction of raising the left arm.

4. The robot summoning and following system according to claim 1, characterized in that... The multi-feature fusion target tracking and re-identification unit is based on the ByteTrack algorithm framework, combines Kalman filtering to predict the target motion trajectory, and introduces an appearance feature re-identification retrieval mechanism; divides the human body pose key points into rigid body regions and non-rigid body regions according to the motion rigidity feature. The rigid body region is the minimum circumscribed rectangle formed by the key points of the left shoulder, right shoulder, left hip, and right hip; extracts the HSV color histogram and LBP texture histogram of the rigid body region, splices them after normalization to form a fusion feature vector, constructs a lost target feature library, and realizes the re-identification of the lost target through similarity matching; Specifically, based on the 17 human body key points output by YOLOv8-Pose, four points—left shoulder (5), right shoulder (6), left hip (11), and right hip (12)—were selected as the rigid body region localization references, because the area enclosed by these four points can preserve the target's appearance features to the greatest extent and avoid background interference. The specific method for rigid body region localization is to extract the pixel coordinates (x5, y5), (x6, y6), (x11, y11), and (x12, y12) of the above four points from the pose detection results of each frame, and calculate the minimum abscissa x_min = min of the four points. Given the coordinates (x5, x6, x11, x12), the maximum x-coordinate x_max = max(x5, x6, x11, x12), the minimum y-coordinate y_min = min(y5, y6, y11, y12), and the maximum y-coordinate y_max = max(y5, y6, y11, y12), the minimum bounding rectangle formed by these four points is determined as the rigid body region (ROI). The width of the minimum bounding rectangle is x_max - x_min, and the height is y_max - y_min. This region is beneficial for analyzing stable feature information of the target pedestrian in the video. A scheme combining HSV color features and LBP texture features is adopted to characterize the features of the target pedestrian region. First, the extracted rigid region (ROI) is preprocessed by converting the RGB image to the HSV color space and performing discretization binning statistics on the chroma and saturation channels to generate an HSV color histogram. At the same time, the ROI region is grayscaled, and the LBP operator is used to capture the details of the clothing surface texture to generate an LBP texture histogram. To eliminate the difference in the dimensions of the two features, the HSV histogram and LBP histogram are normalized and mapped to the [0, 1] interval. Then, the initial fused feature vector is formed by vector concatenation, and the fused feature vector is then L2 normalized to obtain the standardized appearance features for subsequent matching. Furthermore, in addition to retaining the original two IOU motion associations of the ByteTrack model, namely, matching high-scoring detection boxes with active trajectories and matching low-scoring detection boxes with unmatched trajectories, a third appearance feature matching is added to form a motion-first and appearance-filled association logic; when the pedestrian following trajectory state is activated, the standardized fusion feature vector of its rigid body region is extracted in real time, and the trajectory ID, current position coordinates and timestamp are recorded simultaneously and stored in the lost target feature library; When the trajectory is disconnected due to occlusion or other reasons, that is, when the detection box fails to associate with the trajectory motion, the trajectory is marked as "lost state" and its feature vector and last position information are continuously retained in the feature library; when the occlusion is removed and a new detection box appears or a new target appears in the screen, the above rigid body region extraction and feature fusion process is performed on the unmatched detection box to obtain the feature vector to be matched, and the similarity between it and all historical feature vectors in the feature library is calculated. The similarity calculation uses the Pearson correlation coefficient method. The similarity calculation method for both color and texture feature vectors uses the Pearson correlation coefficient method. Let the vectors of the two histograms be C = [c1, c2, ..., c...]. m ,] and D=[d1,d2,...,d m If , then the correlation coefficient is: In the formula, Let C be the mean of the histogram. The mean of histogram D; The weighted fusion of color and texture features is as follows: combined_sim=WT1*color_sim+WT2*lbp_sim WT1 and WT2 are weight parameters, color_sim is the HSV color feature vector, and lbp_sim is the texture feature vector. A preset similarity threshold Th_Sml is used. When combined_sim≥Th_Sml, the target is determined to be the same, the active state of the original trajectory is restored, and its position and feature information are updated. If the similarity is below the threshold and the new detection box has a high score, the target is initialized as a new trajectory. To optimize resource consumption, the feature library adopts a timeout cleanup mechanism to remove trajectory features and information that have not been matched successfully for more than a continuous duration.

5. The robot summoning and following system according to claim 1, characterized in that, The target tracking and re-identification unit relies on the system's multi-threaded architecture to achieve efficient and secure target tracking and re-identification functions. It works in deep collaboration with the multi-threaded scheduling and data synchronization unit, using a scalable thread pool as the core scheduling hub. Combining the ByteTrack algorithm, Kalman filtering, and appearance feature re-identification retrieval mechanism, it adapts to the computing power of edge computing platforms. Its multi-threaded collaborative implementation is as follows: All execution logic of the target tracking and re-identification unit is integrated into the result processing thread of the multi-threaded scheduling and data synchronization unit. This result processing thread, as the post-processing and output core, independently undertakes the responsibilities of target tracking, re-identification, attitude decision, serial communication, and resource cleanup. It executes in parallel with the inference thread in the thread pool throughout the process without resource contention. After the thread starts, it first initializes components such as the ByteTrack tracker, re-identification feature library, and attitude decision counter, and then enters the loop processing flow. The result processing thread obtains the frame ID to be processed and calls the thread pool interface to accurately extract the inference result of the corresponding frame from the result queue. This inference result is generated in parallel by 12 inference threads in the thread pool. The 12 inference threads are the core of parallel computing, focusing on the inference execution of the YOLOv8-Pose model. The thread pool has a built-in load balancing mechanism, and the inference threads obtain tasks in sequence to avoid long-term overload of a single thread. At the same time, an inference timeout threshold is set, and inference tasks that have not been completed within the timeout period are automatically marked as invalid and the corresponding thread is restarted to ensure the overall availability of the thread pool. After the inference is completed, the inference thread encapsulates the output pedestrian target bounding box, confidence score, coordinates of 17 human key points and frame ID into a structured inference result and pushes it to the result queue. If the inference result corresponding to the current frame ID is not ready, the result processing thread will block and wait using a condition variable until the inference thread pushes the result and wakes up. After obtaining the inference result, post-processing is performed in the following sequence: "target tracking → re-identification and retrieval → attitude angle calculation → following state determination → serial port command encapsulation → resource cleanup". Specifically, firstly, the inter-frame target association is completed using the ByteTrack algorithm, and a unique tracking ID is assigned. For unassociated potential lost targets, the HSV color histogram and LBP texture features of their torso rigid body region are extracted and matched with the lost target feature library. If the similarity exceeds Th_Sml, re-identification is completed and the original tracking ID is associated. The multi-threaded collaborative security of the target tracking and re-identification unit relies on multiple fine-grained synchronization mechanisms of the system, including: operations on the task queue and result queue are protected by independent mutex locks, allowing only one thread to operate on the queue at a time to avoid data corruption or race conditions; condition variables are bound to the queue state, and the inference thread is only awakened when the task queue is not empty; the result processing thread is only awakened when the result queue contains the current frame ID to be processed, which can significantly reduce CPU idle consumption; atomic variables ensure the temporal consistency of frame data, the frame IDs submitted by the reading thread are strictly incremented, and the result processing thread processes them in the order of frame IDs to ensure the frame order alignment of "acquisition-inference-processing"; For the shared resource tracking target status mapping table and lost target feature library, only the result processing thread reads and writes, thus avoiding the risk of multi-threaded data competition at the source. For active tracking targets, the result processing thread encapsulates control instructions containing tracking ID, target distance, and bounding box coordinates according to a standardized protocol, and sends them to the lower-level machine via serial port at fixed intervals (Interval), avoiding serial port buffer overflow caused by high-frequency transmission. Every certain number of frames processed, resource cleanup is performed to delete expired tracking IDs and lost target features that have not been retrieved within the time limit, and to release memory resources to adapt to the resource constraints of the edge computing platform.

6. A method for summoning and following a robot, characterized in that, The implementation of the robot summoning and following system as described in claim 1 achieves end-to-end localized closed-loop intelligent control of the "perception-decision-execution" chain, specifically including the following steps: Step 1: Initialize and configure the system, complete hardware connection and verification, software startup, deep learning model loading, multi-thread pool initialization (preset number of threads) and parameter preset, and serial port initialization and parameter configuration; sequentially start the Rockchip RK3588S edge computing platform, STM32 microcontroller and USB camera. During startup, the initialization of each hardware unit and link connectivity verification are completed simultaneously. The camera continuously acquires test images of a certain length of frames to verify the integrity of image data, loads the YOLOv8-Pose model, and configures the INT8 operation mode of the RK3588 NPU; Step 2: Multi-threaded image acquisition and parallel inference. Start the image reading thread to acquire image frames in real time and assign unique frame IDs, and submit them to the inference task queue. The inference thread pool calls the LSCD-YOLOv8pose model in parallel to perform inference calculations, complete pedestrian detection and key point extraction, and store the inference results in the result queue. Relying on the INT8 operation acceleration advantage of the RK3588 NPU, the USB camera screen is called to perform inference detection on each frame of the image. After the inference is completed, the bounding boxes, confidence scores and coordinates of 17 key points of all pedestrian targets in the image are output. The detection boxes are sent to the ByteTrack tracker to perform the target tracking process. First, the inference output results are filtered and optimized, the Kalman filter is initialized, and then the position and size of the candidate tracking box of the target in the current frame are predicted based on the state information of the effective target in the previous frame. The intersection-union ratio (IOU) between the tracking box and the high-confidence detection box is calculated. The IOU matching threshold is preset to determine whether to generate inter-frame target association. A unique tracking ID is assigned to the successfully matched trajectory box. The unmatched trajectory box and the low-score detection box enter the second association matching, and a new trajectory identifier is assigned to improve the anti-occlusion ability. Step 3: Perform posture interaction decision-making to determine the following target, analyze the human skeleton keypoint detection and target tracking results, and generate posture interaction following decisions and control commands to complete the initial target tracking association; for each effectively tracked pedestrian target, symmetrically extract and construct human posture analysis triplet data and right arm keypoint triplet data tuple. rt Tuple of key data points for (right ear 4, right shoulder 6, right elbow 8) and left arm lt (Left ear 3, left shoulder 5, left elbow 7); Calculate the angle between the vectors connecting the key points of the triplet, i.e., the angle θ of the right arm. rt For vectors (Right shoulder → Right ear) and vector The angle between the right shoulder and right elbow, and the angle θ between the left arm and elbow. lt For vectors (Left shoulder → Left ear) and vector The angle between the left shoulder and left elbow; initiate a multi-frame verification mechanism to determine human posture semantics and avoid false triggers, such as the angle θ of a pedestrian's right arm. rt If the angle of inclination is less than the attitude angle threshold Th_Agl for more than N consecutive frames, the host computer will record the pedestrian tuple. rt If a target is identified as a valid keypoint, an "Activate Follow" command is generated, marking the tracking ID as the current target to be followed. A control command packet containing a "START" flag, tracking ID, target center x-coordinate, and actual target distance is generated. If the left arm angle is less than Th_Agl for N consecutive frames, a "Stop Follow" command is generated, clearing the current target marker and generating a control command packet containing a "STOP" flag and tracking ID. For valid tracking IDs of non-targets, only the tracking state is maintained, and no control command is generated. For tracking targets without valid keypoints, attitude determination is temporarily not performed, and the decision state of the previous frame is maintained. Step 4: Initiate the multi-feature fusion re-identification process to address tracking interruptions caused by target occlusion or temporary loss in dynamic scenes. For activated targets, determine the torso rigid body feature region Reg_rigid based on key points, i.e., the minimum bounding rectangle formed by the key points of the left shoulder, right shoulder, left hip, and right hip. Weight the HSV color histogram and LBP texture histogram within the feature region Reg_rigid to generate a fused feature vector, normalize it, and store it in the activated target feature library. Simultaneously record the latest bounding box coordinates, tracking ID, and timestamp of the target. For detected targets in the current frame that do not match a tracking box, including newly detected targets or... If a target is suspected to be lost and then reappears, the above feature extraction process is repeated to generate a feature vector to be matched. The cosine similarity between the vector and the weighted feature vector in the lost target feature library is calculated. If the similarity value is greater than a preset threshold, it is determined to be the same target. The current detection box is associated with the original tracking ID and the location information of the ID is updated. At the same time, it is removed from the lost feature library. If the similarity is less than the threshold, it is determined to be a new target. A new tracking state maintenance mechanism is assigned. Tracking targets that have not matched a detection box for N consecutive frames are marked as "lost state" and moved into the lost target feature library. For targets in the lost feature library that have exceeded a preset time, feature vectors and state information are cleaned up to release the memory resources of the edge computing platform. Step 5: After receiving instructions from the host computer, the STM32 microcontroller integrates attitude perception decision instructions with motion control algorithms to achieve stable following of the vehicle. First, it receives the host computer instruction packet via interrupt. After verification, it parses the identifier bit, tracking ID, target position, and distance parameters. Invalid instructions are discarded, and valid instructions are updated to the following parameter buffer in shared memory. Then, it executes steering control, calculating the lateral error between the target center's horizontal coordinate and the image center. This error is input to a steering smoothing PID controller with multi-frame averaging filtering, dead-zone processing, and integral limiting, outputting the steering control quantity Move_Z and limiting the maximum steering speed. Simultaneously, it executes distance control, calculating the error between the actual target distance and the preset following distance. This error is input to a distance PID controller optimized with low-pass filtering, dead-zone processing, and differential limiting, outputting the forward / backward control quantity Move_X and limiting the maximum forward / backward speed. Move_X (forward / backward) and Move_Z (steering) are connected to the speed loop PI controller and steering loop PD controller respectively to obtain the final PWM values ​​for the left and right wheels. After limiting, these values ​​are sent to the motor drive module to control the motor and complete the vehicle following task. Specifically, the USART peripheral is selected to implement asynchronous serial communication. At the hardware level, the physical link between the host computer and the lower-level control board is completed through the TX / RX pins. The communication fields include the target tracking identifier ID, target category, distance calculated by monocular ranging, coordinates of the detection frame center point, and width and height of the detection frame. A target loss stop frame is set to inform the lower-level computer that a specific ID target has been lost, triggering the safe stop logic. At the lower-level receiving end, a circular buffer is used to temporarily store the serial port received data. The data is cyclically stored and read through the receive index and start index management to avoid buffer overflow. In the interrupt service function, each received byte is stored in the buffer. When the end-of-frame marker is detected, the complete data frame from the start index to the receive index is extracted, each field is parsed according to the protocol format, and the parsing result is stored in a structure array. Step 6: Multi-threaded resource management. Optimize edge computing resource utilization through multi-threaded scheduling and data synchronization units. Employ a multi-threaded parallel architecture to decouple model inference from post-processing tasks, splitting computationally intensive tasks for parallel execution. A synchronization and mutual exclusion mechanism ensures multi-threaded data consistency, reducing resource idleness and task blocking, and improving overall system efficiency. The result processing thread is responsible for cleaning up expired trajectory data, feature library data, counter data, and serial port transmission records according to a preset cycle, releasing memory resources. Simultaneously, it monitors thread exit status, terminating all threads after all image frames have been processed, completing system cleanup. Specifically, the video reading thread is responsible for reading image frames from the camera, generating unique frame IDs, and then submitting the frame data to the task queue of the thread pool for subsequent inference thread processing. Separating video reading from subsequent inference and processing avoids overall efficiency degradation due to blocking. In the YOLOv8pose inference thread, four parallel inference threads complete computationally intensive inference tasks, responsible for executing YOLOv8-pose target detection and pose estimation. Each thread runs independently, responsible for retrieving tasks from the task queue, executing inference, and saving results, significantly increasing the number of frames processed per unit time. In the result processing thread, inference results are retrieved from the result queue of the thread pool in order of frame ID, executing target tracking, trajectory re-identification and retrieval, pose decision-making for trajectory status, and serial port command sending logic. Separating complex post-processing logic from the inference thread avoids the impact of time-consuming inference in a single thread on the decision-making process, ensuring the real-time performance of tracking and control commands.

Images