Mobile robot pedestrian following control method based on real-time behavior cloning

Abstract

The invention relates to a pedestrian following control method for a mobile robot based on real-time behavior cloning. The pedestrian following control method comprises the following steps: acquiring picture data and identifying a target pedestrian in the picture data; relative coordinates of the target pedestrian are calculated, coordinate conversion and splicing are carried out, and a global observation trajectory is obtained; constructing a behavior cloning model to optimize the global observation trajectory, and predicting a noise-removed motion trajectory as a predetermined trajectory of the mobile robot based on the obtained global observation trajectory with noise; constructing a data set to train the behavior clone model, performing parameter optimization on the behavior clone model, and storing the parameter to the mobile robot; and outputting a predetermined trajectory of the mobile robot, and tracking according to the predetermined trajectory to realize pedestrian following control of the mobile robot. Compared with the prior art, the method has the advantages that the motion trail of the mobile robot and the motion trail of the target pedestrian are highly consistent, the calculation resource consumption for generating the smooth trail capable of being directly used for the robot is small, and the accuracy and robustness of target pedestrian search are high.

Description

Technical Field

[0001] This invention relates to the field of robot pedestrian following, and in particular to a mobile robot pedestrian following control method based on real-time behavior cloning. Background Technology

[0002] In target following tasks, intelligent agents (robots) perceive their surroundings through sensors, identify and follow pedestrians to a designated location. This is a fundamental and complete task for achieving artificial intelligence goals, requiring intelligent robots to understand inputs such as images and radar point clouds, infer their current pose, locate the target's position, plan a trajectory, and execute an action at each step. Developments in the field of target following have also spurred the development of natural language interaction (requiring robots to understand voice commands and complete instructed tasks). Therefore, target following has attracted increasing attention from researchers, leading to a large body of work attempting to solve this problem.

[0003] Distance-based pedestrian following methods have been studied for a long time. These methods detect and locate the target, calculate the relative coordinates between the target and the pedestrian (relative to the robot's coordinate system), and directly calculate the output control commands using PID controllers or similar methods based on the error between the current distance between the two and a preset following distance. Although these methods have achieved considerable success in recent years, they suffer from obstacle avoidance failure in environments with poorly perceptible obstacles. A significant limitation is that machine vision and LiDAR struggle to identify obstacles such as transparent glass, which makes their obstacle avoidance strategies highly likely to fail, making following tasks unsustainable in current environments.

[0004] Observing the recent success of navigation-based methods in related tasks, a large number of works have attempted to use navigation-based methods for pedestrian following. Navigation-based methods typically treat pedestrian following as a task where the robot navigates sequentially to a sequence of target points. Unlike traditional point-to-point following methods, navigation-based methods consider the robot's motion strategy over a longer period, belonging to point-to-line following, thus resulting in smoother and more natural following behavior. However, navigation-based pedestrian following methods also face a key challenge: how to obtain a highly feasible reference path.

[0005] Analysis revealed that the path obtained by trajectory reconstruction is inherently highly feasible. Some previous studies have conducted trajectory reconstruction based on video, but almost all of them were offline, which is of little use in tasks with high real-time requirements such as pedestrian following.

[0006] To address the aforementioned issues, we consider the possibility of "line-to-line following." Traditional following directly controls the robot based on the pedestrian's current "point" as the "target point," which is instantaneous. Navigation-based methods, on the other hand, plan a "global path" based on the pedestrian's current "point," allowing for long-range consideration of the robot's path planning. Therefore, it is natural to consider reproducing the target pedestrian's trajectory in real time to obtain the pedestrian's "global trajectory," and then deriving the robot's "global path" based on the pedestrian's "global trajectory," while simultaneously considering the long-range time dependence of the pedestrian's motion behavior and the robot's continuous path planning.

[0007] Chinese invention patent CN107643752A discloses an omnidirectional mobile robot path planning algorithm based on pedestrian trajectory prediction. The algorithm includes: kinematic modeling using a robot chassis and Mecanum wheels mounted on the chassis; path planning by the robot based on current coordinates and target position coordinates; prediction of pedestrian behavior trajectories using a lidar module to calculate the possible range of pedestrians within a certain timeframe; and pedestrian avoidance path planning based on the possible range of pedestrians. This invention's omnidirectional mobile robot path planning algorithm based on pedestrian trajectory prediction can make robot movement more flexible. It tracks pedestrians in the environment in real time using a lidar module, predicts the next pedestrian's position based on the tracking results, and re-plans the path based on the velocity combination near the perpendicular line connecting the predicted points, thus achieving the goal of avoiding pedestrians in advance. However, it still suffers from problems such as high reliance on short obstacle avoidance requirements for mobile robot planning, insufficient smoothness of mobile robot motion control and high computational resource consumption, low efficiency of re-searching after target loss, and a disconnect between trajectory prediction and motion planning.

[0008] In summary, there is currently a lack of a method for controlling pedestrian following in mobile robots based on real-time behavior cloning to solve or partially solve the aforementioned problems. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a mobile robot pedestrian following control method based on real-time behavior cloning, so as to solve or partially solve the problems of mobile robot planning relying on short obstacle avoidance requirements, mobile robot motion control not being smooth enough and consuming a lot of computing resources, as well as the low efficiency of re-searching after target loss and the disconnect between trajectory prediction and motion planning.

[0010] The objective of this invention can be achieved through the following technical solutions: This invention provides a method for controlling pedestrian following in a mobile robot based on real-time behavior cloning, specifically including: S1. Acquire image data and use YOLOv8n to identify target pedestrians in the image data; S2. Establish a three-dimensional spatial coordinate system based on the mobile robot chassis, calculate the relative coordinates of the target pedestrian, perform coordinate transformation and stitching based on the relative coordinates, and obtain the global observation trajectory of the target pedestrian in the map coordinate system; S3. Construct a behavior cloning model to optimize the global observation trajectory, and based on the obtained noisy global observation trajectory, predict a noise-removed motion trajectory as the predetermined trajectory of the mobile robot. S4. Construct a dataset to train the behavior cloning model, use the AdamW optimizer to optimize the parameters of the behavior cloning model, save the final behavior cloning model and deploy it to the mobile robot; S5. Based on the final behavior cloning model after parameter optimization, the mobile robot outputs a predetermined trajectory, and the mobile robot tracks according to the predetermined trajectory to realize pedestrian following control of the mobile robot.

[0011] As a preferred technical solution, YOLOv8n identifies target pedestrians by acquiring image data, outputs the ID and detection box of the target pedestrian, and stores the features of the target pedestrian.

[0012] As a preferred technical solution, the calculation of the global observation trajectory specifically includes: Based on the center pixel coordinates of the target pedestrian detection box ( The relative coordinates of the target pedestrian are calculated by combining the calibrated depth information. The calculation formula is: In the formula, To select the number of pixels in the neighborhood of the center pixel, The depth value for each pixel; The relative coordinates of the target pedestrian with respect to the robot's base_link coordinate system are obtained through coordinate transformation, using the following formula: In the formula, ,for( The calibrated depth value, 1) represents the pixel coordinates of the target pedestrian. Let be the intrinsic parameter matrix, and Π be the transformation matrix from the camera coordinate system to the image coordinate system. For the transformation matrix from the world coordinate system to the camera coordinate system, ( , , () represents the transformed global coordinates. The focal length pixel equivalent in the x-direction. The focal length pixel equivalent in the y-direction. For the camera's focal length, ( , () are the coordinates of the main point. For rotation matrix, It is a translation vector; By using the TF coordinate transformation relationship of the mobile robot, the relative coordinates are transformed into global coordinates in the map coordinate system; By using a sliding window, global coordinates over a period of time are stitched together to obtain trajectory segments, and all trajectory segments are stitched together to obtain a complete global observation trajectory.

[0013] As a preferred technical solution, the encoder of the behavior cloning model includes an initial convolution, a batch normalization layer, a depthwise classifiable convolution, and feature enhancement. The initial convolutional input dimension is mapped to the basic number of channels to establish a linear transformation relationship with the input space and capture the underlying local patterns of the trajectory sequence. The batch normalization layer standardizes the feature distribution of each channel, introduces a learnable scaling parameter γ and offset parameter β, and uses the ReLU activation function to introduce nonlinearity. The depthwise separable convolution adopts a two-stage optimization design, including a serial depthwise convolution stage and a pointwise convolution stage. The depthwise convolution stage uses convolution operations to decouple spatial correlations, and the pointwise convolution stage performs linear combination of the channel features output by the depthwise convolution and completes channel number expansion. Each stage is followed by batch normalization and ReLU activation function. The feature enhancement performs a final expansion of the feature space, utilizing the sparsity of the ReLU activation function to form an implicit feature selection mechanism.

[0014] As a preferred technical solution, the behavior cloning model construction smoothing process includes dilated convolution, batch normalization, nonlinear activation units and random deactivation regularization, and the smoothing process is encapsulated using sequential containers.

[0015] As a preferred technical solution, the decoder of the behavior cloning model adopts a three-level feature reconstruction architecture, including a feature dimensionality reduction layer, a depth-separable refinement layer, and a trajectory reconstruction output layer: The feature dimensionality reduction layer compresses the input feature dimension through convolution kernels, and works with the batch normalization layer to stabilize the training distribution, introducing the ReLU activation function into non-linear expressive power; The depth-separable refinement layer employs a two-stage optimization design to achieve feature refinement, including a depthwise convolution stage and a pointwise convolution stage. The depthwise convolution stage uses convolution kernels to independently extract spatial features from each group of channels. The pointwise convolution stage uses convolution kernels to achieve cross-channel feature fusion, further compressing the dimensionality. Each stage is followed by a batch normalization layer and a ReLU activation function. The trajectory reconstruction output layer uses convolutional kernels to map the refined features back to the original input dimension, completing the final reconstruction of the smooth trajectory.

[0016] As a preferred technical solution, the behavior cloning model performs dimension matching between the encoder output and the final output through a residual connection adaptation layer for further residual connections. The residual connection adaptation layer combines a dimension matching mechanism, linear projection, and residual fusion. The dimension matching mechanism performs an identity mapping when the encoder output and the final output dimension are equal to maintain the integrity of feature transfer; when the dimensions are unequal, it performs a linear projection to linearly map the encoder output to the final output dimension. The linear projection uses convolution kernels to achieve cross-channel feature recombination and completes the linear transformation from the high-dimensional feature space to the original trajectory space through matrix multiplication; The residual fusion performs an element-wise addition operation between the adapted features and the decoder output to form a feature enhancement path, a gradient direct connection path, and an information redundancy mechanism, expressed by the formula: In the formula, The function selects either an identity mapping or convolution for dimension alignment based on the matching of input and output dimensions. For the final output of the model, For mapping implemented using a smoothing module and a decoder, For the encoder output, The mapping implemented for the residual connection adaptation layer. for The input dimension of the function for The expected output dimension of the function.

[0017] As a preferred technical solution, the loss function of the behavioral cloning model is: In the formula, Indicates the total loss. Indicates the losses incurred during reconstruction. Indicates smoothing loss. For the input time step, For time, for True values ​​of the x and y coordinates at any given time. The x and y coordinates are the predicted values ​​at time 1. The weighting coefficients for smoothing loss, express Norm, express Norm.

[0018] As a preferred technical solution, the dataset is obtained by manually collecting the observation trajectory of the target pedestrian, drawing the corresponding ideal smooth trajectory, matching the observation trajectory and the ideal smooth trajectory to generate state-action pairs, and obtaining the final dataset through data augmentation.

[0019] As a preferred technical solution, the method further includes: In the following state, the mobile robot predicts the movement path of the target pedestrian based on a predetermined trajectory; whenever the new coordinates of the target pedestrian are visually located, the deviation between the new coordinates and the predicted pedestrian movement path is calculated; the deviation is used to verify whether the target pedestrian being followed is correct; when the deviation is within a reasonable range, the target being followed is considered correct, and pedestrian following control continues; when the deviation exceeds a reasonable range, the target being followed is considered incorrect, and the robot is forced into a target loss retrieval state; when the mobile robot cannot locate the target pedestrian's position, it enters the target loss retrieval state; when the mobile robot enters the target loss retrieval state, it predicts a movement path based on the predetermined trajectory and continuously performs pedestrian re-identification while moving along the movement path; when the correct target pedestrian is found, pedestrian following control is restored.

[0020] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) High consistency with the movement of the target pedestrian: This invention obtains the movement trajectory of the target pedestrian by observing and optimizing it into a predetermined trajectory of the mobile robot using a behavior cloning model. This achieves the technical effect of high consistency between the movement behavior of the mobile robot in the following state and the movement behavior of the target pedestrian, reducing the need for obstacle avoidance of the mobile robot in complex environments and solving the problem of short planning dependence and high obstacle avoidance requirements of the mobile robot caused by obstacle interference.

[0021] (2) Low computational resource consumption: This invention utilizes the positioning, coordinate transformation and splicing of the target pedestrian, and optimizes it using a behavior cloning model to generate a noise-removed motion trajectory as the predetermined trajectory of the mobile robot. This achieves the generation of a smooth and feasible reference trajectory that can be directly used for robot motion planning with low computational resource consumption, and solves the problems of insufficient smoothness of mobile robot motion control and high computational resource consumption.

[0022] (3) Efficient target retrieval: The present invention uses a predetermined trajectory to predict the movement path of the target pedestrian. When the target is lost, the mobile robot moves along the movement path and continuously performs pedestrian re-identification, which solves the problems of low efficiency of re-searching after the target is lost and the disconnect between trajectory prediction and motion planning, and realizes the improvement of the search efficiency of the target pedestrian.

[0023] (4) Target correctness verification: The present invention uses a predetermined trajectory to predict the movement path of the target pedestrian. In the following state, the deviation degree is calculated by the new coordinates of the identified target pedestrian to help verify whether the target pedestrian is correct, thereby improving the accuracy and robustness of long-term target following. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the relationship between trajectory reproduction and path planning in this invention; Figure 2 This is a schematic diagram of the method framework of the present invention; Figure 3 This is a schematic diagram of the model of the present invention; Figure 4 This is a schematic diagram of the behavioral cloning model of the present invention; Figure 5 This is a diagram showing the trajectory optimization effect of the behavioral cloning model of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] To address the problems existing in the prior art, this embodiment provides a mobile robot pedestrian following control method based on real-time behavior cloning, such as... Figure 1 As shown, by locating the target pedestrian, obtaining the pedestrian's observed trajectory, and optimizing its reproduction, the mobile robot obtains its predetermined trajectory. Simultaneously, the trajectory is predicted to determine whether the tracking target is correct. The specific steps are as follows: Figure 2 As shown, it includes: S1. Acquire image data and use YOLOv8n to identify target pedestrians in the image data.

[0027] The mobile robot first reads the current scene from the front-facing camera, then converts the image data into NumPy format for processing in Python. To detect pedestrians in the image, the NumPy image data is converted into PyTorch tensor format. During the detection phase, YOLOv8n + Bytetrack is used to identify and track target pedestrians in the image. YOLOv8n's tracking mode (track) is used, while the Bytetrack tracker is used for detection updates. YOLOv8n detects pedestrian information from the original image, outputs the pedestrian's ID and detection bounding box, and stores the target pedestrian's features (gender, clothing color, whether wearing glasses, etc.).

[0028] The detection boxes for irrelevant pedestrians are filtered out, retaining only the tracking of the target pedestrian. Bytetrack maintains the same trackID for the same target during target tracking. Experiments have shown that this is indeed the case when the target is not completely occluded by obstacles for an extended period and does not leave the frame. Therefore, under normal circumstances, filtering by trackID is sufficient to filter irrelevant pedestrians. Theoretically, in most cases, the detection box area of ​​the target pedestrian should be the largest (because it is closest to the robot, and objects appear larger when closer). Therefore, by comparing the areas of each detection box, auxiliary filtering of irrelevant pedestrians can be achieved.

[0029] S2. Establish a three-dimensional spatial coordinate system based on the mobile robot chassis, calculate the relative coordinates of the target pedestrian, perform coordinate transformation and stitching based on the relative coordinates, and obtain the global observation trajectory of the target pedestrian in the map coordinate system.

[0030] First, based on the center pixel coordinates of the target pedestrian detection box ( The relative coordinates of the target pedestrian are calculated by combining the calibrated depth information. The calculation formula is: In the formula, To select the number of pixels in the neighborhood of the center pixel, The depth value for each pixel; Furthermore, the relative coordinates of the target pedestrian with respect to the robot's base_link coordinate system are obtained through coordinate transformation, using the following formula: In the formula, ,for( The calibrated depth value of point ( 1) represents the pixel coordinates of the target pedestrian. Let be the intrinsic parameter matrix, and Π be the transformation matrix from the camera coordinate system to the image coordinate system. For the transformation matrix from the world coordinate system to the camera coordinate system, ( , , () represents the transformed global coordinates. The focal length pixel equivalent in the x-direction. The focal length pixel equivalent in the y-direction. For the camera's focal length, ( , () are the coordinates of the main point. For rotation matrix, It is a translation vector; By using the TF coordinate transformation relationship of the mobile robot, the relative coordinates in the robot's base_link coordinate system are transformed to the global coordinates in the map coordinate system; Use a sliding window to display the global coordinates for the most recent period of time. The trajectory fragments obtained by splicing together the data within this time period are obtained. By stitching together all trajectory fragments, a complete global observation trajectory can be obtained. .

[0031] The center pixel coordinates of the pedestrian detection bounding box are considered the target's position. By aligning the RGB and depth images, depth information of the neighborhood of that point can be extracted, and the average value is taken as the distance between the target and the robot. The coordinate transformation matrix between the pixel coordinate system, camera coordinate system, robot base coordinate system, and map coordinate system is calculated. Combined with the target's pixel coordinates and distance, the target's world coordinates are calculated. Stitching together the target's world coordinates over a continuous period can approximate its trajectory during that time. Higher coordinate sampling frequency and higher coordinate calculation accuracy result in a trajectory closer to the true trajectory. However, in practice, due to the low frame rate of target detection and tracking (around 15 FPS) and the low accuracy of vision-based positioning (±10cm), the stitched observation trajectory exhibits strong fluctuations and is difficult to directly use for robot path planning.

[0032] S3. Construct a behavior cloning model to optimize the global observation trajectory. Based on the obtained noisy global observation trajectory, predict the noise-removed motion trajectory as the predetermined trajectory of the mobile robot.

[0033] A behavior cloning model is constructed to optimize the observation model, resulting in a smoother trajectory that is more suitable for path planning. This model is built in the PyTorch framework and includes an encoder, a smoothing module, and a decoder, which are implemented using convolution and residual connections.

[0034] like Figure 3As shown, the observed trajectory is input into the encoder, which extracts its high-level temporal features and outputs a latent feature representation. This representation is then fed into a smoothing module for temporal smoothing and noise suppression. The smoothed features are reconstructed by the decoder to generate a preliminary prediction result (denoted as Output 1). Simultaneously, the latent feature representation output by the encoder is adjusted in terms of feature dimension and structure via a residual adaptation layer to align it with the representation space of Output 1, resulting in another set of trajectory prediction results (denoted as Output 2). Finally, a weighted linear fusion of Output 1 and Output 2 yields the system-fused trajectory output. The residual connection adaptation layer employs a dual-branch design logic. When the number of channels does not match, a dimension-transformed convolution is used, with the convolution type being 1×1 point convolution; when the dimensions match, an identity mapping is used, directly passing the tensor without any transformation.

[0035] The input to this model is the pedestrian's historical motion trajectory, which is derived from the 3D coordinates of the target pedestrian located by a depth camera. This coordinates are then transformed to the map coordinate system using the robot's TF coordinate transformation relationship to synthesize the basic motion trajectory. This data is obtained through calculations using multiple sensors, such as depth camera for target localization and LiDAR for robot localization. This involves numerous coordinate transformations, which introduce errors and noise. After processing these errors and noise, the resulting trajectory can be used as the robot's motion reference trajectory. The output of this model is the robot's predetermined reference trajectory. Based on the collected noisy pedestrian motion trajectory, the model predicts a perfect, noise-free, and error-free pedestrian motion trajectory, which is then used as the ideal reference trajectory for the robot to follow the pedestrian. This output provides a better feasible path for the robot's movement.

[0036] The encoder section includes an initial convolutional module, a batch normalization layer, a depthwise separable convolutional module, and a feature enhancement module, which efficiently extract features from the model input. To ensure the final output better reflects the robot's smooth and natural motion, it needs to have good smoothness. Therefore, an additional smoothing module is added, whose core components include dilated convolutions, batch normalization, non-linear activation units, and random deactivation regularization, forcing the model to learn processing methods that make the trajectory smoother. The decoder section adopts a three-level feature reconstruction architecture, including a feature dimensionality reduction layer, a depthwise separable refinement layer, and a trajectory reconstruction output layer. Finally, residual connections are added.

[0037] 1. The encoder adopts a modular design, with each sub-module having unique design considerations in terms of computational efficiency and feature extraction capabilities. The specific design is as follows: (1) The initial convolutional module serves as the starting entry layer for feature processing. It adopts a standard one-dimensional convolutional architecture to achieve a linear mapping of the input space. Its convolutional kernel size is set to 5×1 to achieve a balance between the receptive field and the computational cost. Compared with a 3×1 convolutional kernel size, it can capture a wider range of temporal patterns. Compared with a 7×1 convolutional kernel size, it reduces the number of parameters by 28.6%. The temporal length is kept constant by using symmetric padding (padding=2) to avoid information loss due to downsampling.

[0038] (2) The batch normalization layer standardizes the feature distribution of each channel and uses learnable scaling parameter γ and offset parameter β to alleviate the gradient vanishing problem that occurs in deep networks during training and ensure training stability. The ReLU activation function is used and nonlinear expression capability is introduced to save memory usage, which is crucial for processing long sequences.

[0039] (3) The depthwise separable convolution module adopts a two-stage optimization design, including a serial depthwise convolution stage and a pointwise convolution stage. The depthwise convolution stage uses convolution operations with a group number of base_channels. Each group of channels performs 5×1 spatial convolution independently to decouple spatial correlation. The pointwise convolution stage uses a 1×1 convolution kernel to linearly combine the channel features output by the depthwise convolution, expanding the number of channels to base_channels×2. Each stage is followed by batch normalization and ReLU activation function.

[0040] (4) The feature enhancement module uses 3×1 convolutional kernels to perform feature expansion. Specifically, it uses a stacked 3×1 convolutional structure. This reduces the number of parameters by 40% compared to 5×1 convolutions while maintaining a similar receptive field. This design, which expands the number of channels by four times, is based on the principle of feature pyramid construction, where higher-level features contain richer semantic information. (Batch normalization layer) The parameter is set to This ensures numerical stability. ReLU activates all sparsity properties, causing some neurons to output zero, naturally forming an implicit feature selection mechanism.

[0041] 2. Construct a smoothing module to process time-series signals. Its core components include dilated convolution, batch normalization, nonlinear activation units, and random deactivation regularization.

[0042] (1) The dilated convolution uses a 5×1 convolution kernel and sets the dilation rate to 2. While keeping the number of parameters unchanged, the temporal receptive field is expanded from 5 to 9 time steps. The special design of padding=4 ensures the consistency of the length of the input and output sequences and guarantees the consistency of the spatiotemporal dimension of the feature map.

[0043] (2) Introducing a batch normalization layer can effectively reduce the internal covariate bias that occurs during deep network training. By using the mean and variance to standardize the model, the convergence of the model can be accelerated. In this way, deep networks can use a higher training learning rate and alleviate the gradient explosion problem.

[0044] (3) The nonlinear activation unit uses the ReLU activation function to introduce nonlinear transformation capability, thereby enhancing the model's ability to express complex trajectory patterns.

[0045] (4) Random deactivation regularization uses the Dropout layer to randomly drop neuron connections with a probability of 0.1, preventing overfitting during training.

[0046] 3. The decoder adopts a three-level feature reconstruction architecture, encapsulating the three processing stages through sequential containers: (1) Feature dimensionality reduction layer: The input feature dimension is compressed from base_channels×4 to base_channels×2 by a 3×1 convolution kernel, and the training distribution is stabilized by a batch normalization layer. The ReLU activation function introduces non-linear expressive power.

[0047] (2) A depth-separable refining layer is used to achieve feature refining through a two-level optimization design, including: In the depthwise convolution stage: a 5×1 convolution kernel with a group size of base_channels×2 is used, and spatial feature extraction is performed independently for each group of channels to maintain computational efficiency.

[0048] Pointwise convolution stage: Cross-channel feature fusion is achieved through 1×1 convolution kernels, and the dimension is adjusted from base_channels×2 to base_channels.

[0049] Each stage is followed by batch normalization and ReLU activation functions.

[0050] (3) Trajectory reconstruction output layer: A 5×1 convolution kernel is used to map the refined features from the base_channels dimension back to the original input dimension to complete the final reconstruction of the smooth trajectory.

[0051] The behavioral cloning model utilizes a residual connection adaptation layer as a dimensionality adaptation bridge between the encoder and the final output, thereby achieving residual connections: 1. Dimension matching mechanism: When the dimensions are equal, the nn.Identity() identity mapping is activated to maintain the integrity of feature transfer; when the dimensions are unequal, the 1×1 convolutional projection is activated to linearly map the base_channels×4 dimension of the encoder output to the original input dimension.

[0052] 2. Linear projection design: A 1×1 convolution kernel is used to realize cross-channel feature recombination. While keeping the time length unchanged, the linear transformation from the high-dimensional feature space to the original trajectory space is completed by matrix multiplication.

[0053] 3. Residual fusion function: The adapted features are added element-wise to the decoder output to form: (1) Feature enhancement path: Preserve the low-level trajectory features extracted by the encoder; (2) Gradient direct connection channel: alleviates the gradient decay problem in deep network training; (3) Information redundancy mechanism: Provides model fault tolerance and prevents information loss.

[0054] The mathematical expression for residual fusion is: In the formula, The function selects either an identity mapping or a 1×1 convolution for dimension alignment based on the matching of input and output dimensions. The final layer uses a linear projection convolution without an activation function to directly output the continuous value prediction result. For the final output of the model, For mapping implemented using a smoothing module and a decoder, For the encoder output, The mapping implemented for the residual connection adaptation layer. for The input dimension of the function for The expected output dimension of the function.

[0055] S4. Construct a dataset to train the behavior cloning model, use the AdamW optimizer to optimize the parameters of the behavior cloning model, save the final behavior cloning model and deploy it to the mobile robot.

[0056] A dataset specifically designed for the current trajectory optimization task is used to train the model. The dataset contains n samples, each representing an ideal trajectory. and observation trajectory Composition of state-action pairs , The observed trajectories of the target pedestrians were manually collected, and their corresponding ideal smooth trajectories were manually drawn. These two trajectories were then matched one-to-one to generate state-action pairs, which were further processed using data augmentation to obtain the final dataset. To ensure high smoothness in the model output while preserving the overall trend of the trajectory, the loss function was designed as follows: In the formula, Indicates the total loss. Indicates the losses incurred during reconstruction. Indicates smoothing loss. For the input time step, For time, for True values ​​of the x and y coordinates at any given time. for Predicted values ​​of the x and y coordinates at any given time. The weighting coefficients for smoothing loss, express Norm, express The norm allows the final model to strike a balance between the smoothness of the output trajectory and its similarity to the original input.

[0057] The AdamW optimizer was used for parameter updates, with a batch size of 64 and a total training duration of 200 epochs. The model was trained and optimized, and the final model was saved and deployed to the robot. During the following task, the global observation trajectory obtained by this model was processed, and its corresponding ideal smooth trajectory was output.

[0058] S5. Based on the final behavior cloning model after parameter optimization, the mobile robot outputs a predetermined trajectory. The mobile robot tracks according to the predetermined trajectory, realizing pedestrian following control of the mobile robot.

[0059] The system initially determines whether the pedestrian is the original target by comparing their ID. If the determination is "yes," steps S2, S3, S4, and S5 are executed sequentially. Using the output mobile robot's predetermined trajectory as a reference, MPC control is used for local planning to track the reference trajectory and thus complete pedestrian following. If the determination is "no," the system checks whether the target is lost by analyzing the bounding box position, movement trend, area size, and pedestrian characteristics (gender, clothing color, whether wearing glasses, etc.). If the target is not lost but its ID has been changed, the new ID of the target pedestrian is updated, and tracking continues. Steps S2, S3, S4, and S5 are executed sequentially, using the output mobile robot's predetermined trajectory as a reference for local planning to complete trajectory tracking. If the target is lost, step S6, the target loss retrieval logic, is executed.

[0060] Based on a predetermined trajectory, the system predicts the pedestrian's movement path. Whenever new coordinates of the target pedestrian are obtained, the deviation between the new coordinates and the predicted pedestrian's movement path is calculated. The deviation is used to verify whether the target pedestrian being followed is correct. When the deviation is within a reasonable range, the target is considered correct and the following status is normal. If the deviation is too high, the target is considered incorrect, the following behavior is interrupted, and the S6 target loss retrieval logic is executed instead.

[0061] When the target leaves the frame, the robot immediately enters the target loss retrieval state and executes the S6 target loss retrieval logic.

[0062] S6. When the mobile robot enters the target loss retrieval state, it predicts the search and retrieval motion path based on the latest predetermined trajectory. While moving along the path, it performs pedestrian re-identification. When the target pedestrian is detected again, it switches to normal following state.

[0063] This invention proposes a real-time behavior cloning-based pedestrian following control method for mobile robots. This method effectively enables the robot to perform pedestrian following tasks "naturally" in environments with poorly perceptible obstacles, and allows it to attempt autonomous retrieval when the target is lost. Pedestrian feature matching is performed only during the retrieval phase, reducing computational overhead and resulting in a smoother, more natural following behavior. Furthermore, a behavior cloning model is created to optimize the observed trajectory. Experiments on a self-built dataset validate some of the functionalities, such as… Figure 5 As shown, the method of this invention has good optimization effects on observation trajectories of different shapes: median filtering has a certain trajectory optimization effect, maintaining the overall trajectory trend, but the trajectory is not smooth and natural enough, and the beginning and end are slightly shorter than the original trajectory; moving average filtering has a high trajectory optimization effect, the trajectory is relatively smooth, and the overall trajectory trend is preserved, but the beginning and end are significantly shorter than the original trajectory; the model output trajectory of this invention has a good optimization effect, and the beginning and end are consistent with the original trajectory, while also preserving the overall trend of the input. In trajectory processing, the trajectory-level behavior cloning method of this invention has a smoothing effect higher than median filtering, and is superior to median filtering and moving average filtering in terms of boundary effects. Simulation experiments were conducted in Gazebo, and the simulation effect of trajectory tracking was visualized using RViz, showing that the robot's reference to reproduce the trajectory for tracking is effective.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for controlling pedestrian following in a mobile robot based on real-time behavior cloning, characterized in that, The method specifically includes: S1. Acquire image data and use YOLOv8n to identify target pedestrians in the image data; S2. Establish a three-dimensional spatial coordinate system based on the mobile robot chassis, calculate the relative coordinates of the target pedestrian, perform coordinate transformation and stitching based on the relative coordinates, and obtain the global observation trajectory of the target pedestrian in the map coordinate system; S3. Construct a behavior cloning model to optimize the global observation trajectory. Based on the obtained noisy global observation trajectory, predict a noise-removed motion trajectory as the predetermined trajectory of the mobile robot. The behavior cloning model includes an encoder, a smoothing module, and a decoder. S4. Construct a dataset to train the behavior cloning model, use the AdamW optimizer to optimize the parameters of the behavior cloning model, save the final behavior cloning model and deploy it to the mobile robot; S5. Based on the final behavior cloning model after parameter optimization, the mobile robot outputs a predetermined trajectory, and the mobile robot tracks according to the predetermined trajectory to realize pedestrian following control of the mobile robot.

2. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The YOLOv8n identifies target pedestrians by acquiring image data, outputs the ID and detection box of the target pedestrian, and stores the features of the target pedestrian.

3. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The calculation of the global observation trajectory specifically includes: Based on the center pixel coordinates of the target pedestrian detection box ( The relative coordinates of the target pedestrian are calculated by combining the calibrated depth information. The calculation formula is: In the formula, To select the number of pixels in the neighborhood of the center pixel, The depth value for each pixel; The relative coordinates of the target pedestrian with respect to the robot's base_link coordinate system are obtained through coordinate transformation, using the following formula: In the formula, ,for( The calibrated depth value of point ( 1) represents the pixel coordinates of the target pedestrian. Let be the intrinsic parameter matrix, and Π be the transformation matrix from the camera coordinate system to the image coordinate system. For the transformation matrix from the world coordinate system to the camera coordinate system, ( , , () represents the transformed global coordinates. The focal length pixel equivalent in the x-direction. The focal length pixel equivalent in the y-direction. For the camera's focal length, ( () are the coordinates of the main point. For rotation matrix, It is a translation vector; By using the TF coordinate transformation relationship of the mobile robot, the relative coordinates are transformed into global coordinates in the map coordinate system; By using a sliding window, global coordinates over a period of time are stitched together to obtain trajectory segments, and all trajectory segments are stitched together to obtain a complete global observation trajectory.

4. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The encoder of the behavior cloning model includes an initial convolution, a batch normalization layer, a depthwise classifiable convolution, and feature augmentation. The initial convolution maps the input dimension to the basic number of channels, establishes a linear transformation relationship with the input space, and captures the underlying local patterns of the trajectory sequence; The batch normalization layer standardizes the feature distribution of each channel, introduces a learnable scaling parameter γ and offset parameter β, and uses the ReLU activation function to introduce nonlinearity. The depthwise separable convolution adopts a two-stage optimization design, including a serial depthwise convolution stage and a pointwise convolution stage. The depthwise convolution stage uses convolution operations to decouple spatial correlations, and the pointwise convolution stage performs linear combination of the channel features output by the depthwise convolution and completes channel number expansion. Each stage is followed by batch normalization and ReLU activation function. The feature enhancement performs a final expansion of the feature space, utilizing the sparsity of the ReLU activation function to form an implicit feature selection mechanism.

5. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The behavior cloning model construction smoothing process includes dilated convolution, batch normalization, nonlinear activation units, and random deactivation regularization, and the smoothing process is encapsulated using sequential containers.

6. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The decoder of the behavior cloning model adopts a three-level feature reconstruction architecture, including a feature dimensionality reduction layer, a depthwise separable refinement layer, and a trajectory reconstruction output layer: The feature dimensionality reduction layer compresses the input feature dimension through convolution kernels, and works with the batch normalization layer to stabilize the training distribution, introducing the ReLU activation function into non-linear expressive power; The depth-separable refinement layer employs a two-stage optimization design to achieve feature refinement, including a depthwise convolution stage and a pointwise convolution stage. The depthwise convolution stage uses convolution kernels to independently extract spatial features from each group of channels. The pointwise convolution stage uses convolution kernels to achieve cross-channel feature fusion, further compressing the dimensionality. Each stage is followed by a batch normalization layer and a ReLU activation function. The trajectory reconstruction output layer uses convolutional kernels to map the refined features back to the original input dimension, completing the final reconstruction of the smooth trajectory.

7. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The behavior cloning model performs dimension matching between the encoder output and the final output through a residual connection adaptation layer for further residual connections. The residual connection adaptation layer combines a dimension matching mechanism, linear projection, and residual fusion. The dimension matching mechanism performs an identity mapping when the encoder output and the final output dimension are equal to maintain the integrity of feature transfer; when the dimensions are unequal, it performs a linear projection to linearly map the encoder output to the final output dimension. The linear projection uses convolution kernels to achieve cross-channel feature recombination and completes the linear transformation from the high-dimensional feature space to the original trajectory space through matrix multiplication; The residual fusion performs element-wise addition between the adapted features and the decoder output to form a feature enhancement path, a gradient direct connection channel, and an information redundancy mechanism, expressed by the following formula: In the formula, The function selects either an identity mapping or convolution for dimension alignment based on the matching of input and output dimensions. For the final output of the model, For mapping implemented using a smoothing module and a decoder, For the encoder output, The mapping implemented for the residual connection adaptation layer. for The input dimension of the function for The expected output dimension of the function.

8. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The loss function of the behavior cloning model is: In the formula, Indicates the total loss. Indicates the losses incurred during reconstruction. Indicates smoothing loss. For the input time step, For time, for True values ​​of the x and y coordinates at any given time. for Predicted values ​​of the x and y coordinates at any given time. The weighting coefficients for smoothing loss, express Norm, express Norm.

9. The method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The dataset is obtained by manually collecting the observation trajectory of the target pedestrian, drawing the corresponding ideal smooth trajectory, matching the observation trajectory and the ideal smooth trajectory to generate state-action pairs, and then obtaining the final dataset through data augmentation.

10. A method for pedestrian following control of a mobile robot based on real-time behavior cloning according to claim 1, characterized in that, The method further includes that, in the following state, the mobile robot predicts the movement path of the target pedestrian based on a predetermined trajectory. Whenever the new coordinates of the target pedestrian are visually located, the deviation between the new coordinates and the predicted pedestrian movement path is calculated. The target pedestrian being followed is verified based on the deviation. When the deviation is within a reasonable range, the target being followed is considered correct, and pedestrian following control continues. When the deviation exceeds a reasonable range, the target being followed is considered incorrect, and the robot is forced to enter the target loss retrieval state. When the mobile robot is unable to locate the target pedestrian, it enters the target loss and retrieval state. When the mobile robot enters the target loss retrieval state, it predicts the movement path based on the predetermined trajectory and continuously performs pedestrian re-identification while moving along the movement path. When the correct target pedestrian is found, pedestrian following control is restored.

Images