Obstacle avoidance method and device for dynamic object by robot

Abstract

The invention provides an obstacle avoidance method and device for a dynamic object by a robot, and relates to the technical field of robot obstacle avoidance, and the method comprises the steps: collecting a multi-modal data set through a multi-sensor system of the robot; the dynamic object recognition model comprises a multi-modal feature extraction unit and a semantic reasoning large model; performing feature extraction and two-stage feature fusion on the multi-modal data set through a multi-modal feature extraction unit to obtain a final fusion feature vector; performing dynamic object recognition on the final fusion feature vector through a cross-modal attention mechanism of a semantic reasoning large model to obtain a category sequence of a dynamic object; the obstacle avoidance model comprises a state analysis unit and a trajectory prediction unit; according to the category sequence, position calculation and speed calculation of the dynamic object are carried out through a state analysis unit, and a state sequence of the dynamic object is obtained; and inputting the state sequence into a trajectory prediction unit for trajectory prediction to obtain a dynamic object prediction trajectory, and controlling the robot to avoid obstacles according to the dynamic object prediction trajectory.

Description

Technical Field

[0001] This application relates to the field of robot obstacle avoidance technology, and in particular to a method and apparatus for robot obstacle avoidance of dynamic objects. Background Technology

[0002] With the rapid development of service robots, industrial mobile robots, and autonomous robots, the ability of robots to autonomously avoid obstacles in dynamic environments has become one of the core technologies to ensure their safe and stable operation. Currently, robot obstacle avoidance technology generally relies on multiple sensors to collect environmental data, generating obstacle avoidance strategies through the identification, state analysis, and trajectory prediction of dynamic objects. However, in complex dynamic scenarios, existing technologies suffer from insufficient accuracy in identifying dynamic objects and inadequate precision in analyzing their states. This results in robots being unable to effectively predict the trajectories of moving objects, hindering effective obstacle avoidance. Specific shortcomings are as follows:

[0003] Existing technologies suffer from isolated feature extraction and analysis of multimodal data. Feature extraction from images, point clouds, and pose data is independent of each other, and the correlation between features of different modalities is not fully explored. This leads to biases in semantic analysis and reduces the accuracy of dynamic object recognition.

[0004] Existing position calculation techniques do not take into account the details of the image and the spatial accuracy of the point cloud, resulting in a large deviation in position calculation. This leads to a significant difference between the generated dynamic object state and the real state, ultimately increasing the error in trajectory prediction and the risk of obstacle avoidance failure. Summary of the Invention

[0005] To address the above problems, this invention provides a method for a robot to avoid obstacles to dynamic objects, comprising:

[0006] Collect multimodal data sets using the robot's multi-sensor system;

[0007] A dynamic object recognition model is constructed, which includes a multimodal feature extraction unit and a large semantic reasoning model.

[0008] The multimodal feature extraction unit performs feature extraction and two-level feature fusion on the multimodal dataset to obtain the final fused feature vector.

[0009] By using the cross-modal attention mechanism of the semantic reasoning big model, dynamic object recognition is performed on the final fused feature vector to obtain the category sequence of dynamic objects;

[0010] Construct an obstacle avoidance model, which includes a state analysis unit and a trajectory prediction unit;

[0011] Based on the category sequence, the position and velocity of the dynamic object are calculated by the state analysis unit to obtain the state sequence of the dynamic object;

[0012] The state sequence is input into the trajectory prediction unit to predict the trajectory, and the predicted trajectory of the dynamic object is obtained. The robot is then controlled to avoid obstacles based on the predicted trajectory of the dynamic object.

[0013] Optionally, the acquisition of multimodal data sets through the robot's multi-sensor system includes:

[0014] Multi-sensor systems include vision cameras, lidar, and inertial measurement units;

[0015] A multimodal data set is formed by acquiring a set of images of the environment through a visual camera, a set of point cloud data of the environment through a LiDAR, and a set of pose data of the robot through an inertial measurement unit.

[0016] Optionally, the step of performing feature extraction and two-level feature fusion on the multimodal data set through the multimodal feature extraction unit to obtain the final fused feature vector includes:

[0017] The multimodal feature extraction unit includes convolutional neural networks, point cloud feature networks, temporal neural networks, temporal weighted networks, and attention fusion models;

[0018] Image feature vectors of image sets are extracted using convolutional neural networks, point cloud feature vectors of point cloud datasets are extracted using point cloud feature networks, and pose feature vectors of pose data are extracted using temporal neural networks.

[0019] The image feature vector, point cloud feature vector, and pose feature vector are aggregated by temporal weighting network to obtain image temporal aggregated features, point cloud temporal aggregated features, and pose temporal aggregated features.

[0020] The attention fusion model is used to perform attention-weighted fusion of temporal aggregated features of images, temporal aggregated features of point clouds, and temporal aggregated features of pose to obtain the final fused feature vector.

[0021] Optionally, the step of using the cross-modal attention mechanism of the semantic reasoning large model to perform dynamic object recognition on the final fused feature vector to obtain the category sequence of dynamic objects includes:

[0022] The large semantic reasoning model includes a cross-modal attention layer and a semantic decoding layer;

[0023] The final fused feature vector is standardized in dimension and redundant information is filtered out to obtain a standardized feature vector.

[0024] The standardized feature vector is input into the cross-modal attention layer for cross-modal feature enhancement to obtain the cross-modal enhanced feature vector;

[0025] The cross-modal enhanced feature vector is input into the semantic decoding layer for contextual semantic reasoning and category matching to obtain the category of the dynamic object at each time point. The categories are then arranged in chronological order to form a category sequence of the dynamic object.

[0026] Optionally, the step of calculating the position and velocity of the dynamic object based on the category sequence through the state analysis unit to obtain the state sequence of the dynamic object includes:

[0027] Based on the category sequence, the two-dimensional coordinate sequence of dynamic objects is extracted from the image set using an object detection algorithm, and the two-dimensional coordinate sequence is converted into a first reference three-dimensional coordinate sequence.

[0028] Based on the category sequence, point cloud clusters of dynamic objects are extracted from the point cloud dataset using point cloud clustering and segmentation algorithms, and a second reference three-dimensional coordinate sequence is calculated based on the point cloud clusters.

[0029] The first and second reference 3D coordinate sequences are fused into a final 3D coordinate sequence using the Kalman filter algorithm.

[0030] The position difference between adjacent time points is obtained from the final three-dimensional coordinate sequence, and the velocity sequence is calculated based on the position difference.

[0031] The category sequence, the final 3D coordinate sequence, and the velocity sequence are used as the state sequence of the dynamic object.

[0032] Optionally, the trajectory prediction unit includes a dynamic object classification model, a pedestrian trajectory prediction model, a vehicle trajectory prediction model, and other object trajectory prediction models.

[0033] Optionally, the step of inputting the state sequence into the trajectory prediction unit for trajectory prediction to obtain the predicted trajectory of the dynamic object includes:

[0034] Obtain the category sequence, final 3D coordinate sequence, and velocity sequence of the state sequence, and classify the category sequence into pedestrians, vehicles, and other objects using a dynamic object classification model;

[0035] The final three-dimensional coordinate sequence and velocity sequence of the pedestrian are input into the pedestrian trajectory prediction model to predict the trajectory and obtain the initial trajectory of the pedestrian.

[0036] The final three-dimensional coordinate sequence and velocity sequence of the vehicle are input into the vehicle trajectory prediction model to predict the trajectory and obtain the initial trajectory of the vehicle.

[0037] The final three-dimensional coordinate sequence and velocity sequence of other objects are input into the trajectory prediction model of other objects to predict the trajectory and obtain the initial trajectory of other objects.

[0038] The initial trajectories of pedestrians, vehicles, and other objects are superimposed to obtain the initial trajectory of the dynamic object. Environmental constraints are introduced into the initial trajectory of the dynamic object, and the trajectory is optimized by the Monte Carlo tree search algorithm to obtain the predicted trajectory of the dynamic object.

[0039] The present invention also provides an obstacle avoidance device for a robot to avoid dynamic objects, for implementing the aforementioned obstacle avoidance method for a robot to avoid dynamic objects, the device comprising:

[0040] The data acquisition module is used to collect multimodal data sets through the robot's multi-sensor system;

[0041] The dynamic object recognition model building module is used to build dynamic object recognition models, which include multimodal feature extraction units and a large semantic reasoning model.

[0042] The final fusion feature vector acquisition module is used to extract features from the multimodal data set and perform two-level feature fusion through the multimodal feature extraction unit to obtain the final fusion feature vector;

[0043] The category sequence acquisition module is used to perform dynamic object recognition on the final fused feature vector through the cross-modal attention mechanism of the semantic reasoning big model, and obtain the category sequence of dynamic objects.

[0044] The obstacle avoidance model construction module is used to build obstacle avoidance models, which include state analysis units and trajectory prediction units.

[0045] The state sequence acquisition module is used to calculate the position and velocity of the dynamic object based on the category sequence through the state analysis unit, and obtain the state sequence of the dynamic object.

[0046] The obstacle avoidance module is used to input the state sequence into the trajectory prediction unit to predict the trajectory, obtain the predicted trajectory of the dynamic object, and control the robot to avoid obstacles based on the predicted trajectory of the dynamic object.

[0047] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot's obstacle avoidance method for dynamic objects.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the robot's obstacle avoidance method for dynamic objects.

[0049] The present invention has the following beneficial effects:

[0050] 1. By using a dynamic object recognition model to perform two-level feature fusion and cross-modal attention on multimodal data, dynamic object recognition can effectively mine multimodal feature associations and adaptively adjust weights when some modalities fail, thereby improving the recognition accuracy and robustness of dynamic object category sequences. By using an obstacle avoidance model to obtain state sequences based on category sequences and introducing environmental constraints and Monte Carlo tree search optimization in trajectory prediction, the predicted trajectory can conform to the motion law of objects and avoid environmental limitations, thereby improving the accuracy of dynamic object trajectory prediction and ensuring the safe and stable movement of robots.

[0051] 2. The multimodal feature extraction unit first extracts features from the three types of data separately, then the effective features of the temporal dimension are aggregated through a temporal weighted network to highlight the feature contribution of high-confidence time steps. Finally, an attention fusion model is used for adaptive weighted fusion to ensure that the fused features simultaneously cover the three key types of information: visual details, spatial structure, and motion dynamics, avoiding the one-sidedness of single-modal features. The cross-modal attention mechanism of the semantic reasoning big model can explore the correlation between features of different modalities. When a modality fails, the weight of the effective modality is automatically increased to reduce the impact of environmental interference on the recognition results. The final generated dynamic object category sequence has high accuracy and strong temporal consistency, providing a reliable category basis for subsequent state analysis and avoiding obstacle avoidance risks caused by object misjudgment.

[0052] 3. The state analysis unit extracts the two-dimensional coordinates of the image data and converts them into three-dimensional coordinates. For point cloud data, clustering is used to extract point cloud clusters and calculate three-dimensional coordinates. Then, Kalman filtering is used to fuse the two types of coordinates. Combining the accuracy advantages of the two types of data reduces the position calculation error, so that the final generated state sequence can completely depict the motion state of the dynamic object, providing high-quality input for the trajectory prediction unit and avoiding prediction deviations caused by state data errors. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0055] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0057] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0058] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0059] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0060] Reference Figure 1 This invention provides a method for a robot to avoid obstacles to dynamic objects, comprising:

[0061] Collect multimodal data sets using the robot's multi-sensor system;

[0062] In some embodiments, the acquisition of multimodal data sets via the robot's multi-sensor system includes:

[0063] Multi-sensor systems include vision cameras, lidar, and inertial measurement units;

[0064] A multimodal data set is formed by acquiring a set of images of the environment through a visual camera, a set of point cloud data of the environment through a LiDAR, and a set of pose data of the robot through an inertial measurement unit.

[0065] In some embodiments, the multi-sensor system consists of a vision camera, a LiDAR, and an inertial measurement unit (IMU). The first step is to select and install the hardware. The vision camera is a binocular industrial camera with a resolution of 3840×2160, a frame rate of 45fps, and a distortion rate of <1%. It is mounted on the top front of the robot, with the lens horizontally facing forward and unobstructed to ensure detailed capture of distant dynamic objects and low distortion. The LiDAR is a semi-solid-state LiDAR with a horizontal field of view of 120°, a ranging range of 0.5-200m, and a ranging accuracy of ±2cm. It is mounted in the lower middle position of the robot. In outdoor scenarios, a dustproof and waterproof cover is required to ensure accurate detection of objects at medium to long distances. The IMU is a 9-axis IMU module, including a 3-axis accelerometer, a 3-axis gyroscope, and a 3-axis magnetometer, with a sampling rate of 200Hz and angular velocity zero-bias stability of <0.1° / h. It is mounted at the robot's center of gravity and fixed with a shock-absorbing bracket, and its coordinate system is strictly aligned with the robot's body coordinate system to reduce the impact of motion noise.

[0066] The visual camera uses the SDK to set exposure time, white balance, and gain, enables dual-channel synchronous output, and acquires RAW format images for the left and right eyes after receiving a trigger signal. Each frame is stamped with a timestamp and camera identifier, and 300 frames are continuously acquired to form an image set. During acquisition, every 10 frames are checked for average brightness and edge sharpness, and invalid frames are filtered out. The LiDAR uses the SDK to set a dense scanning mode, Cartesian coordinate system + reflection intensity output format, and a 0.5-50m ranging range filter. After receiving a trigger signal, it acquires PCD format point cloud data, adds a corresponding timestamp to each frame, and 67 frames are continuously acquired to form a point cloud data set. Each frame is checked for the number of valid points and the stability of reflection intensity. The inertial measurement unit uses the SDK to set a 200Hz sampling rate, a 50Hz acceleration filter bandwidth, and a 20Hz angular velocity filter bandwidth. After receiving a trigger signal, it acquires pose data every 10ms, stores it in CSV format, and 1340 data points are continuously acquired to form a pose data set. Every 100 data points are checked for acceleration stability and angular velocity zero bias.

[0067] A dynamic object recognition model is constructed, which includes a multimodal feature extraction unit and a large semantic reasoning model.

[0068] The multimodal feature extraction unit performs feature extraction and two-level feature fusion on the multimodal dataset to obtain the final fused feature vector.

[0069] In some embodiments, the step of performing feature extraction and two-level feature fusion on the multimodal data set through a multimodal feature extraction unit to obtain a final fused feature vector includes:

[0070] The multimodal feature extraction unit includes convolutional neural networks, point cloud feature networks, temporal neural networks, temporal weighted networks, and attention fusion models;

[0071] Image feature vectors of image sets are extracted using convolutional neural networks, point cloud feature vectors of point cloud datasets are extracted using point cloud feature networks, and pose feature vectors of pose data are extracted using temporal neural networks.

[0072] In some embodiments, the convolutional neural network uses the ResNet-50 model to collect image feature vectors, the point cloud feature network uses the PointNet++ network to collect point cloud feature vectors, and the temporal neural network uses a bidirectional long short-term memory network to collect pose feature vectors.

[0073] The image feature vector, point cloud feature vector, and pose feature vector are aggregated by temporal weighting network to obtain image temporal aggregated features, point cloud temporal aggregated features, and pose temporal aggregated features.

[0074] In some embodiments, for any mode , Corresponding to image mode, point cloud mode, and pose mode respectively, based on the current time... query vector For window Key vector within Sum value vector Perform temporal weighted aggregation to obtain the temporal aggregated features of each modality:

[0075]

[0076]

[0077] in, Indicates time window For time Modal attention weights, Indicates the scaling factor. Representing modes At any moment Temporal aggregation features, For the current time step, The length of the sliding time window. This is the index for any historical time step within the time window. , Represents a continuous time window; Representing modes Temporal aggregation characteristics;

[0078] The attention fusion model is used to perform attention-weighted fusion of temporal aggregated features of images, temporal aggregated features of point clouds, and temporal aggregated features of pose to obtain the final fused feature vector.

[0079] In some embodiments, image temporal aggregation features, point cloud temporal aggregation features, and pose temporal aggregation features are mapped to a unified dimension. MLP() represents a multilayer perceptron. Representing modes The temporal aggregated features are unified in dimension, and then the modality-level attention weights are calculated based on these unified temporal aggregated features. The expression is as follows:

[0080]

[0081] in, Representing modes Modal attention weights, Represents a learnable vector. This represents the hyperbolic tangent activation function. Representing modes The affine transformation matrix, Representing modes The bias parameters; the final fused feature vector at time t. .

[0082] By using the cross-modal attention mechanism of the semantic reasoning big model, dynamic object recognition is performed on the final fused feature vector to obtain the category sequence of dynamic objects;

[0083] In some embodiments, the step of using the cross-modal attention mechanism of a large semantic reasoning model to perform dynamic object recognition on the final fused feature vector to obtain a sequence of dynamic object categories includes:

[0084] The large semantic reasoning model includes a cross-modal attention layer and a semantic decoding layer;

[0085] The final fused feature vector is standardized in dimension and redundant information is filtered out to obtain a standardized feature vector.

[0086] In some embodiments, dimensionality normalization employs the Z-score algorithm, calculating a global statistic based on the final fused feature vector set across all time steps. For the d-th dimension of a single time step's feature vector, the current feature value is subtracted from the mean of that dimension across all time steps, and then divided by the standard deviation of that dimension to obtain the normalized feature value for that dimension. This process transforms the feature vector of each time step into a normalized feature vector with a dimension still of 1×512, ensuring that different modal components are at the same dimensional level.

[0087] Redundant information filtering employs a combination of variance threshold filtering and L1 regularization sparsity screening. First, the variance of all standardized feature dimensions is calculated, and a variance threshold of 1e-4 is set to filter out dimensions with variances less than this threshold. Then, L1 regularization is applied to the remaining feature vectors, using sparsity constraints to make the eigenvalues ​​of some redundant dimensions approach 0, further eliminating dimensions with absolute eigenvalues ​​less than 1e-5. If the number of dimensions after filtering is less than 512, it is padded to 1×512 using zero-padding, ultimately forming the standardized feature vector for each time step. The standardized feature vectors from all time steps constitute the standardized feature vector.

[0088] The standardized feature vector is input into the cross-modal attention layer for cross-modal feature enhancement to obtain the cross-modal enhanced feature vector;

[0089] In some embodiments, three independent fully connected layers are used to map the standardized feature vectors into Query, Key, and Value vectors with a uniform dimension of 1×128, respectively. For each modality's Query vector, the dot product similarity is calculated with the Key vectors of all modalities to obtain a similarity matrix. This matrix is ​​then normalized using the Softmax function to obtain the cross-modal attention weight vector. ,in, Indicates the current query characteristics. Indicates the number of dimensions. Represents the similarity matrix; cross-modal augmentation eigenvectors at time t. = ,in The standardized eigenvector at time t

[0090] The cross-modal enhanced feature vectors are input into the semantic decoding layer for contextual semantic reasoning and category matching to obtain the categories of dynamic objects at each time point. The categories are then arranged in chronological order to form a category sequence of dynamic objects.

[0091] In some embodiments, context semantic reasoning employs a bidirectional GRU network. The cross-modal enhanced feature vector sequence is input into a single-layer bidirectional GRU with 256 hidden units. The forward GRU captures the temporal dependencies from time step 1 to time step T, and the backward GRU captures the temporal dependencies from time step T to time step 1. The output dimension is T×512. Then, a fully connected layer is used to compress and optimize the output features at each time step to obtain the context semantic feature vector at each time step. The context semantic feature vectors at all time steps constitute the context semantic feature vector sequence.

[0092] The preset dynamic object category library is {pedestrians, vehicles, other dynamic objects}. The classification network adopts a fully connected layer + Softmax structure. The fully connected layer has an input dimension of 256 and an output dimension of 3. The Softmax function converts the output into the probability value of the object belonging to each category at each time step. The category corresponding to the maximum probability is taken as the dynamic object category at that time step, as shown in the following expression:

[0093]

[0094]

[0095]

[0096] in, This indicates unnormalized category scoring. This represents the weight of the linear classification head. This indicates the bias of the linear classification head. Represents the probability distribution of categories. express function, () represents the index function with the highest probability. Let be the predicted category at time t.

[0097] Construct an obstacle avoidance model, which includes a state analysis unit and a trajectory prediction unit;

[0098] Based on the category sequence, the position and velocity of the dynamic object are calculated by the state analysis unit to obtain the state sequence of the dynamic object;

[0099] In some embodiments, the step of calculating the position and velocity of the dynamic object based on the category sequence using a state analysis unit to obtain the state sequence of the dynamic object includes:

[0100] Based on the category sequence, the two-dimensional coordinate sequence of dynamic objects is extracted from the image set using an object detection algorithm, and the two-dimensional coordinate sequence is converted into a first reference three-dimensional coordinate sequence.

[0101] In some embodiments, the t-th frame of the stereo image from the image set is input into the YOLOv8 model, and the model outputs the bounding boxes of all dynamic objects within the image. The center coordinates of these bounding boxes are calculated and used as the two-dimensional image coordinates of the dynamic objects at time step t. The two-dimensional coordinates of all time steps constitute a two-dimensional coordinate sequence. The transformation is achieved based on the intrinsic parameters of the stereo camera. For the t-th frame of the stereo image, a disparity map is calculated using a semi-global block matching algorithm, and the disparity value corresponding to the two-dimensional center coordinates is extracted. Using the principle of stereo ranging, combined with the intrinsic parameters of the left eye camera and the stereo baseline distance, the three-dimensional coordinates with the optical center of the left eye camera as the origin are calculated, resulting in the first reference three-dimensional coordinates at time step t. The coordinates of all time steps constitute the first reference three-dimensional coordinate sequence.

[0102] Based on the category sequence, point cloud clusters of dynamic objects are extracted from the point cloud dataset using point cloud clustering and segmentation algorithms, and a second reference three-dimensional coordinate sequence is calculated based on the point cloud clusters.

[0103] In some embodiments, the DBSCAN density clustering algorithm is selected as the point cloud clustering and segmentation algorithm. This algorithm does not require a preset number of clusters and is suitable for the irregular shapes of dynamic objects. Differentiated parameters are set for different categories in the category sequence: if the category is pedestrian, the neighborhood radius eps = 0.3m and the minimum number of points minsamples = 15 are set; if the category is vehicle, eps = 0.8m and minsamples = 50 are set. The point cloud of frame t is input into the DBSCAN algorithm, which outputs multiple point cloud clusters. The category fit of each point cloud cluster is calculated, and the point cloud cluster with the highest category fit is selected as the target point cloud cluster of the dynamic object. The centroid + bounding box center fusion method is used to calculate the coordinates. First, the centroid coordinates of the target point cloud cluster are calculated; then the axis-aligned bounding box of the point cloud cluster is calculated to obtain the minimum and maximum coordinates of the bounding box, and the average of the two is taken as the center coordinates of the bounding box. The centroid coordinates and the center coordinates of the bounding box are weighted and fused to obtain the second reference three-dimensional coordinates at time step t. The coordinates of all time steps constitute the second reference three-dimensional coordinate sequence.

[0104] The first and second reference 3D coordinate sequences are fused into a final 3D coordinate sequence using the Kalman filter algorithm.

[0105] In some embodiments, a prediction-update iteration is performed at each time step t using the Kalman filter algorithm, and the iteration process is as follows:

[0106]

[0107]

[0108]

[0109]

[0110] Where t is the current time, Indicates time interval, The weights of the image-side observations, Represents the first reference three-dimensional coordinate vector. Represents the second reference three-dimensional coordinate vector. Represents the fused coordinate vector. Represents the prior coordinate vector. Indicates measurement residuals, This is the gain coefficient. Let represent the final coordinate vector at time t. Arrange all the final coordinate vectors in time sequence to form the final three-dimensional coordinate sequence.

[0111] The position difference between adjacent time points is obtained from the final three-dimensional coordinate sequence, and the velocity sequence is calculated based on the position difference.

[0112] In some embodiments, it is assumed that the dynamic object moves at a constant speed in adjacent time steps, and the speed is the difference between the final three-dimensional coordinates of adjacent time steps divided by the time interval to obtain the initial speed of each time step; noise is filtered by averaging using a sliding window, and the average of the current speed and the speed of one time step before and after is taken for the speed of the middle time step; the speed of the first and last time steps of the sequence is smoothed by a double-sided window to avoid abrupt speed changes, and the speeds of all time steps after smoothing are used to form a speed sequence.

[0113] The category sequence, the final 3D coordinate sequence, and the velocity sequence are used as the state sequence of the dynamic object.

[0114] The state sequence is input into the trajectory prediction unit to predict the trajectory, and the predicted trajectory of the dynamic object is obtained. The robot is then controlled to avoid obstacles based on the predicted trajectory of the dynamic object.

[0115] In some embodiments, the trajectory prediction unit includes a dynamic object classification model, a pedestrian trajectory prediction model, a vehicle trajectory prediction model, and other object trajectory prediction models.

[0116] In some embodiments, the step of inputting the state sequence into the trajectory prediction unit for trajectory prediction to obtain the predicted trajectory of the dynamic object includes:

[0117] Obtain the category sequence, final 3D coordinate sequence, and velocity sequence of the state sequence, and classify the category sequence into pedestrians, vehicles, and other objects using a dynamic object classification model;

[0118] In some embodiments, the dynamic object classification model takes the category sequence in the state sequence as input and classifies dynamic objects into three categories: pedestrians, vehicles, and other objects, adapting to the subsequent classification trajectory prediction model. The model uses a lightweight fully connected classifier, and the input features are the historical statistical information of the category sequence, that is, the frequency and proportion of the occurrence of pedestrians, vehicles, and other objects in the historical T time steps, forming a 1×6 feature vector. The output is a three-class label, where 0 represents pedestrians, 1 represents vehicles, and 2 represents other objects.

[0119] The final three-dimensional coordinate sequence and velocity sequence of the pedestrian are input into the pedestrian trajectory prediction model to predict the trajectory and obtain the initial trajectory of the pedestrian.

[0120] The final three-dimensional coordinate sequence and velocity sequence of the vehicle are input into the vehicle trajectory prediction model to predict the trajectory and obtain the initial trajectory of the vehicle.

[0121] The final three-dimensional coordinate sequence and velocity sequence of other objects are input into the trajectory prediction model of other objects to predict the trajectory and obtain the initial trajectory of other objects.

[0122] In some embodiments, the pedestrian trajectory prediction model uses the SocialLSTM model. The input is a T×6 motion feature formed by concatenating the final 3D coordinate sequence and velocity sequence of the pedestrian over T historical time steps. The model consists of a 2-layer LSTM, and the output layer is mapped to a K×3 3D coordinate sequence, i.e., the pedestrian's initial trajectory, through a fully connected layer. During prediction, the model learns the pedestrian's historical movement trends and interaction tendencies, and outputs the coordinates for the next K time steps.

[0123] The vehicle trajectory prediction model uses a CNN-LSTM fusion model. The input consists of two parts: first, T×6 motion features from T historical time steps; second, 1×4 road boundary features extracted from visual camera images. The model uses one convolutional layer and one pooling layer to extract road features, and two LSTM layers to capture temporal motion features. The fusion layer concatenates the road features with the LSTM output and inputs them into a fully connected layer, outputting a K×3 three-dimensional coordinate sequence, which is the initial vehicle trajectory.

[0124] For the trajectory prediction model of other objects, a basic LSTM model is selected, with the input being T×6 motion features from T historical time steps. The model consists of 1 LSTM layer and 1 fully connected layer, and outputs a K×3 three-dimensional coordinate sequence, which is the initial trajectory of other objects. The model predicts future trajectories by learning historical motion trends.

[0125] The initial trajectories of pedestrians, vehicles, and other objects are superimposed to obtain the initial trajectory of the dynamic object. Environmental constraints are introduced into the initial trajectory of the dynamic object, and the trajectory is optimized by the Monte Carlo tree search algorithm to obtain the predicted trajectory of the dynamic object.

[0126] In some embodiments, the initial trajectory overlay is centered on time step alignment, integrating three types of initial trajectories. First, it is verified that the number of future time steps for the initial trajectories of pedestrians, vehicles, and other objects is K, and the time step interval is consistent with the historical sequence. If a certain type of object does not exist, only the initial trajectory of the corresponding category is retained, and the other categories are marked as none. Then, using the time step as an index, it is integrated according to the object type-time step-coordinate structure. For example, for the initial trajectory of the k-th time step, if a pedestrian exists, its coordinates are recorded; if a vehicle exists, its coordinates are recorded; if other objects exist, their coordinates are recorded. Finally, a set of dynamic object initial trajectories with K time steps is formed, and each time step contains the coordinates of all dynamic objects at that moment.

[0127] Environmental constraints include: static obstacle constraints: aligning the bounding box with the calculated axes of the static point cloud clusters from the LiDAR, ensuring the predicted trajectory coordinates do not overlap with the bounding box; motion speed constraints: setting a maximum speed based on object type, ensuring the combined speed of adjacent time steps of the trajectory does not exceed the corresponding threshold; and spatial boundary constraints: constraining the trajectory coordinates to remain within the physical range of the robot's working scene.

[0128] Based on the initial trajectory, the trajectory is iteratively optimized through four steps: selection, expansion, simulation, and backtracking, using the Monte Carlo Tree Search algorithm. Tree nodes are defined as time steps minus object coordinate states, with the root node representing the coordinates of the initial trajectory at the first time step. The selection phase involves using the UCT algorithm to select nodes up to the leaf nodes. The expansion phase generates five candidate child nodes for each leaf node based on its trajectory deviation range. The simulation phase randomly generates subsequent trajectories for each candidate child node, verifying whether constraints are met; if satisfied, the node value is assigned 1, otherwise, the node value is assigned -1. The backtracking phase updates the path nodes by backpropagating the values. After iteration, the node coordinates with the highest value at each time step are selected from the root node, forming the dynamic object prediction trajectory.

[0129] Next, the spatiotemporal sequence of the robot's preset path and predicted trajectory is aligned. A safety distance threshold is then set based on the sum of the robot's and dynamic object's safety radii. Combined with the robot's minimum obstacle avoidance response time, four risk levels are determined: no risk, low risk, high risk, and extreme risk. Obstacle avoidance strategies are then generated according to the risk level: in the absence of risk, the original path and motion state are maintained; in low risk, speed adjustment is the primary method, with minor path fine-tuning; in high risk, a dynamic window method is used to search for the optimal local obstacle avoidance path within the kinematically permissible range; and in extreme risk, an emergency stop and audible / visual alarm are triggered. The strategy is then converted into control commands using the robot's kinematic model. After first-order low-pass filtering and smoothing, the commands are output through a hardware interface. Finally, a real-time closed loop is constructed, updating sensor data and predicted trajectory every 20ms, reassessing the risk, and iterating the strategy. A conservative obstacle avoidance mode is triggered when data is lost or errors exceed limits. The implementation effect is verified through indicators such as obstacle avoidance success rate and control response time to ensure safe obstacle avoidance by the robot.

[0130] Reference Figure 2 This invention provides a robot obstacle avoidance device 20 for moving objects, used to implement a robot obstacle avoidance method for moving objects. The device includes:

[0131] Data acquisition module 21 is used to collect multimodal data sets through the robot's multi-sensor system;

[0132] The dynamic object recognition model construction module 22 is used to construct a dynamic object recognition model, which includes a multimodal feature extraction unit and a large semantic reasoning model.

[0133] The final fusion feature vector acquisition module 23 is used to extract features from the multimodal data set and fuse two levels of features through the multimodal feature extraction unit to obtain the final fusion feature vector;

[0134] The category sequence acquisition module 24 is used to perform dynamic object recognition on the final fused feature vector through the cross-modal attention mechanism of the semantic reasoning big model, and obtain the category sequence of dynamic objects.

[0135] The obstacle avoidance model construction module 25 is used to construct the obstacle avoidance model, which includes a state analysis unit and a trajectory prediction unit.

[0136] The state sequence acquisition module 26 is used to calculate the position and velocity of the dynamic object based on the category sequence through the state analysis unit, and obtain the state sequence of the dynamic object.

[0137] The obstacle avoidance module 27 is used to input the state sequence into the trajectory prediction unit to predict the trajectory, obtain the predicted trajectory of the dynamic object, and control the robot to avoid obstacles based on the predicted trajectory of the dynamic object.

[0138] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements a robot obstacle avoidance method for dynamic objects according to any of the above schemes.

[0139] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0140] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0141] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements a robot obstacle avoidance method for dynamic objects according to any of the above-described schemes. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the methods as described in the embodiments of this application.

[0142] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0143] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for obstacle avoidance by a robot over dynamic objects, characterized in that, include: Collect multimodal data sets using the robot's multi-sensor system; A dynamic object recognition model is constructed, which includes a multimodal feature extraction unit and a large semantic reasoning model. The multimodal feature extraction unit performs feature extraction and two-level feature fusion on the multimodal dataset to obtain the final fused feature vector. By using the cross-modal attention mechanism of the semantic reasoning big model, dynamic object recognition is performed on the final fused feature vector to obtain the category sequence of dynamic objects; Construct an obstacle avoidance model, which includes a state analysis unit and a trajectory prediction unit; Based on the category sequence, the position and velocity of the dynamic object are calculated by the state analysis unit to obtain the state sequence of the dynamic object; The state sequence is input into the trajectory prediction unit to predict the trajectory, and the predicted trajectory of the dynamic object is obtained. The robot is then controlled to avoid obstacles based on the predicted trajectory of the dynamic object.

2. The obstacle avoidance method for dynamic objects by a robot according to claim 1, characterized in that, The collection of multimodal data sets through the robot's multi-sensor system includes: Multi-sensor systems include vision cameras, lidar, and inertial measurement units; A multimodal data set is formed by acquiring a set of images of the environment through a visual camera, a set of point cloud data of the environment through a LiDAR, and a set of pose data of the robot through an inertial measurement unit.

3. The obstacle avoidance method for dynamic objects by a robot according to claim 2, characterized in that, The step of extracting features from a multimodal dataset and performing two-level feature fusion using a multimodal feature extraction unit to obtain a final fused feature vector includes: The multimodal feature extraction unit includes convolutional neural networks, point cloud feature networks, temporal neural networks, temporal weighted networks, and attention fusion models; Image feature vectors of image sets are extracted using convolutional neural networks, point cloud feature vectors of point cloud datasets are extracted using point cloud feature networks, and pose feature vectors of pose data are extracted using temporal neural networks. The image feature vector, point cloud feature vector, and pose feature vector are aggregated by temporal weighting network to obtain image temporal aggregated features, point cloud temporal aggregated features, and pose temporal aggregated features. The attention fusion model is used to perform attention-weighted fusion of temporal aggregated features of images, temporal aggregated features of point clouds, and temporal aggregated features of pose to obtain the final fused feature vector.

4. The obstacle avoidance method for dynamic objects by a robot according to claim 1, characterized in that, The method of using a cross-modal attention mechanism of a large semantic reasoning model to perform dynamic object recognition on the final fused feature vector to obtain a sequence of dynamic object categories includes: The large semantic reasoning model includes a cross-modal attention layer and a semantic decoding layer; The final fused feature vector is standardized in dimension and redundant information is filtered out to obtain a standardized feature vector. The standardized feature vector is input into the cross-modal attention layer for cross-modal feature enhancement to obtain the cross-modal enhanced feature vector; The cross-modal enhanced feature vector is input into the semantic decoding layer for contextual semantic reasoning and category matching to obtain the category of the dynamic object at each time point. The categories are then arranged in chronological order to form a category sequence of the dynamic object.

5. The obstacle avoidance method for dynamic objects by a robot according to claim 2, characterized in that, The step of calculating the position and velocity of a dynamic object based on a category sequence using a state analysis unit to obtain the state sequence of the dynamic object includes: Based on the category sequence, the two-dimensional coordinate sequence of dynamic objects is extracted from the image set using an object detection algorithm, and the two-dimensional coordinate sequence is converted into a first reference three-dimensional coordinate sequence. Based on the category sequence, point cloud clusters of dynamic objects are extracted from the point cloud dataset using point cloud clustering and segmentation algorithms, and a second reference three-dimensional coordinate sequence is calculated based on the point cloud clusters. The first and second reference 3D coordinate sequences are fused into a final 3D coordinate sequence using the Kalman filter algorithm. The position difference between adjacent time points is obtained from the final three-dimensional coordinate sequence, and the velocity sequence is calculated based on the position difference. The category sequence, the final 3D coordinate sequence, and the velocity sequence are used as the state sequence of the dynamic object.

6. The obstacle avoidance method for dynamic objects by a robot according to claim 1, characterized in that, The trajectory prediction unit includes a dynamic object classification model, a pedestrian trajectory prediction model, a vehicle trajectory prediction model, and other object trajectory prediction models.

7. The obstacle avoidance method for dynamic objects by a robot according to claim 6, characterized in that, The step of inputting the state sequence into the trajectory prediction unit for trajectory prediction to obtain the predicted trajectory of the dynamic object includes: Obtain the category sequence, final 3D coordinate sequence, and velocity sequence of the state sequence, and classify the category sequence into pedestrians, vehicles, and other objects using a dynamic object classification model; The final three-dimensional coordinate sequence and velocity sequence of the pedestrian are input into the pedestrian trajectory prediction model to predict the trajectory and obtain the initial trajectory of the pedestrian. The final three-dimensional coordinate sequence and velocity sequence of the vehicle are input into the vehicle trajectory prediction model to predict the trajectory and obtain the initial trajectory of the vehicle. The final three-dimensional coordinate sequence and velocity sequence of other objects are input into the trajectory prediction model of other objects to predict the trajectory and obtain the initial trajectory of other objects. The initial trajectories of pedestrians, vehicles, and other objects are superimposed to obtain the initial trajectory of the dynamic object. Environmental constraints are introduced into the initial trajectory of the dynamic object, and the trajectory is optimized by the Monte Carlo tree search algorithm to obtain the predicted trajectory of the dynamic object.

8. A robot obstacle avoidance device for dynamic objects, used to implement the robot obstacle avoidance method for dynamic objects as described in any one of claims 1 to 7, characterized in that, The device includes: The data acquisition module is used to collect multimodal data sets through the robot's multi-sensor system; The dynamic object recognition model building module is used to build dynamic object recognition models, which include multimodal feature extraction units and a large semantic reasoning model. The final fusion feature vector acquisition module is used to extract features from the multimodal data set and perform two-level feature fusion through the multimodal feature extraction unit to obtain the final fusion feature vector; The category sequence acquisition module is used to perform dynamic object recognition on the final fused feature vector through the cross-modal attention mechanism of the semantic reasoning big model, and obtain the category sequence of dynamic objects. The obstacle avoidance model construction module is used to build obstacle avoidance models, which include state analysis units and trajectory prediction units. The state sequence acquisition module is used to calculate the position and velocity of the dynamic object based on the category sequence through the state analysis unit, and obtain the state sequence of the dynamic object. The obstacle avoidance module is used to input the state sequence into the trajectory prediction unit to predict the trajectory, obtain the predicted trajectory of the dynamic object, and control the robot to avoid obstacles based on the predicted trajectory of the dynamic object.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the obstacle avoidance method for dynamic objects by a robot as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the obstacle avoidance method for dynamic objects by a robot as described in any one of claims 1 to 7.

Images