An interactive robot based on multi-envelope sphere human region perception, close-range safe collision avoidance method and model

Abstract

The application relates to the field of interactive robot safety obstacle avoidance, and discloses an interactive robot close-range safety collision avoidance method based on a multi-envelope spherical human body area perception, which constructs a multi-envelope spherical human body area visualization labeling module based on a human body data set, generates high-precision training samples, designs a double-input network model, takes three-dimensional skeleton coordinates of 19 key joints of a human body and three-dimensional point clouds of the human body as double inputs, deeply fuses skeleton features and point cloud features, realizes accurate prediction of multi-envelope spherical parameters of each limb area of the human body, deploys the trained model to an interactive robot end, provides the robot with real-time and reliable human body area space contour perception capability, and supports close-range safety collision avoidance decision-making of the interactive robot. The application improves the perception accuracy and obstacle avoidance safety of the interactive robot on the human body, a lightweight model adapts to the deployment requirement of the interactive robot end side, and can be widely applied to human body safety protection in interactive scenes such as industrial collaborative robots and service robots.

Description

Technical Field

[0001] This invention belongs to the field of interactive robot safety obstacle avoidance. More specifically, it relates to an interactive robot based on multi-envelope sphere human body region perception, a close-range safe collision avoidance method and model, which is applicable to human safety protection in scenarios such as industrial collaborative robots and service robots that require close interaction with the human body. By accurately perceiving the spatial contours of various limb regions of the human body, the interactive robot can actively avoid collisions, thus ensuring personnel safety. Background Technology

[0002] In applications such as industrial collaboration and service robots, close-range interactions between robots and humans are becoming increasingly frequent. Accurately sensing the spatial contours of the human body and achieving safe obstacle avoidance is a core issue in ensuring personnel safety. Current technology has the following shortcomings: 1. Insufficient accuracy in human spatial contour perception: Existing human perception methods cannot accurately capture the three-dimensional spatial contour information of each limb of the human body, resulting in deviations in the quantitative calculation of the space occupied by the human body; thus, they cannot provide accurate decision-making basis for planning the safe activity range of the robot, reducing the free control range of the robot operation and narrowing the robot's activity range.

[0003] 2. Single-modality sensory input: Current human perception solutions are mostly limited to single-modality input data, such as skeleton data, 2D video streams, or 3D point cloud data. Single-modality input leads to a single dimension of feature extraction. The human safety envelope sphere constructed based on this feature has problems such as unreasonable safety redundancy and poor contour compactness. Moreover, the processing flow of single-modality data is prone to large delays, which cannot meet the real-time requirements of human-computer interaction.

[0004] 3. Poor adaptability of multimodal perception models: Existing multimodal human perception models generally have complex network structures and large number of parameters, making it difficult to adapt to the limited computing resources on the robot's edge. Their inference latency is usually ≥100ms and the computing power occupancy rate is high, which cannot meet the deployment requirements of real-time inference on the robot's edge and limits the engineering application of the model in actual robot systems.

[0005] 4. Weak generalization ability of human envelope calculation: Existing human envelope sphere calculation schemes mostly adopt traditional geometric calculation methods, or rely on training datasets with problems such as single type and insufficient sample size; resulting in insufficient model robustness. When there are abnormal situations such as individual differences, posture deviation or environmental interference in the input data, the calculation deviation of the human envelope sphere increases significantly. The generalization ability of human recognition and envelope modeling is weak and cannot adapt to complex and ever-changing human-computer interaction scenarios.

[0006] The aforementioned issues make it difficult for robots to meet the requirements of practical applications in terms of human perception accuracy and obstacle avoidance safety, thus limiting the large-scale implementation of human-machine collaborative scenarios. Summary of the Invention

[0007] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an interactive robot based on multi-envelope sphere human body area perception, a close-range safe collision avoidance method and model, the purpose of which is to solve the technical problem that the perception accuracy and obstacle avoidance safety of robots in the existing technology are difficult to meet the needs of practical applications.

[0008] To achieve the above objectives, according to one aspect of the present invention, a near-range collision avoidance method for interactive robots based on multi-envelope sphere human body region perception is provided, comprising: A multi-envelope spherical human region visualization annotation module was constructed based on the human dataset to generate structured training samples: A dual-input neural network model is constructed for parameter prediction of the human body's multi-envelope sphere region. The dual inputs are the three-dimensional skeleton coordinates of 19 key joints of the human body and the three-dimensional point cloud of the human body. Training a dual-input network model based on labeled structured training samples includes: Training data augmentation is performed by batch loading human sample data containing skeletons, point clouds, and multi-envelope sphere annotations, and synchronous transformation, noise simulation, and oversampling balancing operations are executed; at the same time, a multi-stage training strategy is adopted, and the full model is fine-tuned after pre-training the feature modules. Construct a weighted spatial distance loss function, which includes spatial Euclidean distance loss and radius mean square error loss. The spatial distance loss weight of the limb region envelope sphere is a times that of the core region, and the radius loss weight is b times that of the core region, where b>a>1. A dual-input network model is trained using gradient pruning, cosine annealing learning rate dynamic scheduling, and early stopping strategy. The optimal model is selected based on the spatial distance error of the validation set. It is then determined whether the target accuracy or application requirements have been met. If not, the process returns to the multi-stage training phase for further optimization. The data to be processed is preprocessed; then model inference is performed, which involves data input, feature processing, and outputting envelope sphere parameters. Determine whether the target accuracy or application requirements have been met. If not, return to the multi-stage training phase for further optimization; if the target accuracy has been met, end the training and obtain the interactive robot's close-range safe collision avoidance model.

[0009] Furthermore, the steps for generating structured training samples include: S1.1: Load the 3D point cloud and joint skeleton data of the human body dataset, covering human body samples of multiple body types and poses, and complete the data format standardization and noise filtering; S1.2: Define color coding rules for limb regions based on human anatomical features, configure exclusive visual identifiers for the head, torso, left arm, right arm, left leg, and right leg, and preset the basic size range of the envelope sphere for each limb region; S1.3: Analyze the initial multi-envelope sphere annotation parameters, construct a 3D sphere model and overlay it with the human body point cloud for visualization, verify whether the multi-envelope spheres completely cover the corresponding limb areas and match the actual spatial dimensions of the human body, and correct the sphere center coordinates and radius parameters that exceed the threshold; if not satisfied, return to recalibrate the envelope sphere parameters. S1.4: Extract the three-dimensional coordinate features of 19 key joints of the human body, associate and store the joint skeleton features, point cloud data and multi-envelope sphere parameters, fill in the missing parameters with the mean, and construct a triplet dataset.

[0010] Furthermore, the dual-input neural network model takes the three-dimensional skeleton coordinates of 19 key joints of the human body and the three-dimensional point cloud of the human body as input, including: S2.1: Skeleton feature extraction branch. Taking the three-dimensional coordinates of 19 joints of the human body as input, it extracts high-dimensional skeleton features through a three-layer bottleneck inverted residual structure and a normalization layer. Each bottleneck inverted residual structure includes up-dimensional convolution, depthwise separable convolution, and down-dimensional convolution. The activation function is ReLU6. The final output is the query feature vector. S2.2: Point cloud feature extraction branch receives human point cloud feature input and uses the K-nearest neighbor algorithm to extract local neighborhood features; then, it sequentially performs nonlinear spatial feature extraction and dimensionality upscaling through the first and second layers of a multilayer perceptron with GELU activation function; next, it uses adaptive average pooling to eliminate spatial dimensionality differences to aggregate global point cloud features; finally, it performs independent linear mapping on the global point cloud features through a dual mapping branch, outputting key vector K and value vector V; S2.3: Cross-modal fusion module, which adopts a dual cross-attention mechanism to fuse skeleton and point cloud features. The skeleton features are used as query vector Q. The attention weight matrix is ​​calculated through relative position encoding and then weighted and fused with the point cloud feature value vector V. The residual connection is combined to enhance the feature representation capability. S2.4: Feature enhancement module, consisting of dilated convolution and channel attention units, is used to enhance the feature response of the limb region; S2.5: Output decoding module, which maps the fused features to the center coordinates (x, y, z) and radius parameters of multiple envelope spheres of various limb regions of the human body through a lightweight fully connected layer, and configures independent weight matrices for the envelope sphere parameters of the limb regions to perform differentiated prediction of the envelope spheres of different regions.

[0011] Furthermore, in step S1.3, the correction needs to reserve a safety obstacle avoidance redundancy space of ≥5cm for the interactive robot, and the consistency of the annotation results should be ≥95%.

[0012] Furthermore, in step S2.1, both the up-dimensional convolution and down-dimensional convolution of the bottleneck inverted residual structure are configured with batch normalization layers, and the depth-separable convolution adopts a group convolution strategy.

[0013] Furthermore, in step S2.3, the weight matrix calculation of the dual cross-attention mechanism enhances the correlation matching of cross-modal features.

[0014] Furthermore, the number of parameters in the dual-input network model is ≤5M, and the computational cost is ≤3.5GFLOPs.

[0015] According to another aspect of the present invention, a near-distance safe collision avoidance model for interactive robots based on multi-envelope sphere human body region perception is provided, which is obtained by using a near-distance collision avoidance method for interactive robots based on multi-envelope sphere human body region perception as described in any of the preceding claims.

[0016] According to another aspect of the present invention, a method for near-range safe collision avoidance of interactive robots based on multi-envelope sphere human body region perception is provided. The method deploys the aforementioned near-range safe collision avoidance model of interactive robots based on multi-envelope sphere human body region perception to the interactive robot, inputs human skeleton and point cloud data collected by LiDAR and camera in real time, and outputs human body multi-envelope sphere region parameters to provide human body spatial contour basis for near-range safe collision avoidance decision of interactive robots.

[0017] According to another aspect of the present invention, an interactive robot based on multi-envelope sphere human body region perception is provided, including the aforementioned interactive robot near-distance safe collision avoidance model based on multi-envelope sphere human body region perception.

[0018] In summary, compared with the prior art, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. The core inventive concept of this invention lies in using the three-dimensional skeleton coordinates of 19 key joints of the human body and the three-dimensional point cloud of the human body as dual inputs. Combined with a multi-envelope sphere human body region parameterization method, a lightweight dual-input neural network is constructed. Furthermore, a dual-cross attention mechanism is used to achieve cross-modal fusion of skeleton structure information and point cloud surface geometry information. Based on this technical principle, this invention can more accurately represent the spatial contours of various limb regions of the human body, significantly improving the prediction accuracy of the coordinates and radius parameters of the multi-envelope sphere, thereby providing a more reliable basis for human body spatial contours for near-field safe collision avoidance in interactive robots.

[0019] 2. This invention constructs a multi-envelope sphere human body region visualization annotation module, which overlays and displays human body point clouds with envelope spheres. By combining size thresholds, coverage integrity verification, and safety redundancy reservation rules to correct annotation parameters, it can effectively improve the annotation quality and consistency of training samples, reduce model training errors caused by annotation deviations, and improve the accuracy and stability of the human body region perception model from the source.

[0020] 3. This invention targets high-frequency moving and collision-prone areas such as human limbs. It introduces a differentiated weighting mechanism into the loss function and combines it with a feature enhancement module to strengthen the feature response of the limb area. This makes the model more sensitive to perception and more accurate in predicting high-risk limb parts, which helps the robot to identify potential collision risks earlier and perform obstacle avoidance actions such as deceleration and detour in a timely manner, thereby improving safety during close-range interaction.

[0021] 4. The present invention adopts a bottleneck inverted residual structure, depthwise separable convolution and lightweight decoding design in the network structure, which effectively reduces the number of model parameters and computation while ensuring the multimodal feature fusion effect. This enables the model to adapt to the limited computing resources on the interactive robot edge, shortens the inference latency, and meets the real-time deployment requirements in near-distance safe collision avoidance scenarios.

[0022] 5. This invention constructs skeletal features, point cloud data, and multi-envelope sphere parameter triplet training samples based on human datasets. It also improves the model's adaptability to different body shapes, postures, and complex environmental interference through data augmentation, multi-stage training, and preprocessing mechanisms. Therefore, it has good generalization performance and scene expansion capabilities, and can be applied to various human-machine close interaction scenarios such as industrial collaborative robots and service robots. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the entire technical process of a preferred embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the dual-input network model architecture of a preferred embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the visual annotation of the envelope sphere according to a preferred embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram illustrating a preferred embodiment of the robot deployment application scenario of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] To address the problems of low human perception accuracy, insufficient obstacle avoidance safety, poor quality of labeled data, high deployment cost and poor real-time performance of perception models in existing robots, as well as unreasonable network structure design and poor multimodal fusion effect, this invention provides an integrated solution of "human dataset labeling - dual-modal lightweight network prediction - robot deployment". By accurately predicting the parameters of the human limb envelope sphere, it provides a reliable basis for human spatial contour for robot obstacle avoidance, ensuring the safety of personnel during human-computer interaction.

[0029] This invention provides a near-range safe collision avoidance method for interactive robots based on multi-envelope sphere human body region perception. Applied to human body perception and safe collision avoidance scenarios for interactive robots, this method constructs a lightweight human body multi-envelope sphere region perception algorithm based on multimodal input within the perception context of interactive robot operation. The method includes the following steps: S1: Construct a multi-envelope sphere human region visualization annotation module based on the human dataset to generate structured training samples: S1.1: Load the 3D point cloud and joint skeleton data of the human body dataset, covering human body samples of multiple body types and poses, and complete the data format standardization and noise filtering; S1.2: Define color coding rules for limb regions based on human anatomical features, configure exclusive visual identifiers for the head, torso, left arm, right arm, left leg, and right leg, and preset the basic size range of the envelope sphere for each limb region; S1.3: Analyze the initial multi-envelope sphere annotation parameters, construct a 3D sphere model and overlay it with the human body point cloud for visualization, verify whether the multi-envelope spheres completely cover the corresponding limb areas and match the actual spatial dimensions of the human body, and correct radius parameters that exceed the threshold; if not satisfied, return to recalibrate the envelope sphere parameters. S1.4: Extract the three-dimensional coordinate features of 19 key joints of the human body, associate and store the "joint skeleton features - point cloud data - multi-envelope sphere parameters", fill the missing parameters with mean values, and construct a triplet dataset (divided into training set, validation set and test set in an 8:1:1 ratio). The dataset construction is now complete. S2: Construct a neural network model (including feature extraction, cross-fusion, enhancement, and decoding) to predict parameters of multiple enveloped spherical regions of the human body. S2.1: Skeleton feature extraction branch. Taking the three-dimensional coordinates of 19 joints of the human body as input, it extracts high-dimensional skeleton features through 3-layer bottleneck inverted residual structure and layer normalization layer. Each bottleneck inverted residual structure includes dimension-up convolution (1×1 convolution kernel, number of channels 57→128), depthwise separable convolution (3×3 convolution kernel, stride 1), and dimension-down convolution (1×1 convolution kernel, number of channels 128→57). The activation function is ReLU6, and the output is a 256-dimensional query feature vector. S2.2: Point cloud feature extraction branch receives human point cloud feature input and uses the K-nearest neighbor algorithm to extract local neighborhood features (K=16); then, it sequentially performs nonlinear spatial feature extraction and dimensionality upscaling through the first and second layers of a multilayer perceptron with GELU activation function; next, it uses adaptive average pooling to eliminate spatial dimensionality differences to aggregate global point cloud features; finally, it uses a dual mapping branch to independently linearly map the global point cloud features, outputting a 256-dimensional key vector (K) and a 256-dimensional value vector (V). S2.3: Cross-modal fusion module, which adopts a dual cross-attention mechanism to fuse skeleton and point cloud features. The skeleton features are used as query vector (Q) and the point cloud features are used as key vector (K). The attention weight matrix is ​​calculated through relative position encoding and then weighted and fused with the point cloud feature value vector (V). The feature representation capability is enhanced by combining residual connections. S2.4: Feature enhancement module, consisting of two layers of dilated convolutions (dilation rates of 2 and 4 respectively) and channel attention units, further enhances the feature response of the limb region; S2.5: Output decoding module, which maps the fused features to the coordinates of the center of a multi-envelope sphere for each limb region of the human body through a lightweight fully connected layer. , , The parameters of radius and limb region are used to configure independent weight matrices for the envelope sphere parameters of the limb region, so as to achieve differentiated and accurate prediction. S3: Train a dual-input network model based on labeled samples and verify its accuracy. S3.1: Perform training data augmentation by batch loading human sample data containing skeletons, point clouds, and multi-envelope sphere annotations, and perform synchronous transformation, noise simulation, and oversampling balancing operations; at the same time, adopt a multi-stage training strategy to fine-tune the full model after pre-training the feature modules. S3.2: Construct a weighted spatial distance loss function, including spatial Euclidean distance loss and radius mean square error loss, wherein the spatial distance loss weight of the limb region envelope sphere is 1.8 times that of the core region, and the radius loss weight is 2 times that of the core region; S3.3: The model is trained using gradient pruning (threshold 1.0), cosine annealing learning rate dynamic scheduling and early stopping strategy (patience value 20 rounds). The optimal model is selected based on the spatial distance error of the validation set. It is then determined whether the target accuracy or application requirements have been met. If not, the process returns to the multi-stage training stage for further optimization. S3.4: Preprocess the data to be processed (convert it into a standardized skeleton / point cloud format); then perform model inference, which involves data input, feature processing, and outputting the envelope sphere parameters. S3.5: Determine whether the target accuracy or application requirements have been met. If not, return to the multi-stage training phase for further optimization; if met, output the human envelope sphere model. S4: Deploy the trained lightweight model to the interactive robot, input the human skeleton and point cloud data collected by LiDAR and camera in real time, and output the human multi-envelope sphere region parameters to provide the human spatial contour basis for the interactive robot's close-range safe collision avoidance decision.

[0030] Preferably, in step S1.3, the multi-envelope sphere parameter correction needs to reserve ≥5cm of safety obstacle avoidance redundancy space for the interactive robot, and the consistency of the annotation results is ≥95%.

[0031] Preferably, in step S2.1, both the up-dimensional convolution and down-dimensional convolution of the bottleneck inverted residual structure are configured with batch normalization layers, and the depthwise separable convolution adopts a grouped convolution strategy (number of groups = number of channels / 4) to reduce the computational load of the model.

[0032] Preferably, in step S2.3, the weight matrix calculation of the dual cross-attention mechanism enhances the correlation matching of cross-modal features.

[0033] Preferably, in step S2, the number of parameters of the lightweight dual-input network model is ≤5M and the computational cost is ≤3.5GFLOPs, which meets the computational power constraints of the interactive robot end-side.

[0034] Preferably, in step S4, the model deployment adopts an edge-side lightweight inference framework with a model inference latency of ≤50ms, which meets the real-time response requirements of the interactive robot for safe collision avoidance at close range.

[0035] Furthermore, the technical solution of this invention consists of three core components: human dataset visualization and annotation, lightweight bimodal network prediction, and robot-side deployment and application (the entire process is shown in Figure 1), as detailed below: Step 1: Visual annotation of multi-enveloping sphere human body regions This step generates high-quality envelope sphere labeled samples adapted to obstacle avoidance scenarios based on a human dataset. It loads 3D point cloud and joint skeleton data from the human dataset, filters samples covering multiple body types and poses, removes point cloud noise through statistical filtering, and unifies the data format to a standardized 3D coordinate system. Subsequently, it performs rapid limb segmentation, employing line segment constraints, distance filtering, and symmetry plane differentiation strategies to accurately separate the point clouds of each limb. Finally, it develops parameterized representations and size threshold rules for multiple envelope spheres based on robot obstacle avoidance safety standards and human anatomical features. For a single envelope sphere Its spatial geometric features are uniformly represented by four-dimensional parameter vectors ( , , ) is used to express, where ( , , Let the coordinates be the center of the sphere. The radius is the physical radius, including a safety margin of 5cm for obstacle avoidance. To determine whether the radius parameter "exceeds the threshold," this invention sets the minimum and maximum radius ranges for the envelope sphere of different body parts based on prior knowledge of human anatomy, and assigns exclusive visual identifiers to different limbs. The specific rules are shown in Table 1. Table 1: Parametric Representation and Threshold Definition Rules of Human Multi-Enveloping Sphere Region

[0036] In the envelope sphere parameter calibration stage, point cloud density is the core focus. The point cloud is analyzed sequentially, outlier removal is performed, and distribution boundaries and surface smoothing operations are corrected. The initial envelope sphere parameters are transformed into a 3D sphere model, which is then overlaid on the human body point cloud for display. Annotators use a visualization interface to determine if the envelope sphere visualization meets the requirements, focusing on verifying whether the envelope sphere coverage meets the standards, and rigorously reviewing it according to the thresholds in Table 1. If the generated envelope sphere radius is smaller than the corresponding part's "minimum radius threshold" or larger than the "maximum radius threshold," it is explicitly defined as an abnormal parameter exceeding the threshold, requiring manual correction of the sphere center coordinates or radius parameters; otherwise, the process is repeated. Finally, the 3D coordinates of 19 key joints of the human body are extracted. Skeletal features, point cloud data, and corrected envelope sphere parameters are associated and stored to construct triplet data. Missing parameters are filled with the mean, and a triplet dataset is constructed (divided into training, validation, and test sets in an 8:1:1 ratio), generating standardized training samples.

[0037] Step 2: Lightweight Bimodal Network Envelope Sphere Prediction The core innovation of this invention lies in the design of a lightweight bimodal network. Through skeleton feature extraction, cross-modal fusion, and precise loss function optimization, it achieves rapid prediction of the human body envelope sphere. The specific technical solution is as follows: 2.1 Skeleton Feature Extraction Branch The skeleton feature extraction adopts a three-layer cascaded bottleneck inverted residual structure, which balances feature representation capability and computational efficiency. Each layer includes four major steps: dimensionality increase, feature extraction, dimensionality reduction, and residual connection. First, the dimensionality of the input 57-dimensional skeleton features is increased by expanding the number of channels to 128 dimensions through 1×1 convolution. At the same time, batch normalization (BN) is introduced to avoid gradient vanishing, as shown in Equation (1): (1) in, This represents the skeleton feature vector after dimensionality increase. This indicates a batch normalization operation. This represents the initial skeleton feature vector of the input (57-dimensional, containing 19 parameters for each of the three coordinate axes). Indicates the kernel size as The convolution operation is used to increase the feature dimension from 57 to 128.

[0038] After dimensionality increase, local features are extracted through depthwise separable convolution. A 3×3 convolution kernel and 32 groups are used to reduce computation. Batch normalization is used to stabilize the training process, as shown in equation (2). (2) in, This represents the local skeleton feature vector extracted after depthwise separable convolution. Indicates the number of groups. Indicates the kernel size as A depthwise separable convolution operation with a group size of 32.

[0039] After local feature extraction, dimensionality reduction is performed. The number of channels is restored to 57 dimensions through 1×1 convolution to maintain the consistency of feature dimensions and adapt to residual connections, as shown in Equation (3): (3) in, This represents the skeleton feature output vector after residual fusion and activation. Indicates the kernel size as The convolution operation is used to convert features The dimension was reduced from 128 to 57. 。

[0040] Finally, by combining residual connections with the ReLU6 activation function, feature propagation is enhanced and gradient decay is mitigated, resulting in a 256-dimensional skeleton query feature vector, as shown in Equation (4): (4) in, This represents the skeleton feature output vector after residual fusion and activation. This represents the features after dimensionality reduction. Input features After performing element-wise addition of residual connections, a non-linear mapping operation is performed using the ReLU6 activation function.

[0041] 2.2 Point Cloud Feature Extraction Branch The point cloud feature extraction branch aims to extract features with rich geometric details and global contextual information from the 3D point cloud of the human body. Specifically, it includes steps such as local feature aggregation, deep mapping, and dual-branch output. First, the initial human point cloud features (dimension 1) are received as input. The K-Nearest Neighbors (KNN) algorithm is used to extract local geometric structure. With a neighborhood size of K=16, the features of the 16 nearest neighbors of each center point are extracted and locally concatenated and aggregated. The output dimension is... The local point cloud feature vector.

[0042] Subsequently, the local point cloud feature vector is input into the first layer of the multilayer perceptron (MLP1), and the GELU activation function is used for nonlinear feature mapping to increase the dimensionality of the feature channels, resulting in an output dimension of [dimensionality missing]. The intermediate layer features.

[0043] Next, the intermediate layer features are input into the second layer, a multilayer perceptron (MLP2), and the GELU activation function is used again for deep spatial feature extraction, further expanding the feature dimension. The output dimension is... The high-dimensional point cloud feature vector.

[0044] To extract global context information and compress computational dimensionality, an adaptive average pooling operation is applied to the high-dimensional point cloud features to eliminate spatial dimensional differences and output a global point cloud feature vector with a fixed dimension of 256.

[0045] Finally, the 256-dimensional global point cloud features are input into the Dual Mapping Branch. Through two independent linear transformations, the global point cloud features are mapped into 256-dimensional key vectors (K) and 256-dimensional value vectors (V), which are then passed to the subsequent cross-modal fusion module for attention calculation.

[0046] 2.3 Cross-modal fusion module The cross-modal fusion module employs a dual cross-attention mechanism to output skeleton feature vectors. Using the query vector (Q) and point cloud features as the key vector (K) and value vector (V), bimodal feature alignment and fusion are achieved through linear projection, positional encoding, and weight calculation. The first step is to perform linear projection on the bimodal features to obtain the skeleton features. Mapped to query vector Q, human point cloud features The mapping is done to the key vector K and the value vector V, and the projection process is shown in equation (5): (5) in, This represents the linear projection weight matrix that maps skeleton features to query vectors. This represents the corresponding query vector bias term. This represents the linear projection weight matrix that maps point cloud features to key vectors. This represents the corresponding key vector bias term. This represents the linear projection weight matrix that maps point cloud features to value vectors. This represents the corresponding value vector bias term.

[0047] To enhance the positional alignment accuracy of bimodal features, relative positional encoding is introduced. The position offset is calculated based on the spatial distance δ of the bimodal features, and the encoding formula is shown in equation (6): (6) The attention weight matrix α is calculated based on projection features and location encoding, and then... After function normalization, the sum is weighted and summed with the value vector V, and then combined with residual connections to output fused features. The complete fusion process is shown in equation (7): (7) 2.4 Feature Enhancement Module The features fused across modalities are further input into a feature enhancement module. This module consists of two layers of dilated convolutions (with dilation rates set to 2 and 4 respectively) connected in series with channel attention units. The dilated convolutions can effectively expand the receptive field and capture a wider range of limb spatial context information without increasing the number of network parameters; the channel attention units adaptively weight the feature channels to further enhance the feature responses of collision-prone areas of the limbs, such as the left arm, right arm, left leg, and right leg, and output the enhanced fused features.

[0048] 2.5 Output Decoding Module The network employs an output decoding module at the end, which maps the enhanced high-dimensional features to physical space through a lightweight fully connected layer. This module outputs the coordinates (x, y, z) of the center of the multi-enveloped sphere and its radius parameters for each limb region of the human body. Independent weight matrices are configured for the envelope sphere parameters of the core region (head, torso) and limb regions, taking into account the different ranges of motion and collision risks of each limb, to achieve differentiated and accurate parameter prediction output.

[0049] 2.6 Model Training and Loss Function Design The model training employs a multi-stage training strategy. First, the skeleton and point cloud feature extraction modules are pre-trained, followed by full model fine-tuning via the fusion and decoding modules. During the training preparation phase, triplet datasets are loaded in batches, and data augmentation operations such as synchronous transformation, noise simulation (e.g., random point cloud jitter), and oversampling balancing are performed to improve the model's robustness to complex environments.

[0050] The model optimization adopts a weighted multi-task loss function, which enhances the prediction accuracy of collision-prone parts of the limbs by weighting the spatial distance loss and radius loss. The total loss function is defined as shown in Equation (8): (8) in, For spatial Euclidean distance loss, the coordinates of the center of the 16 envelope spheres for a single limb are ( , , ) and predicted coordinates ( , , ) Calculation error As shown in equation (9): (9) The radius mean square error loss is calculated for the actual radius of 66 envelope spheres (16 for each of the four limbs and 16 for the trunk and head). Error in predicting radius calculation As shown in equation (10): (10) To enhance the prediction accuracy of limbs, a regional weighting coefficient is set. and The weights of the left arm, right arm, left leg, and right leg are higher than those of the head and torso, as specifically defined in equation (11):

[0051] (11) 1.0, 1.8, and 2.0 are preferred empirical values ​​in this embodiment.

[0052] In terms of regularization and training monitoring, the AdamW optimizer (initial learning rate 5e-5, weight decay 1e-5) is used, combined with cosine annealing learning rate scheduling, gradient pruning and early stopping strategies to ensure model convergence stability and avoid overfitting.

[0053] During the inference and validation phases, the validation / test set data is preprocessed and uniformly transformed into standardized skeleton and point cloud formats. Then, model inference is executed, with the data sequentially processed through input and feature processing, ultimately outputting the predicted envelope sphere parameters. The error between the predicted and true values ​​is calculated to determine if the target accuracy or practical application requirements have been met. If not, the process returns to the multi-stage training phase to adjust hyperparameters and continue optimization. If the target accuracy is met, the final human envelope sphere model is output.

[0054] Step 3: Deployment and Application of Robots This step involves deploying the trained lightweight model to the interactive robot control system (scenario as attached). Figure 4 As shown in the figure, it enables human body area perception and close-range safe collision avoidance landing.

[0055] Leveraging the advantages of lightweight models, the optimized model is deployed to the embedded control system of the interactive robot. It is loaded through a lightweight inference framework, and combined with batch inference and hardware acceleration optimization, ensuring that the inference latency is stable within 50ms.

[0056] In the close-range safe collision avoidance execution process, the interactive robot synchronously collects human point cloud and image data through LiDAR (sampling frequency 10Hz) and camera (frame rate 30fps), and extracts 19 joint skeleton features in real time; inputs the dual-modal features into the lightweight model, outputs multi-envelope sphere parameters of each limb region of the human body and constructs the spatial contour of the human body region; the interactive robot control system calculates the minimum distance between the motion trajectory and the human body, and when the distance is less than the 5cm safety threshold, it triggers active deceleration (deceleration ratio ≥75%) or turns around to achieve close-range safe collision avoidance through the actuator.

[0057] This invention utilizes a dual-cross attention mechanism to deeply align the skeletal structure with point cloud surface features. Combined with multi-objective weighted joint optimization of spatial distance and radius, it effectively overcomes the contour blurring caused by a single perceptual modality. This results in an average absolute error of ≤0.5cm for spatial distance prediction and ≤0.3cm for radius prediction in the human limb region using multi-envelope spheres. For high-frequency moving and collision-prone areas such as limbs, this invention applies differentiated penalty weights to the loss layer and introduces safety redundancy annotations, forcing the model to remain highly sensitive to high-risk areas. This allows the interactive robot to perceive collision risks more than 50ms in advance, achieving an obstacle avoidance success rate of ≥99.5%. By employing bottleneck inverted residuals and depthwise separable convolutions to significantly reduce redundant feature map computations, this "lightweight + dual-modal cross + dilated convolution" combination design ensures effective feature fusion while compressing the model parameters to ≤5M, computational cost to ≤3.5G FLOPs, and inference latency to ≤50ms (a reduction of over 70% in parameters and a 15%-20% improvement in prediction accuracy compared to ordinary dual-modal models). It breaks through the bottleneck of edge computing power and does not rely on high-performance hardware. In addition, relying on the visualization annotation module, it processes public human datasets through rule constraints and mean filling, eliminating the complex physical scene collection process (reducing the cost of dataset construction by more than 80%), and naturally covers multiple body poses to ensure the model's generalization ability. The overall architecture decouples the perception and decision-making links. Only the annotation safety threshold (3-10cm) and model inference latency threshold (30-100ms) need to be fine-tuned to quickly adapt to the close-range safety collision avoidance requirements of interactive robots in different scenarios such as industry, service, and medical care, with extremely strong scenario scalability.

[0058] Example 1: Visual annotation of human body dataset Load human point cloud and skeleton data from a publicly available human dataset, and select 1000 human samples covering different body types (height 150-190cm) and postures (standing, walking, bending over, waving). Remove point cloud noise using statistical filtering (neighborhood point count threshold 10), and unify the coordinate system to the robot's base coordinate system. In accordance with industrial collaborative robot obstacle avoidance safety standards, define annotation rules: head envelope radius 8-10cm, torso 15-20cm, limbs 5-8cm. Assign white, purple, red, green, blue, and yellow visual identifiers to the head, torso, left arm, right arm, left leg, and right leg, respectively. The initial envelope sphere parameters were converted into a 3D sphere model and overlaid with the human body point cloud. The annotators found that the radius of the left arm envelope sphere was only 5cm, which did not cover the entire arm point cloud and had no safety redundancy. The radius was corrected to 10cm, and the coordinates of the sphere center were fine-tuned to the anatomical center of the arm (x=0.32m, y=0.15m, z=1.12m). The 3D coordinates of 19 joints of the human body were extracted, and the missing coordinates of the left wrist joint were filled with the average values ​​of the four limb joints (x=0.41m, y=0.18m, z=0.95m). The skeletal features, point cloud data, and corrected envelope sphere parameters were stored together and divided into a training set (5600 data points), a validation set (700 data points), and a test set (700 data points) in an 8:1:1 ratio.

[0059] Example 2: Lightweight Bimodal Network Training A point cloud skeleton dual-input network model was constructed, with a 3-layer bottleneck inverted residual structure in the skeleton branches (channel count 64→128→64). The point cloud branches were downsampled to 1024 points + 4 layers of depthwise separable convolutions. A dual cross-attention fusion module outputs 512-dimensional features, and the feature enhancement unit contains 2 layers of dilated convolutions. The output decoding module is split into two branches. The hardware environment is a GPU (NVIDIA RTX 3090), with a batch size of 8 and 100 training epochs. The learning rate is linearly increased to 5e-5 in the first 50 epochs, and then cosine annealing is used. The model converged after 70 epochs. The validation set spatial distance MAE is 4.42cm, the radius MAE is 0.28cm, and the spatial distance MAE of the limb regions is 0.45cm. The model has 1.6M parameters, which meets the lightweight requirements.

[0060] Example 3: Robot Deployment and Obstacle Avoidance Application The trained lightweight model was deployed to the embedded control system (2TOPS computing power) of the industrial collaborative interactive robot. The lightweight inference framework was used for loading, and combined with hardware acceleration optimization, the model inference latency was stabilized at 38-45ms, which met the requirements for close-range safe collision avoidance response.

[0061] The close-range safe collision avoidance test was conducted within a 3m×3m working area of ​​the interactive robot. The robot simultaneously acquired 3D point cloud and image data of the human body using a LiDAR (sampling frequency 10Hz) and a depth camera (frame rate 30fps), extracting skeletal features of 19 key joints in real time. After inputting the bimodal features into a lightweight model, the model quickly outputs multi-envelope sphere parameters for each limb region, accurately constructing the spatial contour of the human body. When the distance between the human body and the interactive robot is less than 5cm, active deceleration (deceleration ratio ≥75%) or turning to detour is triggered, achieving close-range safe collision avoidance with a success rate ≥99.5%.

[0062] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A near-range collision avoidance method for interactive robots based on multi-envelope spherical human body region perception, characterized in that, include: A multi-envelope spherical human region visualization annotation module was constructed based on the human dataset to generate structured training samples: A dual-input neural network model is constructed for parameter prediction of the human body's multi-envelope sphere region. The dual inputs are the three-dimensional skeleton coordinates of 19 key joints of the human body and the three-dimensional point cloud of the human body. Training a dual-input network model based on labeled structured training samples includes: Training data augmentation is performed by batch loading human sample data containing skeletons, point clouds, and multi-envelope sphere annotations, and synchronous transformation, noise simulation, and oversampling balancing operations are executed; at the same time, a multi-stage training strategy is adopted, and the full model is fine-tuned after pre-training the feature modules. Construct a weighted spatial distance loss function, which includes spatial Euclidean distance loss and radius mean square error loss. The spatial distance loss weight of the limb region envelope sphere is a times that of the core region, and the radius loss weight is b times that of the core region, where b>a>1. A dual-input network model is trained using gradient pruning, cosine annealing learning rate dynamic scheduling, and early stopping strategy. The optimal model is selected based on the spatial distance error of the validation set. It is then determined whether the target accuracy or application requirements have been met. If not, the process returns to the multi-stage training phase for further optimization. The data to be processed is preprocessed; then model inference is performed, which involves data input, feature processing, and outputting envelope sphere parameters. Determine whether the target accuracy or application requirements have been met. If not, return to the multi-stage training phase for further optimization; if the target accuracy has been met, end the training and obtain the interactive robot's close-range safe collision avoidance model.

2. The near-distance collision avoidance method for interactive robots based on multi-envelope sphere human body region perception as described in claim 1, characterized in that, The steps to generate structured training samples include: S1.1: Load the 3D point cloud and joint skeleton data of the human body dataset, covering human body samples of multiple body types and poses, and complete the data format standardization and noise filtering; S1.2: Define color coding rules for limb regions based on human anatomical features, configure exclusive visual identifiers for the head, torso, left arm, right arm, left leg, and right leg, and preset the basic size range of the envelope sphere for each limb region; S1.3: Analyze the initial multi-envelope sphere annotation parameters, construct a 3D sphere model and overlay it with the human body point cloud for visualization, verify whether the multi-envelope spheres completely cover the corresponding limb areas and match the actual spatial dimensions of the human body, and correct the sphere center coordinates and radius parameters that exceed the threshold; if not satisfied, return to recalibrate the envelope sphere parameters. S1.4: Extract the three-dimensional coordinate features of 19 key joints of the human body, associate and store the joint skeleton features, point cloud data and multi-envelope sphere parameters, fill in the missing parameters with the mean, and construct a triplet dataset.

3. A near-distance collision avoidance method for interactive robots based on multi-envelope sphere human body region perception as described in claim 1 or 2, characterized in that, The dual-input neural network model takes the three-dimensional skeleton coordinates of 19 key joints of the human body and the three-dimensional point cloud of the human body as input, including: S2.1: Skeleton feature extraction branch. Taking the three-dimensional coordinates of 19 joints of the human body as input, it extracts high-dimensional skeleton features through a three-layer bottleneck inverted residual structure and a normalization layer. Each bottleneck inverted residual structure includes up-dimensional convolution, depthwise separable convolution, and down-dimensional convolution. The activation function is ReLU6. The final output is the query feature vector. S2.2: Point cloud feature extraction branch receives human point cloud feature input and uses the K-nearest neighbor algorithm to extract local neighborhood features; then, it sequentially performs nonlinear spatial feature extraction and dimensionality upscaling through the first and second layers of a multilayer perceptron with GELU activation function; next, it uses adaptive average pooling to eliminate spatial dimensionality differences to aggregate global point cloud features; finally, it performs independent linear mapping on the global point cloud features through a dual mapping branch, outputting key vector K and value vector V; S2.3: Cross-modal fusion module, which adopts a dual cross-attention mechanism to fuse skeleton and point cloud features. The skeleton features are used as query vector Q. The attention weight matrix is ​​calculated through relative position encoding and then weighted and fused with the point cloud feature value vector V. The residual connection is combined to enhance the feature representation capability. S2.4: Feature enhancement module, consisting of dilated convolution and channel attention units, is used to enhance the feature response of the limb region; S2.5: Output decoding module, which maps the fused features to the center coordinates (x, y, z) and radius parameters of multiple envelope spheres of various limb regions of the human body through a lightweight fully connected layer, and configures independent weight matrices for the envelope sphere parameters of the limb regions to perform differentiated prediction of the envelope spheres of different regions.

4. The near-distance collision avoidance method for interactive robots based on multi-envelope spherical human body region perception as described in claim 2, characterized in that, In step S1.3, the correction needs to reserve a safety obstacle avoidance redundancy space of ≥5cm for the interactive robot, and the consistency of the annotation results should be ≥95%.

5. The near-distance collision avoidance method for interactive robots based on multi-envelope sphere human body region perception as described in claim 3, characterized in that, In step S2.1, both the up-dimensional convolution and down-dimensional convolution of the bottleneck inverted residual structure are configured with batch normalization layers, and the depth-separable convolution adopts a group convolution strategy.

6. The near-distance collision avoidance method for interactive robots based on multi-envelope sphere human body region perception as described in claim 3, characterized in that, In step S2.3, the weight matrix calculation of the dual cross-attention mechanism enhances the correlation matching of cross-modal features.

7. The near-distance collision avoidance method for interactive robots based on multi-envelope sphere human body region perception as described in claim 1, characterized in that, The dual-input network model has ≤5M parameters and ≤3.5GFLOPs computation.

8. A near-range safe collision avoidance model for interactive robots based on multi-envelope spherical human body region perception, characterized in that, The method for near-distance collision avoidance of an interactive robot based on multi-envelope sphere human body region perception, as described in any one of claims 1 to 7, is obtained.

9. A method for near-range safe collision avoidance of interactive robots based on multi-envelope spherical human body region perception, characterized in that, The interactive robot's near-range safety collision avoidance model based on multi-envelope sphere human body region perception, as described in claim 8, is deployed to the interactive robot. The human skeleton and point cloud data collected by LiDAR and camera are input in real time, and the human multi-envelope sphere region parameters are output, providing the human spatial contour basis for the interactive robot's near-range safety collision avoidance decision.

10. An interactive robot based on multi-envelope spherical human body region perception, characterized in that, This includes the near-distance safe collision avoidance model for interactive robots based on multi-envelope sphere human body region perception as described in claim 8.

Images