A hip joint angle prediction method based on environment perception fusion

Abstract

The application discloses a hip joint angle prediction method based on environment perception fusion, and relates to the fields of artificial intelligence and rehabilitation robots. The method first acquires a depth image and kinematics data, extracts spatial statistical features of the depth image, performs non-uniform segmentation on the depth image, obtains a double-vision feature sequence through sinusoidal position encoding and physical position encoding, fuses the double-vision feature sequence with the kinematics data respectively, generates two angle prediction sequences, and finally performs environment perception fusion to output a hip joint angle. Compared with the prior art, the application has the advantages of adaptive fusion of multiple position encodings, can accurately predict the hip joint angle under variable terrains, and has strong adaptability to environment changes.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and rehabilitation robots, and specifically to a method for predicting hip joint angles based on environmental perception fusion. Background Technology

[0002] Hip angle prediction is a crucial prerequisite for on-demand assistance in lower limb rehabilitation exoskeleton robots. Visual measurement schemes based on depth cameras are widely used due to their cost-effectiveness and ease of deployment. These methods typically segment depth images longitudinally, add positional encoding, generate feature sequences, and then output joint angles using a temporal prediction model. However, in real-world scenarios, depth images often suffer from noise and data loss due to terrain changes, leading to insufficient reliability of visual features and severely weakening the accuracy and robustness of hip angle prediction.

[0003] To improve the adaptability of prediction methods to the environment, existing studies have attempted to perform multi-source fusion, forming multiple prediction sequences from different angles and then weighting the output. However, such methods usually adopt static presets for fusion weights, and cannot dynamically adjust the fusion weights of different information sources according to environmental changes. The improvement in accuracy in flat terrain is limited, and the robustness in abrupt terrain is insufficient, lacking the ability to adapt to complex environments. Summary of the Invention

[0004] This invention proposes a hip joint angle prediction method based on environmental perception fusion, aiming to improve prediction accuracy and adaptability under varying terrain by adaptively fusing multiple position codes. First, the subject's forward-looking terrain image and kinematic data are acquired, and data preprocessing is performed to extract spatial statistical features from the images. The processed depth image is then segmented longitudinally. Two sets of angle prediction sequences are generated using sinusoidal position coding and physical position coding. Finally, environmental perception fusion is performed to output the hip joint angle. The method is implemented through the following technical solutions:

[0005] Step 1: Acquire multimodal data and perform data preprocessing.

[0006] Step 1.1: Normalize the depth images acquired by the depth camera.

[0007] Step 1.2: Collect kinematic data, perform low-pass filtering on the kinematic data, and take the maximum hip extension as the gait event.

[0008] Step 1.3: Segment the continuous kinematic data into individual strides based on gait events, and resample each stride to a preset time length.

[0009] Step 1.4: Segment each stride using a fixed time window, project it linearly onto a preset dimension, and obtain a kinematic feature sequence.

[0010] Step 2: Extract spatial statistical features from the depth image.

[0011] Step 2.1: Divide the depth image from top to bottom into an upper non-interest area and a lower interest area. Then, divide the lower interest area from top to bottom into a reference area, a middle area, and a near area.

[0012] Step 2.2: Extract the gradient mean of the depth image Local variance and depth difference The specific expression is as follows:

[0013] ,

[0014] ,

[0015] ,

[0016] in, and Representing pixels The horizontal and vertical gradient values ​​at the location, It is the number of pixels in the lower area of ​​interest. This represents the total number of sliding windows in the lower attention area. Representing the A sliding window, It is the first The number of pixels within a window Represents pixels Depth value at that location, For the first The average depth within each window It is the average depth of the near region. This represents the average depth of the reference area.

[0017] Step 3: Segment the depth image vertically.

[0018] Step 3.1: Divide the depth image vertically into multiple strips using a preset exponential function, and assign importance weights to each strip:

[0019] ,

[0020] in, For the strip index, counted from the top down of the image, It is the number of target stripes. This represents the preset segmentation density parameter. This represents the importance weight of each strip position.

[0021] Step 3.2: Determine the segmentation boundary so that the strip height in the bottom region is smaller than that in the top region, thereby obtaining higher spatial resolution.

[0022] Step 4: Generate two sets of angle prediction sequences.

[0023] Step 4.1: Based on the physical center height of each strip Generate sinusoidal position code This is fused with visual feature vectors to form a sinusoidal visual feature sequence:

[0024] ,

[0025] ,

[0026] in, The total dimension for location encoding. It is a frequency index.

[0027] Step 4.2: Based on the physical coordinates of each strip in the camera coordinate system Fourier feature encoding is used to generate physical location codes. This is fused with visual feature vectors to form a sequence of physical visual features:

[0028] ,

[0029] in, It is the Gaussian projection matrix.

[0030] Step 4.3: The dual visual feature sequences are fused with the kinematic feature sequences respectively, and each outputs an angle prediction sequence.

[0031] Step 5: Perform environmental perception fusion and output the hip joint angle.

[0032] Step 5.1: Calculate the instantaneous environment confidence of the current depth image.

[0033] Step 5.2: Calculate the cumulative environmental confidence level Secondary smoothing is performed to obtain the smoothed environment confidence score. Then, it is converted into dynamic fusion weights through linear mapping. :

[0034] ,

[0035] ,

[0036] ,

[0037] in, The confidence level of the smooth environment at the previous time step. This represents the instantaneous environmental confidence level of the current depth image. It is a smoothing factor. Based on basic trust weights, This represents the confidence gain coefficient.

[0038] Step 5.3: In the inference phase, predict the hip joint angle online. :

[0039] ,

[0040] in, This is a sinusoidal hip joint angle sequence. It is a sequence of physical hip joint angles.

[0041] This invention adaptively generates dynamic fusion weights by extracting spatial statistical features from depth images. On flat terrain, sinusoidal position coding can improve prediction accuracy, while on abrupt terrain, physical position coding can enhance prediction robustness. This achieves all-terrain adaptation that cannot be achieved by a single coding method and is suitable for real-time deployment on lower limb rehabilitation exoskeletons. Attached Figure Description

[0042] Figure 1 This is an overall flowchart of an embodiment of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0044] Figure 1 This is a flowchart of an embodiment. This embodiment provides a hip joint angle prediction method based on environmental perception fusion. The specific process includes: acquiring multimodal data and performing data preprocessing; extracting spatial statistical features of the depth image; segmenting the depth image along the longitudinal direction based on a preset exponential function; then outputting two sets of visual feature sequences through sinusoidal position encoding and physical position encoding respectively, generating two sets of angle prediction sequences; and finally performing environmental perception fusion to output the hip joint angle.

[0045] A method for predicting hip joint angles based on environmental perception fusion is implemented according to the following steps:

[0046] Step 1: Acquire multimodal data and perform data preprocessing.

[0047] Step 1.1: Preprocess the depth image acquired by the depth camera fixed to the subject's chest and scale it to 224×224 pixels.

[0048] Step 1.2: Kinematic data is collected using inertial measurement units located in the pelvis and bilateral thighs of the lower limbs. The kinematic data is then low-pass filtered, and the maximum hip extension is taken as the gait event.

[0049] Step 1.3: Segment the continuous kinematic data into individual strides based on gait events, and resample each stride to 100 time points.

[0050] Step 1.4: Segment each stride using a fixed time window, and then linearly project it into a 256-dimensional feature space to obtain the kinematic feature sequence.

[0051] Step 2: Extract spatial statistical features from the depth image.

[0052] Step 2.1: Divide the depth image from top to bottom into an upper non-interest area (40%) and a lower interest area (60%). The lower interest area is further divided from top to bottom into a reference area (5%), a middle area (50%), and a near area (5%).

[0053] Step 2.2: Extract the gradient mean Local variance and depth difference Spatial statistical features of three depth images:

[0054] ,

[0055] ,

[0056] ,

[0057] in, and Representing pixels The horizontal and vertical gradient values ​​at the location, It is the number of pixels in the lower area of ​​interest. This represents the total number of sliding windows in the lower attention area. Representing the A sliding window, It is the first The number of pixels within a window Represents pixels Depth value at that location, For the first Mean depth within each window It is the average depth of the near region. This represents the average depth of the reference area.

[0058] Step 3: Segment the depth image vertically.

[0059] Step 3.1: Divide the depth image vertically into multiple strips using a preset exponential function, and assign importance weights to each strip:

[0060] ,

[0061] in, For the strip index, counted from the top down of the image, It is the number of target stripes. This represents the preset segmentation density parameter. This represents the importance weight of each strip position.

[0062] Step 3.2: Determine the segmentation boundary so that the strip height in the bottom region is smaller than that in the top region, thereby obtaining higher spatial resolution.

[0063] Step 4: Generate two sets of angle prediction sequences.

[0064] Step 4.1: Based on the physical center height of each strip Generate sinusoidal position code This is fused with visual feature vectors to form a sinusoidal visual feature sequence:

[0065] ,

[0066] ,

[0067] in, The total dimension for location encoding. It is a frequency index. .

[0068] Step 4.2: Based on the physical coordinates of each strip in the camera coordinate system Fourier feature encoding is used to generate physical location codes. This is fused with visual feature vectors to form a sequence of physical visual features:

[0069] ,

[0070] in, It is the Gaussian projection matrix.

[0071] Step 4.3: After fusing the dual visual feature sequences with the kinematic feature sequences, two sets of angle prediction sequences are output.

[0072] Step 5: Perform environmental perception fusion and output the hip joint angle.

[0073] Step 5.1: Calculate the instantaneous environment confidence of the current depth image.

[0074] Step 5.2: Calculate the cumulative environmental confidence level Secondary smoothing is performed to obtain the smoothed environment confidence score. Then, it is converted into dynamic fusion weights through linear mapping. :

[0075] ,

[0076] ,

[0077] ,

[0078] in, The confidence level of the smooth environment at the previous time step. This represents the instantaneous environmental confidence level of the current depth image. It is a smoothing factor. Based on basic trust weights, This represents the confidence gain coefficient.

[0079] Step 5.3: In the inference phase, predict the hip joint angle online. :

[0080] ,

[0081] in, This is a sinusoidal hip joint angle sequence. It is a sequence of physical hip joint angles.

Claims

1. A method for predicting hip joint angles based on environmental perception fusion, characterized in that, Includes the following steps: Step 1: Acquire multimodal data and perform data preprocessing; Step 2: Extract spatial statistical features from the depth image; Step 3: Segment the depth image vertically; Step 4: Generate two sets of angle prediction sequences; Step 4.1: Based on the physical center height of each strip Generate sinusoidal position code This is fused with visual feature vectors to form a sinusoidal visual feature sequence: ， ， in, The total dimension for location encoding. It is a frequency index; Step 4.2: Based on the physical coordinates of each strip in the camera coordinate system Fourier feature encoding is used to generate physical location codes. This is fused with visual feature vectors to form a sequence of physical visual features: ， in, It is the Gaussian projection matrix; Step 4.3: Output two sets of angle prediction sequences; Step 5: Perform environmental perception fusion and output the hip joint angle.

2. The hip joint angle prediction method based on environmental perception fusion according to claim 1, characterized in that, The environmental perception fusion described in step 5 is implemented in the following steps: Step 5.1: Calculate the instantaneous environment confidence of the current depth image; Step 5.2: Calculate the cumulative environmental confidence level Secondary smoothing is performed to obtain the smoothed environment confidence score. Then, it is converted into dynamic fusion weights through linear mapping. : ， ， ， in, The confidence level of the smooth environment at the previous time step. This represents the instantaneous environmental confidence level of the current depth image. It is a smoothing factor. Based on basic trust weights, Represents the confidence gain coefficient; Step 5.3: In the inference phase, predict the hip joint angle online. : ， in, This is a sinusoidal hip joint angle sequence. It is a sequence of physical hip joint angles.

Images