Unmanned aerial vehicle perspective gait analysis method and medical auxiliary system

Abstract

The application belongs to the field of medical rehabilitation equipment and computer vision, and discloses a UAV visual angle gait analysis method and a medical auxiliary system. The method collects continuous gait videos through a UAV from a following visual angle behind a target person, sequentially inputs each frame of image into a spatial feature encoder and an auxiliary segmentation decoder, combines symmetric decoupling, bidirectional time sequence modeling, multi-task prediction, multi-task joint loss calculation and key point coordinate normalization, finally inputs a continuous image sequence, a normalized leg region segmentation label, left and right foot gait phase labels and left and right heel coordinate labels into a network, performs multi-task joint training and reasoning in an end-to-end manner, and outputs gait analysis results. The method is applied to multi-task collaborative analysis of phase recognition, leg segmentation and heel key point positioning of patient gait in a downward visual angle in a medical institution, and is applied to gait evaluation and rehabilitation monitoring of diseases such as Parkinson's disease, stroke and orthopedic postoperative rehabilitation.

Description

Technical Field

[0001] This invention relates to the fields of medical rehabilitation equipment and computer vision, specifically to a gait analysis method and medical assistance system based on a symmetric bidirectional state space model. It further relates to a multi-task collaborative analysis method for phase recognition, leg segmentation, and heel key point localization of patient gait from a top-down perspective within medical institutions, applicable to gait assessment and rehabilitation monitoring for diseases such as Parkinson's disease, stroke, and post-orthopedic rehabilitation. Background Technology

[0002] Gait is the outward manifestation of the comprehensive function of the human neuromuscular and skeletal system, encompassing key parameters such as gait frequency, stride length, stance / swing ratio, left-right symmetry, and heel strike. In clinical settings, gait abnormalities are widely used for the assessment and follow-up of diseases / conditions such as Parkinson's disease, post-stroke hemiplegia, spinal cord injury, post-hip and knee joint surgery rehabilitation, lower limb fracture recovery, and fall risk assessment in the elderly. In recent years, gait has been considered a "digital biomarker," and its quantitative detection and long-term tracking can facilitate objective assessment, efficacy comparison, and individualized rehabilitation guidance.

[0003] Existing medical gait detection methods mainly include: (1) Optical motion capture systems and force tables, which can provide high-precision three-dimensional kinematic / dynamic data, but the equipment is expensive, the layout is complex, and it is highly dependent on the site and calibration, making it difficult to be routinely applied in general wards, community clinics and home rehabilitation; (2) Wearable inertial navigation / pressure sensors, which are relatively flexible in deployment, but there are problems such as insufficient wearing compliance, sensor drift, cross-brand consistency and disinfection management; (3) Gait analysis based on monocular / multi-view video, which has the advantages of non-contact and low cost, but in real medical environments it is often affected by occlusion, changes in lighting, differences in clothing, changes in viewing angle and interference from multiple people, and it puts forward higher requirements for privacy protection and auditability.

[0004] In practical applications of medical-grade gait analysis, clinical diagnosis and rehabilitation assessment place extremely high demands on the objectivity and interpretability of data. Unlike consumer applications, gait analysis systems in medical settings must possess extremely high stability and traceability of evidence: the output indicators need to directly correspond to the rehabilitation process or fall risk, rather than vague predictive probabilities. However, existing general-purpose computer vision methods often fail to directly meet these stringent clinical standards. Methods relying solely on keypoint localization are prone to drift when limbs are mutually occluded, leading to measurement failure; while simple classification tasks can determine the state, they cannot provide intuitive kinematic spatial evidence. This single-dimensional analysis approach often lacks sufficient robustness and reliability when facing complex pathological gait, making it difficult to serve as an effective basis for auxiliary diagnosis.

[0005] In addition, the common top-down / top-down / side-down view acquisition modes in medical institutions (such as corridor ceiling cameras, rehabilitation training room cameras, and mobile platforms in specific scenarios) have advantages such as less occlusion and greater privacy protection (weak identity features), but they also bring new technical challenges: the left and right lower limbs are mirror-symmetrical in the top-down back view, the leg boundaries and heel areas are small in scale and easily occlude each other; gait events (such as heel strike and lift-off) often require the use of contextual information from several frames before and after the event, and unidirectional temporal models can only use "past information" and are insufficient in distinguishing the event boundaries; while global attention models such as Transformer have strong modeling capabilities, their computational complexity is high, which is not conducive to real-time deployment on medical edge devices / low-power platforms.

[0006] Based on the aforementioned clinical needs and engineering constraints, there is an urgent need for a gait analysis method oriented towards a top-down perspective that can output clinically relevant elements end-to-end: ensuring interpretable output while maintaining real-time performance and stability. To this end, this invention proposes a multi-task gait analysis framework based on a symmetric bidirectional state-space model (SymMamba). Utilizing a left-right symmetric decoupling strategy to fit the characteristics of the top-down structure, the bidirectional state-space model fuses information from previous and subsequent frames while maintaining linear complexity. Through collaborative learning of segmentation, phase recognition, and heel keypoint localization, it outputs intermediate evidence and indicators that can be used for clinical quantification, thereby better supporting the requirements of medical device products in terms of performance verification, risk control, and clinical usability. Summary of the Invention

[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a gait analysis method and medical assistance system based on an unmanned aerial vehicle (UAV) perspective. Its purpose is to conduct multi-task collaborative analysis of patient gait phase recognition, leg segmentation, and heel key point localization from a top-down perspective within medical institutions. This method can be applied to gait assessment and rehabilitation monitoring for diseases such as Parkinson's disease, stroke, and post-orthopedic rehabilitation. This solves the technical problem that existing methods lack sufficient robustness and reliability, making them unsuitable as effective diagnostic aids.

[0008] To achieve the above objectives, according to one aspect of the present invention, a method for gait analysis from the perspective of a UAV is provided, comprising the following steps: S1: Control the drone to capture continuous gait video from the perspective of following the target person from behind, extract continuous image frames from the video, crop the target area containing the lower limbs and uniformly scale it, and form a continuous image sequence according to the preset sequence length and sliding step length. S2 Spatial Feature Encoder: The images of each frame obtained in step S1 are sequentially input into the spatial feature encoder. After convolution downsampling, ConvNeXt V2 feature extraction and bidirectional visual state spatial feature extraction stages, the spatial feature map is output. S3 Coordinate Attention Enhancement: After the ConvNeXt V2 feature extraction stage in step S2, a coordinate attention module is introduced. The coordinate attention module performs one-dimensional global pooling along the horizontal and vertical directions respectively, encodes the features in the two directions and separates them to generate horizontal attention weights and vertical attention weights, and recalibrates the spatial features of the corresponding stage. S4 Auxiliary Segmentation Decoder: The spatial feature map output from step S2 and recalibrated by coordinate attention in step S3 is input into the auxiliary segmentation decoder. After passing through a convolutional layer with a kernel size of 1×1, the unnormalized segmentation score of the leg region with the same size as the input image is output. The unnormalized segmentation score is converted into a leg region probability mask using a sigmoid activation function, and the leg region segmentation result is obtained based on the probability mask. S5 Symmetric Decoupling Module: Based on the leg region probability mask obtained in step S4, calculate its weighted centroid along the horizontal direction, and use the centroid as the adaptive left and right boundary position of the target region; generate complementary left and right soft masks according to the boundary position, and apply them to the spatial feature map output in step S2 respectively; then, in step S3, use spatial attention pooling to aggregate the left and right features into left and right feature vectors respectively, and calculate the difference between the two to obtain the left and right difference feature vectors; S6 Bidirectional Temporal Modeling Module: Input the left feature vector sequence, right feature vector sequence, and left-right difference feature vector sequence obtained in step S5 into three independent state-space temporal modeling modules to obtain left temporal features, right temporal features, and left-right difference temporal features; After applying adaptive symmetric constraints to the left and right temporal features, the three temporal features are concatenated and temporal fusion is performed. S7 Multi-task Prediction: Based on the left-side temporal features, right-side temporal features, and left-right difference temporal features obtained and fused in step S6, left and right gait phase prediction and left and right heel coordinate prediction are performed to obtain the left and right gait phase prediction results and left and right heel coordinates. Among them, the left and right foot swing phase is marked as 1 and the support phase is marked as 0. The moment when the gait phase changes from 0 to 1 is determined as the take-off event moment, and the moment when the gait phase changes from 1 to 0 is determined as the landing event moment. The leg region segmentation results, left and right gait phase prediction results, left and right gait event moments, and left and right heel coordinates obtained in step S4 are output as gait analysis results. S8 Multi-task Joint Loss Calculation: Based on the leg region segmentation results obtained in step S4 and the left and right gait phase prediction results and left and right heel coordinates output in step S7, the segmentation loss, gait phase loss and key point coordinate regression loss are calculated respectively, and the weighted sum of each loss is obtained to obtain the multi-task joint loss function; further, temporal smoothing constraints are applied to the key point coordinates and gait phase of adjacent frames. S9 Keypoint Coordinate Normalization: Based on the leg region segmentation results obtained in step S4, the leg region contour is determined, and a leg region bounding box is constructed based on the minimum, maximum, minimum, and maximum x and y coordinates of each pixel in the contour. The left and right heel coordinates obtained in step S7 are normalized using the upper left corner of the bounding box as the origin and the width and height of the bounding box as the normalization scale. During the training phase, the leg region segmentation annotations, left and right gait phase annotations, and normalized left and right heel coordinate annotations are used as supervision labels for multi-task joint training. S10 Data Processing and Training Strategy: Arrange consecutive image frames in chronological order, construct training samples using a sliding window with a preset number of frames, and slide adjacent windows according to a preset step size; During the training phase, the continuous image sequence obtained in step S1 is input into the network, and the leg region segmentation label, the left and right gait phase labels, and the normalized left and right heel coordinate labels after step S9 are used as supervision labels for multi-task joint training; during the inference phase, the continuous image sequence is input into the trained network, and the gait analysis results are output.

[0009] Preferably, the continuous image sequence input in step S1 is:

[0010] in, For batch size, For sequence length, and This represents the height and width of the image.

[0011] Preferably, the spatial feature encoder in step S3 includes a convolutional downsampling layer, a first ConvNeXt V2 feature extraction stage, a first coordinate attention module, a downsampling layer, a second ConvNeXt V2 feature extraction stage, a second coordinate attention module, a downsampling layer, and a bidirectional visual state spatial feature extraction stage connected in sequence, and the scale of the output spatial feature map is one-sixteenth of the original image size.

[0012] Preferably, the auxiliary segmentation decoder in step S4 includes two levels of 4x bilinear upsampling, convolutional layers, batch normalization layers, and activation layers, with the following specific structure:

[0013]

[0014] in, This represents transposed convolution. Indicates group normalization, It is leg predictive coding. These are bottleneck layer features; the calculation methods for the remaining layers use... arrive The code indicates that the Decoder is used to restore the extracted abstract features to pixel-level leg region prediction results; GELU is a Gaussian error linear unit used to perform smooth nonlinear transformation on the features, enabling the network to learn complex gait features.

[0015] Preferably, the left feature vector in step S5 Represented as:

[0016] in, It is the left feature map. It is a left eigenvector. For batch size, d is the sequence length, and d is the dimension of the feature vector; GAP is global average pooling, which is used to average the spatial positions of each channel of the feature map and summarize the leg region features into a vector. Right eigenvector Represented as:

[0017] in, It is the left feature map. It is a left eigenvector; Left and right difference feature vectors Represented as:

[0018] in, These are the left and right difference feature vectors.

[0019] Preferably, in step S6, the left-side temporal features, right-side temporal features, and left-right difference temporal features are represented using independent bidirectional Mamba modules as follows:

[0020]

[0021]

[0022] in, This represents the left-side temporal characteristics. This represents the right-hand time series characteristics. The left-right difference temporal features are represented by H, where H is the time length and diff is the difference. BiMamba is a bidirectional Mamba module used to extract gait sequence information from the forward and reverse temporal directions and to determine the leg movement state by combining the preceding and following frames. As a preferred embodiment, the fusion of the three time-series features in step S6 is represented as follows:

[0023] in, These are the characteristics after fusion.

[0024] As a preferred embodiment, the multi-task joint loss function in step S8 is expressed as follows:

[0025] in, For keypoint coordinate loss, For gait phase loss, To divide the loss, , , These are the weight parameters.

[0026] Preferably, the loss of key point coordinates is calculated using mean square error, which is expressed as follows:

[0027] The gait phase loss uses binary cross-entropy, which is expressed as:

[0028] The segmentation loss uses binary cross-entropy with Logits, which is expressed as:

[0029] Where B is the batch size, T is the sequence length, b is the batch size, and t is the time index. Predict the keypoint coordinates for the b-th sample in frame t. Let the coordinates of the real key points in the t-th frame of the b-th sample be given. Let be the predicted probability that the k-th foot is in the swing phase. For the corresponding true gait phase label, The raw values ​​for the predicted segmentation of the leg region. is the corresponding ground truth segmentation mask; BCEWithLogits is the binary cross-entropy loss, used to measure the difference between the leg region prediction and the ground truth label, and used to guide network training.

[0030] To achieve the above objectives, according to another aspect of the present invention, a medical assistance system for gait analysis from a UAV perspective is provided. The system employs the aforementioned UAV perspective gait analysis method and includes a spatial feature encoder, an auxiliary segmentation decoder, a symmetric decoupling module, a bidirectional temporal modeling module, a multi-task prediction module, a multi-task joint loss calculation module, a key point coordinate normalization module, and a data processing and training strategy module. It is applied to gait assessment and rehabilitation monitoring for Parkinson's disease, stroke, and post-orthopedic rehabilitation.

[0031] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art: This invention focuses on multi-task collaborative analysis of patient gait phase recognition, leg segmentation, and heel key point localization from a top-down perspective within medical institutions. It is applied to gait assessment and rehabilitation monitoring for diseases such as Parkinson's disease, stroke, and post-orthopedic rehabilitation. Specifically: 1. Meets medical-grade accuracy requirements: Employing a bidirectional Mamba mechanism, which utilizes both past and future temporal information, it can more accurately capture subtle changes in gait cycle characteristics and pathological gait compared to a unidirectional model, outputting traceable quantitative indicators to meet the requirements of clinical diagnosis and rehabilitation assessment for data accuracy and interpretability.

[0032] 2. Applicable to medical institution deployment environment: For the common top-down acquisition perspectives in medical institutions (corridor ceiling cameras, fixed cameras in rehabilitation rooms), a symmetrical decoupling strategy is proposed to make full use of the symmetrical distribution of the left and right legs, and to capture gait asymmetry (such as unilateral abnormality in hemiplegic patients) through differential features, thereby improving the ability to identify pathological gait.

[0033] 3. Multi-task collaboration provides clinical evidence: By jointly learning three tasks—segmentation, phase recognition, and key point localization—the system outputs multi-dimensional clinical evidence, such as leg region, gait event time, and heel trajectory, which facilitates comprehensive interpretation by medical staff and enhances the credibility of the system in medical scenarios.

[0034] 4. Robust adaptation to individual differences: The use of normalized coordinates relative to the leg bounding box, combined with segmentation-assisted bounding box estimation, improves the robustness of key point localization to individual patient differences (height, body type), clothing occlusion, and viewpoint changes, ensuring stability in real medical environments.

[0035] 5. Facilitates integration and deployment with medical devices: The entire network is trained and inferred in an end-to-end manner, maintaining linear computational complexity. It can run in real time on medical edge devices or low-power platforms without complicated post-processing steps, making it easy to integrate into rehabilitation training equipment or remote monitoring systems. Attached Figure Description

[0036] Figure 1 This is a flowchart of the gait analysis method provided by the present invention.

[0037] Figure 2 This is the overall network framework diagram of SymMamba provided by the present invention.

[0038] Figure 3 This invention provides a continuous gait diagram of pedestrians from the perspective of an unmanned aerial vehicle (UAV).

[0039] Figure 4This is a visualization of the reasoning results provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0041] Please see Figure 1 The present invention provides a method for gait analysis from the perspective of a UAV, comprising the following steps: S1: Control the drone to capture continuous gait video from the perspective of following the target person from behind, extract continuous image frames from the video, crop the target area containing the lower limbs and uniformly scale it, and form a continuous image sequence according to the preset sequence length and sliding step length. S2 Spatial Feature Encoder: The images of each frame obtained in step S1 are sequentially input into the spatial feature encoder. After convolution downsampling, ConvNeXt V2 feature extraction and bidirectional visual state spatial feature extraction stages, the spatial feature map is output. S3 Coordinate Attention Enhancement: After the ConvNeXt V2 feature extraction stage in step S2, a coordinate attention module is introduced. The coordinate attention module performs one-dimensional global pooling along the horizontal and vertical directions respectively, encodes the features in the two directions and separates them to generate horizontal attention weights and vertical attention weights, and recalibrates the spatial features of the corresponding stage. S4 Auxiliary Segmentation Decoder: The spatial feature map output from step S2 and recalibrated by coordinate attention in step S3 is input into the auxiliary segmentation decoder. After passing through a convolutional layer with a kernel size of 1×1, the unnormalized segmentation score of the leg region with the same size as the input image is output. The unnormalized segmentation score is converted into a leg region probability mask using a sigmoid activation function, and the leg region segmentation result is obtained based on the probability mask. S5 Symmetric Decoupling Module: Based on the leg region probability mask obtained in step S4, calculate its weighted centroid along the horizontal direction, and use the centroid as the adaptive left and right boundary position of the target region; generate complementary left and right soft masks according to the boundary position, and apply them to the spatial feature map output in step S2 respectively; then, in step S3, use spatial attention pooling to aggregate the left and right features into left and right feature vectors respectively, and calculate the difference between the two to obtain the left and right difference feature vectors; S6 Bidirectional Temporal Modeling Module: Input the left feature vector sequence, right feature vector sequence, and left-right difference feature vector sequence obtained in step S5 into three independent state-space temporal modeling modules to obtain left temporal features, right temporal features, and left-right difference temporal features; After applying adaptive symmetric constraints to the left and right temporal features, the three temporal features are concatenated and temporal fusion is performed. S7 Multi-task Prediction: Based on the left-side temporal features, right-side temporal features, and left-right difference temporal features obtained and fused in step S6, left and right gait phase prediction and left and right heel coordinate prediction are performed to obtain the left and right gait phase prediction results and left and right heel coordinates. Among them, the left and right foot swing phase is marked as 1 and the support phase is marked as 0. The moment when the gait phase changes from 0 to 1 is determined as the take-off event moment, and the moment when the gait phase changes from 1 to 0 is determined as the landing event moment. The leg region segmentation results, left and right gait phase prediction results, left and right gait event moments, and left and right heel coordinates obtained in step S4 are output as gait analysis results. S8 Multi-task Joint Loss Calculation: Based on the leg region segmentation results obtained in step S4 and the left and right gait phase prediction results and left and right heel coordinates output in step S7, the segmentation loss, gait phase loss and key point coordinate regression loss are calculated respectively, and the weighted sum of each loss is obtained to obtain the multi-task joint loss function; further, temporal smoothing constraints are applied to the key point coordinates and gait phase of adjacent frames. S9 Keypoint Coordinate Normalization: Based on the leg region segmentation results obtained in step S4, the leg region contour is determined, and a leg region bounding box is constructed based on the minimum, maximum, minimum, and maximum x and y coordinates of each pixel in the contour. The left and right heel coordinates obtained in step S7 are normalized using the upper left corner of the bounding box as the origin and the width and height of the bounding box as the normalization scale. During the training phase, the leg region segmentation annotations, left and right gait phase annotations, and normalized left and right heel coordinate annotations are used as supervision labels for multi-task joint training. S10 Data Processing and Training Strategy: Arrange consecutive image frames in chronological order, construct training samples using a sliding window with a preset number of frames, and slide adjacent windows according to a preset step size; During the training phase, the continuous image sequence obtained in step S1 is input into the network, and the leg region segmentation label, the left and right gait phase labels, and the normalized left and right heel coordinate labels after step S9 are used as supervision labels for multi-task joint training; during the inference phase, the continuous image sequence is input into the trained network, and the gait analysis results are output.

[0042] This invention provides a medical assistance system for gait analysis from a UAV perspective, comprising a spatial feature encoder, an auxiliary segmentation decoder, a symmetric decoupling module, a bidirectional temporal modeling module, a multi-task prediction module, a multi-task joint loss calculation module, a key point coordinate normalization module, and a data processing and training strategy module. It is applied to gait assessment and rehabilitation monitoring for diseases such as Parkinson's disease, stroke, and post-orthopedic surgery rehabilitation. Its overall network framework is as follows: Figure 2 As shown. The analysis object of this invention is a pedestrian gait video sequence from a drone's top-down perspective, such as... Figure 3 The system shown receives continuous The RGB image sequence of the frame is used as input, and the output includes leg segmentation mask, left and right gait phase, and coordinates of key points of left and right heels, which can be directly used for quantitative calculation of clinical gait parameters and rehabilitation assessment.

[0043] 1. Overall Network Architecture The SymMamba network proposed in this invention consists of the following core modules: spatial feature encoder, auxiliary segmentation decoder, symmetric decoupling module, bidirectional temporal modeling module, and multi-task prediction head.

[0044] The continuous image sequence input in step S1 is:

[0045] in, For batch size, For sequence length, and The height and width of the image (set to [value] in this invention) ).

[0046] The overall forward propagation process of the network can be represented as:

[0047] in, For the predicted key point coordinates, For the predicted gait phase, For the predicted segmentation mask, Here are the network parameters: Xseq represents the input image sequence, fSymMamba represents the SymMamba network mapping function proposed in this invention, Θ represents the set of all learnable parameters of the network, K (originally denoted as C) represents the keypoint coordinate prediction result, P represents the gait phase prediction result, M represents the segmentation mask prediction result, B represents the batch size, T represents the time series length, H represents the image height, W represents the image width, 3 represents the number of input image channels, 4 represents the output dimension of keypoint coordinates per frame, 2 represents the output dimension of gait phase per frame, and 1 represents the number of segmentation mask channels.

[0048] 2. Spatial Feature Encoder To extract spatial features from the input image, this invention employs an encoder based on the ConvNeXt V2 architecture and a bidirectional visual state space model. This encoder consists of an initial feature mapping layer (Stem), two ConvNeXt V2 feature extraction stages, and a bidirectional visual state space feature extraction stage. Specifically, the encoder includes a convolutional downsampling layer, a first ConvNeXt V2 feature extraction stage, a first coordinate attention module, a downsampling layer, a second ConvNeXt V2 feature extraction stage, a second coordinate attention module, another downsampling layer, and a bidirectional visual state space feature extraction stage, all connected sequentially. The scale of the output spatial feature map is one-sixteenth of the original image size.

[0049] The convolutional downsampling layer maps the input image from 3 channels to 32 channels and performs a 4x downsampling:

[0050] in, This indicates a convolution operation with a kernel size of 4 and a stride of 4. Presentation layer normalization operation, This represents the feature map output by the initial feature mapping layer (Stem); where the convolution maps the number of channels from 33 to 3232 and achieves a 44x downsampling in the spatial dimension.

[0051] The first two ConvNeXt V2 feature extraction stages each contain multiple ConvNeXt Blocks, and a coordinate attention module is set after each stage. The structure of the ConvNeXt Block is as follows:

[0052] in, These are input features. These are features resulting from depthwise convolution. These are the normalized features. These are features after channel expansion and activation. These are the characteristics after the global response; the above are the results after gradual changes. This maps the channel dimension from 4d to d. Separable convolution with a depth of 7x7. For the global response normalization layer, the calculation method below is the same. The calculation method is as follows:

[0053]

[0054] in, and For learnable parameters, To prevent small constants from being divided by zero. For the input feature tensor, It is the L2 norm response. It is the normalized response coefficient.

[0055] The number of channels in the three feature extraction stages are 32, 64, and 128, respectively, and the corresponding feature map sizes are respectively , , Downsampling is performed using a step size of 2. Convolution implementation:

[0056] in, This is the output feature map after downsampling. The input feature map for the downsampling operation This indicates a convolution operation with a kernel size of 2×2 and a stride of s of 2. LayerNorm indicates a layer normalization operation, where s is the convolution stride.

[0057] 3. Coordinate Attention Module To enhance the network's ability to perceive the location of key points, this invention introduces a coordinate attention module at each feature extraction stage. This module focuses on the coordinate attention along the horizontal (…). ) and vertical ( Pooling is performed along the direction to capture long-range spatial dependencies:

[0058] The features from both directions are concatenated and then encoded.

[0059] in, For Hardswish activation function, This indicates a splicing operation.

[0060] The encoded features are then separated and vertical and horizontal attention weights are generated:

[0061] The final output is:

[0062] in, This indicates element-wise multiplication.

[0063] 4. Auxiliary segmentation decoder To provide precise localization information for the leg region and assist in the normalization of key point coordinates, this invention designs a segmentation decoder module. This decoder employs a progressive upsampling structure to restore the bottleneck layer features to their original resolution. It includes two levels of four-fold bilinear upsampling, convolutional layers, batch normalization layers, and activation layers, with the specific structure as follows:

[0064]

[0065] in, This represents transposed convolution. Indicates group normalization, It is leg predictive coding. These are bottleneck layer features; the calculation methods for the remaining layers use... arrive express.

[0066] 5. Symmetric decoupling module From a drone's top-down view, a pedestrian appears as a back view, with their left and right legs symmetrically distributed in the image. This invention proposes a symmetrical decoupling strategy, dividing the feature map into left and right parts along the horizontal central axis, and horizontally flipping the right-side features to achieve alignment.

[0067] Global average pooling (GAP) is used to aggregate the features in the space into vector representations, where F is the feature map, v is the vector feature, and left and right are left and right respectively.

[0068] in, It is the left feature map. It is a left eigenvector. It is the left feature map. It is a left eigenvector. For batch size, d is the sequence length, and d is the dimension of the feature vector; In addition, left-right difference features are calculated to capture gait asymmetry:

[0069] in, These are the left and right difference feature vectors.

[0070] 6. Bidirectional Mamba timing modeling module To fully utilize the temporal information of gait sequences, this invention proposes a bidirectional Mamba (BiMamba) module that performs state space modeling from both forward and backward directions.

[0071] The state-space equations of standard Mamba are:

[0072] in, Here is the state transition matrix. For the input matrix, For the output matrix, This is a directly connected term. The parameters have the following meanings: x (t): The input signal at time t (e.g., the current frame feature).

[0073] h (t): The hidden state at time t, representing the system's internal memory of historical information.

[0074] h′ (t): The derivative of the hidden state with respect to time, representing the rate of change of the state over time.

[0075] y (t): Output signal at time t.

[0076] A: The state transition matrix determines how the "past state h(t)" evolves to the next moment.

[0077] B: Input mapping matrix, which determines the strength of the influence of "current input x(t)" on state changes.

[0078] C: The state-to-output mapping matrix determines how the hidden state is projected into the output.

[0079] D: Skip / feedthrough, indicating that the input can directly affect the output without having to go through the state memory.

[0080] The discretized recursive form is as follows:

[0081] in, , , For time step.

[0082] Mamba's core innovation lies in its selective mechanism, which enables... , and Depends on input:

[0083] This is the projection function, which dynamically generates the parameters required by subsequent SSM modules through learnable mappings. The forward propagation process of bidirectional Mamba is as follows:

[0084] The parameters have the following meanings: v : Input time series feature sequence.

[0085] Mamba fwd : Forward Mamba module, modeling in ascending time sequence, output .

[0086] Flip ): Reverse the time dimension (dim=1 indicates the time dimension).

[0087] Mamba bwd The backward Mamba module models the reversed sequence and outputs... .

[0088] Flip ): Reverse the backward result back to the original time order to obtain .

[0089] Concat( ): The forward and backward features are spliced ​​together in the channel dimension.

[0090] Linear: Linear fusion layer, outputting the final temporal features. .

[0091] The left-side timing features, right-side timing features, and left-right difference timing features of this invention are represented using independent bidirectional Mamba modules as follows:

[0092]

[0093]

[0094] in, This represents the left-side temporal characteristics. This represents the right-hand time series characteristics. The left and right time series features are represented by H, where H is the time length and diff is the difference.

[0095] 7. Multi-task prediction head The three time-series features are fused and fed into the prediction head, represented as follows:

[0096] in, These are the characteristics after fusion, and the formula calculation method has been mentioned earlier.

[0097] And the key point coordinate prediction head:

[0098] in, The result is the prediction of the key point coordinates; This represents the first linear layer that maps the feature dimension from 3D to 256D. GELU represents the Gaussian error linear unit activation function, and Dropout represents the random deactivation operation. This represents the second linear layer that maps 256-dimensional features to 4-dimensional coordinates. The four output components correspond to the left heel x-coordinate, left heel y-coordinate, right heel x-coordinate, and right heel y-coordinate, respectively.

[0099] The output is the normalized keypoint coordinates. , respectively, represent the relative positions of the left and right heels within the leg bounding box.

[0100] Gait phase prediction head:

[0101] in, The results are the predictions of the left and right foot phase probabilities. This represents the first linear layer that maps the feature dimension from 3D to 128D, and GELU represents the Gaussian error linear unit activation function. This represents the second linear layer that maps 128-dimensional features to 2-dimensional gait phase predictions, and Sigmoid represents the sigmoid activation function that transforms the predictions to the interval between 0 and 1.

[0102] The output is the phase probability of the left and right foot swings. .

[0103] 8. Loss Function Design This invention uses a multi-task joint loss function, expressed as follows:

[0104] in, For keypoint coordinate loss, For gait phase loss, To divide the loss, , , These are the weight parameters.

[0105] The loss of keypoint coordinates is calculated using mean square error, and is expressed as follows:

[0106] The gait phase loss uses binary cross-entropy, which is expressed as:

[0107] The segmentation loss uses binary cross-entropy with Logits, which is expressed as:

[0108] Where B is the batch size, T is the sequence length, b is the batch size, and t is the time index. Predict the keypoint coordinates for the b-th sample in frame t. Let the coordinates of the real key points in the t-th frame of the b-th sample be given. Let be the predicted probability that the k-th foot is in the swing phase. For the corresponding true gait phase label, The raw values ​​for the predicted segmentation of the leg region. is the corresponding ground truth segmentation mask; BCEWithLogits is the binary cross-entropy loss, used to measure the difference between the leg region prediction and the ground truth label, and used to guide network training.

[0109] || · || 2^2 : is the squared L2 norm (squared Euclidean distance).

[0110] In this invention, the weighting parameter is set to , , .

[0111] 9. Keypoint coordinate normalization and denormalization To improve the generalization ability of key point localization, this invention uses normalized coordinates relative to the leg bounding box. Let the leg bounding box be... The original keypoint coordinates are Then the normalized coordinates are:

[0112] During the inference phase, the bounding box is calculated using the predicted segmentation mask, and the coordinates are inversely normalized.

[0113] Where x and y are the pixel coordinates of the original keypoints in the image coordinate system.

[0114] x norm y norm These are normalized coordinates relative to the leg bounding box, typically ranging from 0 to 1.

[0115] bx1 and by1 are the coordinates of the top-left corner of the bounding box.

[0116] bx2 and by2 are the coordinates of the bottom right corner of the bounding box.

[0117] x pixel y pixel Pixel coordinates obtained by inverse calculation from normalized coordinates (used for final visualization or clinical measurement).

[0118] ε: A small constant to prevent the denominator from being zero, thus improving numerical stability.

[0119] 10. Data Processing and Training Strategies This invention employs a sliding window strategy to construct training samples. Let the length of the original video frame sequence be... Window length is Then it can be constructed Training samples.

[0120] Data annotation was performed using the LabelMe tool, and the annotation content included: Leg polygonal regions: used to generate segmentation masks; left heel point heel Key point coordinates; Right heel point (right) heel Key point coordinates; Left foot phase (left) foot_phase ): Boolean flag indicating whether it is in the oscillation phase; Right foot phase (right) footphase ): Boolean flag indicating whether it is in the oscillation phase; Training was performed using the AdamW optimizer with an initial learning rate of The weight decays to A cosine annealing learning rate scheduling strategy is adopted.

[0121] The key system parameters of this invention are shown in Tables 1 and 2: Table 1 Network structure related parameters

[0122] Table 2 Training-related parameters

[0123] In the experimental verification, drones were used to collect walking video data from multiple subjects in an indoor corridor environment. After extracting the video frame by frame, manual annotation was performed, resulting in approximately 1,500 valid annotated frames, which were used to construct approximately 1,800 training sequence samples.

[0124] Figure 3 This is a continuous gait image of a pedestrian from the perspective of an unmanned aerial vehicle (UAV) provided in this embodiment. Figure 4 The visualization of the model's inference results is shown, where green represents the ground truth; red represents the model's predictions; circles represent the left heel, and squares represent the right heel; through... Figure 4It can be seen that: (1) the segmentation mask can accurately locate the leg area; (2) the key point positioning can track the movement trajectory of the left and right heels; (3) the gait phase recognition can correctly determine the switching time of the swing phase and the support phase.

[0125] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for gait analysis from the perspective of an unmanned aerial vehicle (UAV), characterized in that, Includes the following steps: S1: Control the drone to capture continuous gait video from the perspective of following the target person from behind, extract continuous image frames from the video, crop the target area containing the lower limbs and uniformly scale it, and form a continuous image sequence according to the preset sequence length and sliding step length. S2 Spatial Feature Encoder: The images of each frame obtained in step S1 are sequentially input into the spatial feature encoder. After convolution downsampling, ConvNeXt V2 feature extraction and bidirectional visual state spatial feature extraction stages, the spatial feature map is output. S3 Coordinate Attention Enhancement: After the ConvNeXt V2 feature extraction stage in step S2, a coordinate attention module is introduced to generate horizontal and vertical attention weights and recalibrate the spatial features of the corresponding stage. S4 Auxiliary Segmentation Decoder: Input the spatial feature map output from step S2 and recalibrated by coordinate attention in step S3 into the auxiliary segmentation decoder, and output the unnormalized segmentation score of the leg region with the same size as the input image; use the S-shaped activation function to convert the unnormalized segmentation score into a leg region probability mask, and obtain the leg region segmentation result based on the probability mask; S5 Symmetric Decoupling Module: The leg region segmentation result obtained in step S4 is applied to the spatial feature map output in step S2. Then, in step S3, spatial attention pooling is used to aggregate the left and right features into left and right feature vectors respectively, and the difference between the two is calculated to obtain the left and right difference feature vectors. S6 Bidirectional Temporal Modeling Module: Input the left feature vector sequence, right feature vector sequence, and left-right difference feature vector sequence obtained in step S5 into three independent state space temporal modeling modules to obtain left temporal features, right temporal features, and left-right difference temporal features; S7 Multi-task prediction: Based on the left temporal features, right temporal features and left-right difference temporal features obtained and fused in step S6, perform left and right foot phase prediction and left and right heel coordinate prediction to obtain the left and right foot phase prediction results and left and right heel coordinates. S8 Multi-task Joint Loss Calculation: Based on the leg region segmentation results obtained in step S4 and the left and right gait phase prediction results and left and right heel coordinates output in step S7, the segmentation loss, gait phase loss and key point coordinate regression loss are calculated respectively, and the weighted sum of each loss is obtained to obtain the multi-task joint loss function; further, temporal smoothing constraints are applied to the key point coordinates and gait phase of adjacent frames. S9 Key point coordinate normalization: Determine the leg region outline based on the leg region segmentation result obtained in step S4, construct the leg region bounding box based on the coordinates of each pixel in the outline, and normalize the left and right heel coordinates obtained in step S7. S10 Data Processing and Training Strategy: Supervised labels are obtained through multi-task joint training. Continuous image sequences are input into the trained network in conjunction with inference, and gait analysis results are output.

2. The UAV perspective gait analysis method as described in claim 1, characterized in that, The continuous image sequence input in step S1 is: ； in, For batch size, For sequence length, and This represents the height and width of the image.

3. The UAV perspective gait analysis method as described in claim 1, characterized in that, In step S2, the spatial feature encoder includes a convolutional downsampling layer, a first ConvNeXt V2 feature extraction stage, a first coordinate attention module, a downsampling layer, a second ConvNeXt V2 feature extraction stage, a second coordinate attention module, a downsampling layer, and a bidirectional visual state spatial feature extraction stage, connected in sequence. The scale of the output spatial feature map is one-sixteenth of the original image size. In step S3, the coordinate attention module performs one-dimensional global pooling along the horizontal and vertical directions respectively, encodes the features in the two directions and separates them to generate horizontal attention weights and vertical attention weights.

4. The UAV perspective gait analysis method as described in claim 1, characterized in that, The auxiliary segmentation decoder in step S4 includes two levels of 4x bilinear upsampling, convolutional layers, batch normalization layers, and activation layers, with the following specific structure: ； in, This represents transposed convolution. Indicates group normalization, It is leg predictive coding. These are bottleneck layer features; the calculation methods for the remaining layers use... arrive The code indicates that the Decoder is used to restore the extracted abstract features to pixel-level leg region prediction results; GELU is a Gaussian error linear unit used to perform smooth nonlinear transformation on the features, enabling the network to learn complex gait features.

5. The UAV perspective gait analysis method as described in claim 1, characterized in that, In step S4, the segmentation method is to calculate the weighted centroid of the leg region along the horizontal direction based on the probability mask of the leg region obtained in step S4, and use the centroid as the adaptive left and right boundary position of the target region; and generate complementary left and right soft masks according to the boundary position. Left feature vector in step S5 Represented as: ； in, It is the left feature map. It is a left eigenvector. For batch size, d is the sequence length, and d is the dimension of the feature vector; GAP is global average pooling, which is used to average the spatial positions of each channel of the feature map and summarize the leg region features into a vector. Right eigenvector Represented as: ； in, It is the left feature map. It is a left eigenvector; Left and right difference feature vectors Represented as: ； in, These are the left and right difference feature vectors.

6. The UAV perspective gait analysis method as described in claim 1, characterized in that, In step S6, after applying adaptive symmetric constraints to the left and right time series features, the three time series features are concatenated and time series fused. Specifically, the left time series feature, the right time series feature, and the left-right difference time series feature are represented using independent bidirectional Mamba modules as follows: ； in, This represents the left-side temporal characteristics. This represents the right-hand time series characteristics. The left-right difference temporal features are represented by H, where H is the time length and diff is the difference. BiMamba is a bidirectional Mamba module used to extract gait sequence information from the forward and reverse temporal directions and to determine the leg movement state by combining the preceding and following frames. The fusion representation of the three-way time-series features is as follows: ； in, These are the characteristics after fusion.

7. The UAV perspective gait analysis method as described in claim 1, characterized in that, The multi-task joint loss function in step S8 is expressed as follows: ； in, For keypoint coordinate loss, For gait phase loss, To divide the loss, , , These are the weight parameters.

8. The UAV perspective gait analysis method as described in claim 7, characterized in that, The loss of keypoint coordinates is calculated using mean square error, and is expressed as follows: ； The gait phase loss uses binary cross-entropy, which is expressed as: ； The segmentation loss uses binary cross-entropy with Logits, which is expressed as: ； Where B is the batch size, T is the sequence length, b is the batch size, and t is the time index. Predict the keypoint coordinates for the b-th sample in frame t. Let the coordinates of the real key points in the t-th frame of the b-th sample be given. Let be the predicted probability that the k-th foot is in the swing phase. For the corresponding true gait phase label, The raw values ​​for the predicted segmentation of the leg region. is the corresponding ground truth segmentation mask; BCEWithLogits is the binary cross-entropy loss, used to measure the difference between the leg region prediction and the ground truth label, and used to guide network training.

9. The UAV perspective gait analysis method as described in claim 1, characterized in that, Step S10 specifically includes the following steps: arranging consecutive image frames in chronological order, constructing training samples using a sliding window with a preset number of frames, and sliding adjacent windows according to a preset step size; During the training phase, the continuous image sequence obtained in step S1 is input into the network, and the leg region segmentation label, the left and right gait phase labels, and the normalized left and right heel coordinate labels after step S9 are used as supervision labels for multi-task joint training; during the inference phase, the continuous image sequence is input into the trained network, and the gait analysis results are output.

10. A medical assistance system for gait analysis from a drone's perspective, employing the drone gait analysis method according to any one of claims 1-9, characterized in that, It includes a spatial feature encoder, an auxiliary segmentation decoder, a symmetric decoupling module, a bidirectional temporal modeling module, a multi-task prediction module, a multi-task joint loss calculation module, a key point coordinate normalization module, and a data processing and training strategy module. It is applied to gait assessment and rehabilitation monitoring for Parkinson's disease, stroke, and post-orthopedic rehabilitation.

Images