A human action recognition method and device

Abstract

The application discloses a human action recognition method and device, and relates to the technical field of computer vision and pattern recognition. The method comprises the following steps: collecting and preprocessing video data containing human bodies; recalibrating feature maps in a feature extraction network; removing large target deep feature layers and fusing medium and small scale feature layers; integrating multi-scale information by using an adaptive spatial feature fusion module; detecting and classifying by using a dynamic detection head integrating scale, space and task perception attention; further grading the importance of behavior data in combination with diagnosis and treatment information, and iteratively updating the diagnosis and treatment information through the recognition result. Meanwhile, the application introduces prior information in a medical scene, so that the recognition result has clinical semantic hierarchical features, and the practicability is further improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and pattern recognition, specifically to a method and apparatus for human motion recognition. Background Technology

[0002] Human action recognition is an important research area in computer vision, aiming to automatically identify and understand human behavior and actions from video or image sequences. With the development of deep learning technology, object detection and action recognition methods based on convolutional neural networks have achieved high accuracy in controlled environments.

[0003] However, in real-world environments (such as shopping malls, hospitals, nursing homes, and other public spaces), human motion recognition still faces the following challenges: First, varying shooting distances and angles result in significant differences in the scale of human targets in the image, making it difficult to effectively extract features from small targets at long distances; second, complex backgrounds and crowd occlusions cause feature interference, making it difficult for traditional feature fusion methods to fully integrate multi-scale information; third, existing models typically employ a uniform detection and recognition strategy, failing to combine prior information from specific application scenarios (such as medical rehabilitation) for targeted optimization, resulting in recognition results lacking a contextual semantic hierarchy and limiting practicality.

[0004] Therefore, there is an urgent need for a technical solution that can efficiently and accurately recognize human actions in real and complex environments and can be combined with prior information about the scene. Summary of the Invention

[0005] To achieve accurate human motion recognition in real and complex environments, this invention provides a human motion recognition method, which includes the following steps: Obtain video data containing one or more identified targets, and preprocess the video data to obtain an input image sequence; The input image sequence is input into a feature extraction network to obtain a first feature map; wherein the feature extraction network includes a shallow feature layer and a middle feature layer; The feature layers in the first feature map are fused to obtain the second feature map; The second feature map is input into the detection head, which is used to recalibrate each feature map in the second feature map and perform weighted processing on the second feature map according to the recalibration result to obtain the third feature map; The third feature map is processed to obtain the first row of data.

[0006] The human motion recognition method provided by this invention discards the deepest layer used for large target detection in the feature extraction network, retains and fuses shallow and medium-level features, which can concentrate computational resources on small and medium scales and improve the accuracy of small target detection. At the same time, the detection head recalibrates each feature map in the second feature map, and the attention mechanism weights the feature maps in the channel and / or spatial dimensions, thereby suppressing background noise and further improving the recognition accuracy of occluded and small-scale human bodies.

[0007] Furthermore, traditional multi-scale feature fusion typically employs a direct stitching method, simply superimposing feature maps of different resolutions along the channel dimension. This can easily lead to information conflicts between different layers of features at the same spatial location—for example, shallow features might classify a region as background while deeper features classify it as a target. Direct stitching retains these conflicting information simultaneously, affecting subsequent classification accuracy. To address this issue, the specific method for processing the third feature map in the aforementioned human action recognition method is as follows: The third feature map is fused using a spatial feature fusion module to obtain a fused feature map; The target object is classified based on the fused feature map to obtain the first action data.

[0008] Furthermore, traditional detection heads typically employ fixed convolution weights, regular sampling positions, and uniform channel allocation, making it difficult to adaptively handle human targets of varying sizes, positions, and poses in an image—especially in densely populated scenes. Fixed sampling positions easily introduce significant background noise, and fixed channel allocation makes it difficult to simultaneously optimize both detection and classification tasks. To improve the detection head's adaptability to varying targets, the detection head is a dynamic detection head. This dynamic detection head adjusts one or more of the scale weights, sampling positions, and task channels based on the second feature map.

[0009] Furthermore, target detection networks typically require a pre-set set of anchor boxes of fixed sizes as detection references. To ensure that the anchor boxes are consistent with the size distribution of real human targets in specific scenarios of human action recognition (such as surveillance view, drone view, etc.), thereby improving the initial matching degree between the anchor boxes and the real target boxes and solving the problems of difficult target localization and regression and slow model convergence in real recognition scenarios, the method further includes: Obtain the size distribution data of the human target, perform cluster analysis on the size distribution data to obtain the first data, which is used to describe the anchor frame size and aspect ratio of the human target; Configure the first data in the detection head.

[0010] Furthermore, in real-world environments, the target human body to be identified is often occluded by objects or other people, or truncated by image edges, resulting in missing key points in the subsequent human body keypoint detection results (i.e., human skeleton sequence) used for action recognition. This directly affects the reliability and continuity of subsequent action classification. To address this issue, in the human action recognition method, the input image sequence includes human pose data, and the specific method for preprocessing the video data is as follows: Identify key human body points in the video data to generate first human posture data; The human posture data is completed and / or filtered to obtain second human posture data; After obtaining the third feature map, the third feature map is processed according to the second human posture data to obtain the first behavior data.

[0011] Furthermore, in deep convolutional neural networks, feature information gradually degrades during layer-by-layer convolution and pooling. Detailed information from lower layers (such as edges, textures, and precise locations) is difficult to effectively transmit to subsequent feature fusion layers, leading to the loss of small target and fine action features. To reduce the degradation and loss of feature information during transmission in deep networks, a cross-layer connection is established between the feature extraction network and the feature fusion network. This cross-layer connection allows the feature map of a specified layer in the feature extraction network to be directly bypassed and transmitted to the corresponding layer in the feature fusion network, thereby reducing the loss of feature information during transmission.

[0012] Furthermore, YOLO (You Only Look Once) is a single-stage object detection network architecture that unifies object bounding box regression and category classification in a single end-to-end network, eliminating the need to generate candidate regions before classification. Therefore, it offers advantages such as fast inference speed and suitability for real-time processing. To balance detection speed and accuracy, the feature extraction network and / or the detection head are implemented based on the YOLO network architecture.

[0013] Furthermore, in settings such as hospitals and nursing homes, computer vision-based human motion recognition can effectively assist staff in observing or caring for identified targets. In such scenarios, optimizing the recognition results based on prior information can endow the human motion recognition results with realistic semantic features, further improving the practicality of human motion recognition. Therefore, the human motion recognition method also includes: Obtain diagnostic and treatment data corresponding to the identified target; The diagnostic data is matched with the first behavioral data, and the first behavioral data is classified according to the diagnostic data to obtain the second behavioral data.

[0014] Furthermore, in settings such as hospitals and nursing homes, to further improve recognition accuracy, it is necessary to guide the model to focus on the body parts that match the treatment data, thereby improving the recognition accuracy of these parts (e.g., the lower limbs after knee surgery, the upper limbs of stroke patients), achieving targeted and accurate recognition, and further enhancing the practicality of human motion recognition results. Therefore, the human motion recognition method also includes: The third feature map is weighted based on the diagnostic data to obtain the fourth feature map; The fourth feature map is processed to obtain the third row of data.

[0015] Furthermore, in medical rehabilitation scenarios, diagnostic and treatment assessments (such as rehabilitation stage and disease status) typically rely on manual updates by medical staff, lagging behind actual changes in patient behavior. This makes it difficult to detect abnormal behavioral fluctuations in a timely manner, and the assessments lack objective quantitative data support. The relationship between diagnostic and treatment information and recognition results is unidirectional, failing to form a closed loop. To promptly detect abnormal changes in patient behavior and automatically update the diagnostic and treatment assessment status, forming a closed-loop feedback between recognition results and diagnostic and treatment information, the human motion recognition method further includes: Obtain behavioral feature data corresponding to the second behavioral data, wherein the behavioral feature data includes behavioral type, behavioral frequency, behavioral duration, and behavioral amplitude features; Based on the diagnostic data, determine whether the behavioral characteristic data deviates from a preset threshold, and obtain a determination result; The diagnostic data is updated based on the judgment result.

[0016] To achieve the above objectives, the present invention also provides a human motion recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described human motion recognition methods.

[0017] One or more technical solutions provided by this invention have at least the following technical effects or advantages: (1) By introducing an attention mechanism module, the feature recognition network can focus on key areas and important channels, effectively suppress complex background noise in real recognition scenarios, and improve the feature extraction capability of occluded and small-scale human targets. (2) By removing the deep feature layer of large targets and fusing the feature layer of small and medium scales, the detection sensitivity of small targets is further enhanced while reducing the number of model parameters and computational load. (3) Adaptive spatial feature fusion is used instead of direct splicing, adaptive balance of feature contributions at each scale is achieved, information conflict between different layers is resolved, and recognition robustness in complex scenarios is improved. (4) Use dynamic detection head to unify scale, space and task 3D attention, adaptively process human targets of different sizes, positions and poses, and improve recognition performance in dense crowd scenes; (5) By combining diagnosis and treatment information, we can focus on different areas of the target and classify the importance of the identification results, so that the identification results have good practicality in medical rehabilitation scenarios. By continuing to update the diagnosis and treatment information through the identification results, we can realize the dynamic iterative update of diagnosis and treatment assessment, which further enhances the practical value of the method provided by this invention in medical rehabilitation scenarios. Attached Figure Description

[0018] The accompanying drawings, which are provided to further illustrate embodiments of the invention and constitute a part of this invention, are not intended to limit the scope of the invention. Figure 1 This is a schematic diagram of the human motion recognition method provided by the present invention. Detailed Implementation

[0019] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, where there is no conflict, the embodiments of the present invention and the features thereof can be combined with each other.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0021] Example 1 Please refer to Figure 1 Embodiment 1 of the present invention provides a human motion recognition method, which includes the following steps: Obtain video data containing one or more identified targets, and preprocess the video data to obtain an input image sequence; The input image sequence is input into a feature extraction network to obtain a first feature map; wherein the feature extraction network includes a shallow feature layer and a middle feature layer; The feature layers in the first feature map are fused to obtain the second feature map; The second feature map is input into the detection head, which is used to recalibrate each feature map in the second feature map and perform weighted processing on the second feature map according to the recalibration result to obtain the third feature map; The third feature map is processed to obtain the first row of data.

[0022] The target to be identified is the human body whose actions are to be identified. The preprocessing of the video includes, but is not limited to, existing video preprocessing methods such as video frame decoding, size normalization, and data augmentation. The method for preprocessing the video data is determined according to actual needs, and this embodiment does not impose specific limitations on it.

[0023] The feature extraction network can be implemented based on convolutional neural network architectures such as ResNet-50, EfficientNet, or MobileNet; this embodiment does not impose specific limitations. Shallow and mid-level feature layers refer to feature layers of different resolutions obtained through downsampling at different magnifications in the feature extraction network. Taking an input image size of 640×640 as an example, the feature map size of the shallow feature layer (such as layer P2, with a downsampling magnification of 4) is 160×160, offering high resolution, a small receptive field, and preservation of edge and texture details. The feature map sizes of the mid-level feature layers (such as layers P3 and P4, with downsampling magnifications of 8 and 16 respectively) are 80×80 and 40×40, achieving a balance between spatial detail and abstract semantics. In some feature extraction networks, the feature layers also include deep feature layers for large object detection (such as layer P5, with a downsampling magnification of 32 and a feature map size of 20×20). The human motion recognition method provided by this invention removes the deep feature layers in the feature extraction network, reducing redundant parameters and system computation.

[0024] The fusion of feature layers in the first feature map can be achieved using multi-scale feature fusion structures known in the art, including but not limited to: Feature Pyramid Network (FPN), Path Aggregation Network (PANet), or Bidirectional Feature Pyramid Network (BiFPN), or other equivalent feature fusion structures in the prior art. The above fusion structure integrates feature maps of different resolutions into a unified multi-scale feature representation through operations such as upsampling, downsampling, 1×1 convolution, and element-wise addition or channel concatenation. The specific fusion method is not specifically limited in this embodiment.

[0025] The recalibration of the feature map can be achieved through the SE (Squeeze-and-Excitation) channel attention mechanism, which compresses the spatial dimension through global average pooling and then learns the importance weights of each channel through a fully connected layer; alternatively, it can be achieved through the CBAM (Convolutional Block Attention Module) convolutional block attention mechanism, which learns attention weights sequentially in the channel and spatial dimensions and then weights them serially; or, it can be achieved through the CA (Coordinate Attention) coordinate attention mechanism, which performs global average pooling in both the height and width spatial dimensions to capture long-distance spatial dependencies and retain precise positional information, and then generates attention weights in two directions after convolutional transformation and sigmoid activation. The detection head can be a YOLO series coupled or decoupled detection head, a RetinaNet detection head, an FCOS detection head, etc. The recalibration method and detection head are determined according to specific needs, and this embodiment does not impose specific limitations.

[0026] Specifically, in one embodiment of the present invention, an improved YOLOv5 network architecture is used to implement the above method. The input image size is uniformly 640×640 pixels, and Mosaic data augmentation and random horizontal flipping are used in the preprocessing stage. The feature extraction network adopts the CSPDarknet-53 backbone network, which generates five sets of feature maps P1 to P5 through multi-level downsampling, wherein the feature map sizes of layers P2, P3, and P4 are 160×160, 80×80, and 40×40, respectively; the P5 layer (feature map size 20×20) used for large target detection is discarded, and layers P2, P3, and P4 are retained for multi-scale feature fusion. The neck network adopts an FPN+PAN bidirectional structure for feature fusion. The detection head employs a coordinate attention (CA) mechanism to recalibrate each feature map: first, global average pooling is performed in both the height and width directions; the pooled features from both directions are concatenated along the channel dimension and then processed by 1×1 convolution, batch normalization, and SiLU activation; finally, they are split into two independent feature tensors, which are then processed by 1×1 convolution and Sigmoid activation to obtain attention weights in the height and width directions, respectively; finally, the weights in both directions are element-wise multiplied onto the original feature map. The recalibrated feature map outputs human body bounding boxes, action categories, and confidence scores through the bounding box regression branch and action classification branch of the detection head. Both the feature extraction network and the detection head are implemented based on the YOLOv5 network architecture, which includes four parts: the input (data augmentation, anchor box calculation, image scaling), the backbone network (CSPDarknet-53, including CBS, C3, and SPPF modules), the neck network (FPN+PAN), and the detection head.

[0027] Furthermore, the specific method for processing the third feature map is as follows: The third feature map is fused using a spatial feature fusion module to obtain a fused feature map; The target object is classified based on the fused feature map to obtain the first action data.

[0028] The spatial feature fusion module can be implemented using the Adaptive Spatial Feature Fusion (ASFF) method, or other equivalent weighted feature fusion methods. Compared with direct concatenation, ASFF can adaptively adjust the contribution ratio of features at each scale at each location according to the input content, avoiding information conflicts between features from different layers.

[0029] The classification of the target based on the fused feature map can be achieved through a fully connected layer or a global average pooling plus a fully connected layer, which maps the fused feature map to an action category probability distribution and normalizes it using the Softmax function, taking the category with the highest probability as the classification result; alternatively, a convolutional layer can be used to directly output the classification result, or traditional classifiers such as support vector machine (SVM) or random forest can be used to classify the extracted features. This embodiment does not make specific limitations here.

[0030] Specifically, in one embodiment of the present invention, ASFF adaptive spatial feature fusion is employed. Taking fusion to the P2 scale as an example: the feature maps of P2, P3, and P4 after passing through the PANet structure are denoted as FM1 (160×160), FM2 (80×80), and FM3 (40×40), respectively. First, the number of channels of FM2 and FM3 is compressed to be consistent with FM1 through 1×1 convolution. Then, the spatial size of FM2 and FM3 is adjusted to 160×160 through bilinear interpolation upsampling. Next, the spatial weight parameters of the three adjusted feature maps are learned through 1×1 convolution and normalized using the Softmax function. Finally, the three feature maps are multiplied by their corresponding weights and summed element-wise to obtain the fused feature map. Based on the fused feature map, the action class probability is output through the decoupled classification branch of the detection head (global average pooling + fully connected layer + Softmax), and the class with the highest probability is taken as the first row of data.

[0031] Furthermore, the detection head is a dynamic detection head, which adjusts one or more of the scale weight, sampling position, and task channel according to the second feature map.

[0032] The dynamic detection head can be implemented through a combination of one or more attention mechanisms: scale-aware attention, which evaluates the semantic importance of features at each scale through global average pooling and 1×1 convolution, and generates scale weights through a hard-sigmoid activation function to weight and fuse features at each scale; spatial awareness attention, which learns the offset and importance scalar of sparse sampling locations through deformable convolution, enabling sampling points to adaptively focus on the target region rather than fixed grid positions; and task-aware attention, which dynamically switches the activation state of feature channels after dimensionality reduction through global pooling, followed by a fully connected layer and a sigmoid activation function, adapting to both bounding box regression and action classification tasks. These three attention mechanisms can be combined sequentially to adaptively adjust the feature map in terms of scale, space, and task dimensions.

[0033] Specifically, in one embodiment of the present invention, the dynamic detection head sequentially connects three attention modules: the scale-aware attention module first performs global average pooling on the input feature map in the spatial dimension, and then generates weights for each scale through 1×1 convolution and a hard-sigmoid activation function, and weights and fuses features of different scales; the spatial awareness attention module uses deformable convolution to learn the position offset ΔP and importance scalar Δm for each sampling point, so that K sparse sampling points adaptively aggregate to the human target region instead of fixed grid positions; the task-aware attention module reduces the dimensionality through global average pooling, passes through two fully connected layers and a sigmoid activation function, and outputs two sets of channel control parameters, using the max function to dynamically switch the activation state of feature channels between the two tasks of bounding box regression and action classification. The three modules process sequentially, completing the adaptive adjustment of scale weights, sampling positions and task channels in turn.

[0034] Furthermore, the method also includes: obtaining size distribution data of the human target; performing cluster analysis on the size distribution data to obtain first data, the first data being used to describe the anchor frame size and aspect ratio of the human target; and configuring the first data on the detection head.

[0035] The clustering analysis can be implemented using clustering algorithms known in the art, including but not limited to: K-means clustering algorithm, which uses the width and height (or aspect ratio and area) of human target bounding boxes in the training set as feature vectors for clustering, and uses the coordinates of the cluster centers as the size and aspect ratio of the anchor boxes; K-means++ algorithm, which optimizes the selection of initial cluster centers based on K-means to improve clustering stability; and Gaussian Mixture Model (GMM), which fits the multimodal distribution of target size through a probabilistic model. The anchor box sizes obtained from clustering are assigned to detection layers of different resolutions according to their scale, for example, small anchor boxes are assigned to high-resolution feature layers (such as layer P2), and large anchor boxes are assigned to low-resolution feature layers (such as layer P4).

[0036] Specifically, in one embodiment of the present invention, the width and height of all human target bounding boxes in the training set are collected as feature vectors. A K-means clustering algorithm (K value determined according to the number of detection layers, for example, 3 detection layers corresponding to 9 anchor boxes) is used for cluster analysis. The intersection-over-union (IoU) ratio is used as the distance metric, and the coordinates of the cluster centers are used as the initial size and aspect ratio of the anchor boxes. After clustering, smaller anchor boxes are assigned to high-resolution feature layers (such as layer P2, 160×160), medium-sized anchor boxes are assigned to layer P3 (80×80), and larger anchor boxes are assigned to layer P4 (40×40), ensuring that the anchor box size of each detection layer matches the receptive field of that layer.

[0037] Furthermore, the input image sequence includes human pose data. The specific method for preprocessing the video data is as follows: identifying human key points in the video data to generate first human pose data; completing and / or filtering the human pose data to obtain second human pose data; after obtaining the third feature map, processing the third feature map according to the second human pose data to obtain first behavior data.

[0038] The human keypoint detection can be implemented using a two-dimensional human pose estimation algorithm, including but not limited to OpenPose, AlphaPose, OpenPifPaf, HRNet, etc., to detect the two-dimensional coordinates of 17 human keypoints (including nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles) in COCO format. Keypoint completion can be achieved using a temporal linear interpolation method, utilizing the coordinates of known keypoints in adjacent frames before and after the missing frame for linear estimation; alternatively, spline interpolation or kinematic model-based prediction methods can be used. Keypoint filtering can be achieved by setting a threshold for the number of valid keypoints (e.g., no less than 9) to filter out keypoints with excessive missing points. The third feature map is processed based on the second human pose data. This can involve mapping the skeleton keypoint coordinates to spatial attention weights on the feature map to guide the model to focus on the human body region, or concatenating the skeleton feature vector with image features for classification.

[0039] Specifically, in one embodiment of the present invention, the OpenPifPaf pose estimation algorithm is used to detect human keypoints in each frame of the image, obtaining the two-dimensional coordinates and confidence scores of 17 keypoints in COCO format. For keypoints missing due to occlusion or truncation, temporal linear interpolation is used to complete them using the known coordinates of the keypoints in the adjacent frames before and after the missing frame. For each frame of the image, the number of keypoints with confidence scores higher than a threshold is counted; if there are fewer than 9 valid keypoints, the frame is marked as an invalid frame and filtered. After obtaining the third feature map, the coordinates of the completed skeleton keypoints are mapped to spatial attention weights on the feature map, guiding the model to focus on the human body region during classification, and outputting the first row of data by combining image appearance features and skeleton structure information.

[0040] Furthermore, a cross-layer connection is provided between the feature extraction network and the feature fusion network. Through the cross-layer connection, the feature map of a specified layer in the feature extraction network is directly bypassed and transmitted to the corresponding layer of the feature fusion network to reduce the loss of feature information during transmission.

[0041] The cross-layer connections can be implemented using residual connections, such as the identity mapping in ResNet, which directly adds the input feature map to the output feature map; or using dense connections, such as concatenating each layer with all preceding layers along the channel dimension in DenseNet; or using U-Net-style skip connections, which directly pass the feature maps from the encoder to the corresponding layers in the decoder for concatenation or addition. The feature fusion method for cross-layer connections can be element-wise addition or channel-dimensional concatenation.

[0042] Specifically, in one embodiment of the present invention, the feature map of layer P2 (160×160) in the feature extraction network is adjusted for the number of channels by 1×1 convolution and then directly bypassed to the fusion node corresponding to layer P2 in the neck network FPN+PAN structure. It is then added element-wise with the feature map upsampled from the deep layer, so that the detailed information of layer P2 can directly reach the fusion layer without going through the convolution and pooling of the intermediate layers, effectively preserving the edge and texture features of small targets.

[0043] In some embodiments provided by the present invention, the feature extraction network and the detection head are implemented based on the YOLOv5 network architecture. The YOLO series also includes YOLOv3, YOLOv7, YOLOv8 and other versions. Those skilled in the art can choose the appropriate version according to actual needs. This embodiment does not limit the specific version.

[0044] Example 2 This embodiment, based on Embodiment 1, further incorporates diagnostic data to optimize the human motion recognition results. Please continue to refer to... Figure 1 The human motion recognition method provided in Embodiment 2 of the present invention further includes the following steps: Obtain diagnostic and treatment data corresponding to the identified target; The diagnostic data is matched with the first behavioral data, and the first behavioral data is classified according to the diagnostic data to obtain the second behavioral data.

[0045] The diagnostic and treatment data refers to information related to the health status and medical care of the identified target (such as a patient or rehabilitation trainee). This data may include one or more of the following: diagnosis type, rehabilitation stage, past medical history, surgical records, medical orders, medication information, and rehabilitation assessment scale scores. The data can be structured (such as database records or JSON / XML files) or unstructured (such as medical record text). For unstructured data, structured information can be extracted using natural language processing (NLP) techniques known in the art (such as named entity recognition and relation extraction). The correspondence between the diagnostic and treatment data and the identified target in the video can be established through patient identification, for example, by retrieving corresponding diagnostic and treatment data from a medical information system using unique identifiers such as inpatient or outpatient numbers, or by binding the target in the video data to the diagnostic and treatment data using identification technologies such as facial recognition or RFID wristbands.

[0046] The classification of the first behavioral data based on the diagnostic and treatment data can be achieved in one or more of the following ways: classification based on preset rules, i.e., pre-establishing a mapping table between diagnostic types and behavioral categories (for example, marking rehabilitation training actions such as knee flexion, leg raising, and walking as high priority for post-knee surgery patients, marking daily living actions such as sitting and eating as medium priority, and marking actions unrelated to rehabilitation as low priority); classification based on weight assignment, i.e., assigning importance weights to different behavioral categories according to the diagnostic and treatment data (e.g., 0.8 weight for rehabilitation training actions and 0.3 weight for daily living actions), and classifying levels according to weight size; classification based on clinical scales, i.e., determining the importance level of the behavioral data based on rehabilitation assessment scale scores (e.g., Fugl-Meyer score, Barthel index) in the diagnostic and treatment data. The second behavioral data is behavioral data with added classification labels or importance weights based on the first behavioral data (body location box, action category, confidence level).

[0047] Specifically, in one embodiment of the present invention, a rehabilitation monitoring scenario for a target after knee replacement surgery is taken as an example. In the human motion recognition method, the target's diagnostic data is obtained, including: the diagnosis type is post-knee replacement surgery, the rehabilitation stage is the 4-week post-operative recovery period, the body parts to be monitored are both lower limbs, and the doctor's orders require daily knee flexion and straight leg raise exercises. A behavior grading table is established based on the diagnostic data: rehabilitation training movements such as knee flexion, straight leg raises, and walking with a walker are mapped to high priority (level 1); daily activities such as sitting-to-standing transitions and walking on flat ground are mapped to medium priority (level 2); and static behaviors such as sitting still and lying in bed are mapped to low priority (level 3). The first behavioral data (action category, occurrence time, duration) output in Embodiment 1 is matched with the mapping table, so that the second behavioral data obtained in the human motion recognition method has a grading label. Medical staff can refer to the second data to complete the subsequent care of the identified target.

[0048] Furthermore, the human motion recognition method also includes: The third feature map is weighted based on the diagnostic data to obtain the fourth feature map; The fourth feature map is processed to obtain the third row of data.

[0049] The weighting of the third feature map based on the diagnostic data can be achieved in one or more of the following ways: Spatial attention weighting, which involves determining the spatial region corresponding to the marked body parts in the feature map using the coordinates of key points on the human body, generating a spatial attention weight mask, assigning higher weights (e.g., 1.5-2.0 times) to the features of the regions of interest, and lower weights (e.g., 1.0 times) to the regions of non-interest, and multiplying the weight mask element-wise with the third feature map to obtain the fourth feature map; Feature linear modulation (FiLM), which involves encoding the diagnostic data into a conditional vector, generating affine transformation parameters (scaling factor γ and translation factor β) through a fully connected layer, and performing an affine transformation (γ·F+β) on the third feature map in the channel dimension; and Anchor box hierarchical weighting, which involves configuring more refined and denser anchor box sizes for the detection regions corresponding to the body parts of interest based on the diagnostic data (combined with the anchor box configuration of claim 4) to improve the detection sensitivity of key regions. The weighting process can be performed in the spatial dimension, the channel dimension, or both simultaneously.

[0050] The method of processing the fourth feature map to obtain behavioral data is the same as the method of processing the third feature map to obtain the first behavioral data in Embodiment 1. A spatial feature fusion module can be used to fuse the fourth feature map, and then the detection head outputs the human body location box, action category, and confidence score. For specific implementation details, please refer to the relevant description in Embodiment 1; this embodiment will not repeat them here. The behavioral data is the recognition result after weighted optimization of diagnostic data, resulting in higher recognition accuracy for body parts of concern in diagnosis and treatment.

[0051] Specifically, in one embodiment of the present invention, a rehabilitation monitoring scenario following knee replacement surgery is taken as an example. The body parts of interest in the diagnostic data are the lower limbs, corresponding to the left and right knees (keypoints 13 and 14) and left and right ankles (keypoints 15 and 16) in the human body keypoints. Based on the second human posture data (completed skeletal keypoint coordinates) obtained in Embodiment 1, the coordinates of the knee and ankle joints are mapped to the spatial scale of the third feature map. A Gaussian heatmap is generated centered on the keypoint coordinates as a spatial attention weight mask. The mask has a value of 1.8 in the region near the keypoint and a value of 1.0 in the region far from the keypoint. This weight mask is multiplied element-wise with the third feature map to enhance the feature response of the lower limb region, obtaining the fourth feature map. Simultaneously, combined with the anchor box configuration of claim 4, a smaller anchor box is configured for the feature map region corresponding to the lower limbs (high-resolution feature map of layer P2), improving the detection sensitivity of lower limb movements. After ASFF fusion and detection head classification, the fourth feature map outputs behavioral data, including human body bounding boxes, movement categories, and confidence scores. Compared to the unweighted recognition results, this embodiment significantly improves the recognition accuracy of fine lower limb movements such as knee flexion angle and leg lift height.

[0052] Furthermore, to enable the diagnostic data to be dynamically updated based on the patient's actual behavior, forming a closed loop of "identification-assessment-feedback," the human motion recognition method also includes: Obtain behavioral feature data corresponding to the second behavioral data, wherein the behavioral feature data includes behavioral type, behavioral frequency, behavioral duration, and behavioral amplitude features; Based on the diagnostic data, determine whether the behavioral characteristic data deviates from a preset threshold, and obtain a determination result; The diagnostic data is updated based on the judgment result.

[0053] The behavior type can be directly obtained from the action classification results of the second behavior data (such as walking, standing, sitting, lying down, going up and down stairs, falling, etc.). The behavior frequency refers to the number of times each type of behavior occurs within a preset time window (such as per hour, per day), which can be obtained by segmenting the action tags of consecutive frames into action segments (for example, when the action tags of adjacent frames change, it is determined as a new behavior segment) and counting. The behavior duration includes the cumulative duration and single duration of each type of behavior within the preset time window, which can be calculated by tracking the action tags of consecutive frames and multiplying them by the frame interval time. The behavior amplitude refers to an indicator that quantifies the magnitude of the action amplitude, which can be calculated based on the coordinates of human body key points, including but not limited to: joint angles (such as calculating the knee flexion angle using the coordinates of the hip-knee-ankle three points), limb movement speed (key points of adjacent frames are removed and the frame interval is used), stride, arm swing range, trunk tilt angle, etc.

[0054] The preset threshold is determined based on the diagnosis type and rehabilitation stage in the treatment data, with different threshold standards corresponding to different diagnosis types and rehabilitation stages. The judgment can be achieved using one or more of the following methods: single-indicator threshold comparison, comparing a single behavioral characteristic with its corresponding preset threshold (upper or lower limit); multi-indicator comprehensive scoring, comparing the sum of multiple behavioral characteristics according to preset weights with a comprehensive threshold; trend analysis, determining whether the trend of behavioral characteristics changes abnormally over multiple consecutive time windows (e.g., a continuous decrease in behavior frequency). Updating the treatment data can include one or more of the following: updating the rehabilitation stage or condition status (e.g., recommending an upgrade to the next rehabilitation stage when behavioral indicators consistently reach the next stage standard), updating the risk level (e.g., increasing the risk level when abnormal behaviors such as falls are detected), generating nursing suggestions or early warning information, updating the body parts to focus on, or adjusting the rehabilitation training plan, etc. The update can be automatic or generated update suggestions for staff review and confirmation before updating.

[0055] Specifically, in one embodiment of the present invention, a rehabilitation monitoring scenario following knee replacement surgery is taken as an example. The patient is in the 4-week recovery period after surgery, with preset thresholds as follows: at least 8 independent walks per day, at least 2 minutes of continuous walking time per session, a knee flexion angle of at least 90°, and no falls. The system extracts behavioral feature data from the second behavioral data: behavioral types include walking, standing, sitting, climbing stairs, etc.; it counts the daily walking frequency and the duration of each walking session; based on the coordinates of the hip (key points 11, 12), knee (key points 13, 14), and ankle (key points 15, 16) in the second human posture data, it calculates the knee flexion angle using a vector angle formula. The system compares the above behavioral feature data with the preset thresholds: if the walking frequency is less than 5 times / day for 3 consecutive days, or the knee flexion angle is less than 80°, it is determined to be a deviation from the threshold, indicating slow rehabilitation progress; if a fall is detected, it is immediately determined to be a deviation and a high-risk warning is generated. The system updates the diagnosis and treatment data based on the judgment results: daily behavioral characteristics statistics and deviation judgment results are written into the target file; for patients with detected abnormal deviations, warning information is generated and recorded in the diagnosis and treatment information.

[0056] Example 3 Please refer to Figure 1 Embodiment 3 of the present invention provides a human motion recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any one of the above-described human motion recognition methods.

[0057] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0058] The memory can be used to store the computer program and / or modules. The processor implements the various steps of the human motion recognition method of the invention by running or executing the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart memory card, secure digital card, flash memory card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0059] If the human motion recognition device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by storing a computer program in a computer-readable storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory, random access memory, dot carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0060] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0061] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for recognizing human actions, characterized in that, The human motion recognition method includes the following steps: Obtain video data containing one or more identified targets, and preprocess the video data to obtain an input image sequence; The input image sequence is input into a feature extraction network to obtain a first feature map; wherein the feature extraction network consists of a shallow feature layer and a middle feature layer; The feature layers in the first feature map are fused to obtain the second feature map; The second feature map is input into the detection head, which is used to recalibrate each feature map in the second feature map and perform weighted processing on the second feature map according to the recalibration result to obtain the third feature map; The third feature map is processed to obtain the first row of data.

2. The human motion recognition method according to claim 1, characterized in that, The specific method for processing the third feature map is as follows: The third feature map is fused using a spatial feature fusion module to obtain a fused feature map; The target object is classified based on the fused feature map to obtain the first action data.

3. The human motion recognition method according to claim 1, characterized in that, The detection head is a dynamic detection head, which adjusts one or more of the scale weight, sampling position, and task channel according to the second feature map, wherein the task channel is the channel used by the detection head for detection and / or classification.

4. The human motion recognition method according to claim 1, characterized in that, The method further includes: Obtain the size distribution data of the human target, perform cluster analysis on the size distribution data to obtain the first data, which is used to describe the anchor frame size and aspect ratio of the human target; Configure the first data in the detection head.

5. The human motion recognition method according to claim 1, characterized in that, The specific method for preprocessing the video data is as follows: Identify key human body points in the video data to obtain first human posture data; Complete and / or filter the first human posture data to obtain the second human posture data; After obtaining the third feature map, the third feature map is processed according to the second human posture data to obtain the first behavior data.

6. The human motion recognition method according to claim 1, characterized in that, A cross-layer connection is provided between the feature extraction network and the feature fusion network. Through the cross-layer connection, the feature map of a specified layer in the feature extraction network is directly bypassed and transmitted to the corresponding layer of the feature fusion network to reduce the loss of feature information during transmission.

7. A human motion recognition method according to any one of claims 1-6, characterized in that, The method further includes: Obtain diagnostic and treatment data corresponding to the identified target; The diagnostic data is matched with the first behavioral data, and the first behavioral data is classified according to the diagnostic data to obtain the second behavioral data.

8. The human motion recognition method according to claim 7, characterized in that, The method further includes: The third feature map is weighted based on the diagnostic data to obtain the fourth feature map; The fourth feature map is processed to obtain the third row of data.

9. A human motion recognition method according to claim 7, characterized in that, The method further includes: Obtain behavioral feature data corresponding to the second behavioral data, wherein the behavioral feature data includes behavioral type, behavioral frequency, behavioral duration, and behavioral amplitude features; Based on the diagnostic data, determine whether the behavioral characteristic data deviates from a preset threshold, and obtain a determination result; The diagnostic data is updated based on the judgment result.

10. A human motion recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the human motion recognition method as described in any one of claims 1-9.

Images