Motion recognition algorithm platform construction system based on multi-algorithm fusion

Abstract

This application discloses a system for building an action recognition algorithm platform based on multi-algorithm fusion, relating to the field of machine learning technology. The system includes: a model inference layer, an application algorithm layer, and a scenario strategy layer. The model inference layer receives multimodal input data, performs model inference processing on the multimodal input data through multiple parallel inference links, and outputs standardized inference results with a unified timestamp. The application algorithm layer aligns multi-source data based on a unified timestamp and executes multiple general algorithm functions in parallel, outputting standardized algorithm function results with a unified timestamp. The scenario strategy layer receives the standardized algorithm function results with a unified timestamp from the application algorithm layer, combines and calls corresponding general action recognition algorithm functions according to the business requirements of the target medical assessment scenario, executes action assessment logic matching the target medical assessment scenario, and finally outputs the action assessment results for that medical assessment scenario.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a system for building an action recognition algorithm platform based on multi-algorithm fusion. Background Technology

[0002] In clinical scenarios such as motor function assessment for Parkinson's disease and evaluation of the effectiveness of neurorehabilitation training, AI-based automated motion recognition technology is the core means to replace manual visual assessment and achieve standardization and quantification of assessment results. Currently, the industry has formed three mainstream technical implementation paths for this scenario: direct detection methods based on a single model, end-to-end motion recognition systems, and multi-model serial processing architectures.

[0003] The specific implementation methods of the three mainstream technologies mentioned above differ. Among them, direct detection methods based on a single model often use pre-trained object detection or pose estimation models such as YOLOv8, HigherHRNet, and MediaPipe to process the input video frames or images, directly outputting basic information such as human / hand detection boxes and skeletal keypoint coordinates. Then, through simple geometric calculations, threshold judgments, or rule matching, the system completes the determination of whether the action is completed and the evaluation of the action quality. End-to-end action recognition systems integrate the entire process of object detection, pose estimation, and action classification into a unified deep learning network. The input of raw video frames can directly output the action category and evaluation score. The network automatically learns the mapping relationship from raw pixels to the final evaluation result, without the need for explicit separation of detection, pose estimation, and other intermediate processing steps. Multi-model serial processing architectures connect AI models with different functions such as human detection, pose estimation, keypoint recognition, and action classification in sequence according to the processing order. The output of the previous model serves as the input of the next model, forming a linear processing flow. Each model completes training and deployment independently, and the process is connected only through a data interface.

[0004] The aforementioned existing technologies all suffer from core flaws that make them ill-suited to the needs of medical action assessment scenarios. The root cause lies in the lack of a reasonable layered abstraction and decoupling mechanism at the architectural design level. Specifically, single-model direct detection methods fail to decouple the underlying model inference from the upper-level business logic; end-to-end action recognition systems deeply couple scenario-specific assessment strategies with model parameters; and multi-model serial processing architectures only achieve horizontal decomposition of functional modules, failing to achieve vertical layering according to abstraction levels. This results in a lack of hierarchical abstraction capabilities in the overall solution, requiring significant code modifications or even model retraining when adding or adjusting assessment scenarios, making it difficult to adapt to multiple scenario requirements. Furthermore, the underlying basic models cannot be shared and reused across scenarios; different scenarios require independent deployment of complete model chains, leading to redundant consumption of computing resources and a lack of a unified model inference interface, significantly increasing system complexity and maintenance costs. In addition, existing technologies generally suffer from deep coupling between algorithm functionality and scenario strategies. The optimization of basic algorithms and the adjustment of scenario assessment rules are mutually constrained, making it impossible to achieve refined iteration for a single algorithm function and easily requiring modifications to the underlying algorithm code due to adjustments in assessment standards, affecting system stability. Furthermore, existing solutions suffer from several drawbacks. Firstly, they lack a unified architecture and interface specifications. The inconsistent technology stacks and interface standards across different functional modules easily lead to system integration problems such as interface mismatches and inconsistent data formats. Secondly, regarding intelligent capabilities, existing technologies primarily focus on processing single visual modalities, failing to design a unified multimodal fusion mechanism. This hinders the effective integration of multimodal information such as text commands, voice prompts, and sensor data. Moreover, they lack supporting semantic understanding, action alignment, and context awareness capabilities. They cannot translate natural language-described evaluation criteria into executable detection logic, nor can they achieve coherent recognition of long-term actions or accurate assessment of action quality. Additionally, the lack of a robust temporal alignment mechanism between multimodal input data prevents effective fusion of multidimensional information. Finally, existing solutions generally employ a centralized model training model, neglecting privacy protection technologies such as federated learning, differential privacy, and homomorphic encryption. This not only poses compliance risks related to the leakage of sensitive medical data but also prevents data sharing among multiple institutions while protecting patient privacy. Each institution can only rely on its own limited data to complete model training, severely limiting the improvement of model generalization capabilities. Summary of the Invention

[0005] This invention provides a system for building an action recognition algorithm platform based on multi-algorithm fusion. The system employs a base model inference layer to process multimodal data (visual, text, speech, etc.) in parallel and output standardized inference results. An application algorithm layer aligns multi-source data based on a unified timestamp and executes general algorithm functions such as hand and human posture recognition and multimodal fusion. A scene strategy layer combines and calls corresponding functions for specific medical assessment scenarios to output customized action assessment results. This constructs an efficient, accurate, and scalable medical action recognition and assessment system. The technical solution provided in this application is as follows: This application provides a system for constructing an action recognition algorithm platform based on multi-algorithm fusion. The system includes: a base model inference layer, an application algorithm layer, and a scene strategy layer. The base model inference layer receives multimodal input data, performs model inference processing on the multimodal input data through multiple parallel inference links, and outputs standardized inference results with unified timestamps. The application algorithm layer is communicatively connected to the base model inference layer and receives the standardized inference results with unified timestamps output by the base model inference layer. After aligning multi-source data based on the unified timestamps, it executes multiple general algorithm functions in parallel and outputs standardized algorithm function results with unified timestamps. The scene strategy layer is communicatively connected to the application algorithm layer and receives the standardized algorithm function results with unified timestamps output by the application algorithm layer. Based on the business requirements of the target medical assessment scenario, it combines and calls corresponding general action recognition algorithm functions, executes action assessment logic matching the target medical assessment scenario, and finally outputs the action assessment results for that medical assessment scenario.

[0006] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. By adopting a three-layer vertical hierarchical architecture consisting of a basic model inference layer, an application algorithm layer, and a scene strategy layer, coupled with a complete set of technical designs including dual-link parallel processing of hand and human posture, multimodal fusion and semantic understanding, temporal alignment, multimodal data quality assessment and weight adaptation, federated learning for privacy protection, anomaly tolerance, personalized assessment, interpretability analysis, and edge-cloud collaboration, the system achieves efficient reuse of the underlying model, decoupling of algorithm functions and scene strategies, rapid expansion of new assessment scenarios, accurate fusion and alignment of multimodal data, precise quantitative assessment of actions, stable and robust system operation, patient data privacy and security, personalized adaptation of rehabilitation assessment, and interpretable and traceable assessment results. This results in the core objectives of a medical action recognition algorithm platform: high scalability, high accuracy, high real-time performance, high security, high intelligence, and high usability. It significantly reduces system development, deployment, and maintenance costs, and comprehensively improves the efficiency, accuracy, and user experience of medical rehabilitation action assessment.

[0007] 2. By deploying a hand target detection module optimized for small hand targets and close-range scenes, and a hand keypoint recognition module adapted to multiple scene occlusions and pose changes in the hand inference chain, an optimized YOLO series detection network, a dedicated training strategy and inference post-processing flow are adopted. Combined with occlusion labeling, skeletal constraints, temporal smoothing and geometric verification mechanisms for keypoint detection, the hand region is accurately located and standardized 2D / 3D keypoint coordinates of the hand are stably output. This effectively adapts to complex visual scenes with different lighting, skin color, occlusion, shooting angle and scale, significantly improving the accuracy, robustness and generalization ability of hand detection and keypoint recognition. In turn, it provides reliable and accurate hand feature input for upper-layer action recognition and evaluation algorithms, ensuring the accuracy and stability of subsequent core functions such as hand action judgment and analysis.

[0008] 3. By setting up a human target detection module, a human pose estimation module, and a key point standardization processing module in the human pose inference chain, and adopting a YOLOv8 detection model adapted to rehabilitation assessment scenarios and an optimized HigherHRNet / HRNet pose estimation network, coupled with a dedicated training strategy, multi-scale feature fusion, occlusion point masking, temporal smoothing, and skeletal geometric constraint verification, and further through key point format standardization, missing anomaly handling, and integrity verification, the system can accurately locate human regions and stably output standardized 2D / 3D key points of the human skeleton. This effectively adapts to complex rehabilitation assessment scenarios with multiple poses, multiple perspectives, occlusion, and multiple interferences, significantly improving the accuracy, robustness, and data standardization of human pose estimation. In turn, it provides standard, reliable, and high-quality human pose feature input for functions such as upper-level limb angle determination, trunk stability analysis, and joint motion recognition, ensuring the accuracy and stability of subsequent human-related action assessments.

[0009] 4. By setting up a multimodal data preprocessing module, a unified timestamp synchronization module, and a standardized encapsulation module in the multimodal fusion inference link, unified preprocessing, quality control, and format standardization are performed on four types of heterogeneous data: visual, text, speech, and sensor data. The DTW / attention mechanism is used to achieve accurate temporal synchronization of multimodal data. Combined with cross-attention fusion and CLIP / BLIP visual language understanding, cross-modal feature interaction and text-visual semantic alignment are completed. This transforms heterogeneous, asynchronous, and multi-source data into unified multimodal features with standardized format, temporal synchronization, and semantic association. This effectively solves the problems of inconsistent multimodal data formats, temporal misalignment, and semantic fragmentation, and significantly improves the accuracy, robustness, and semantic understanding capabilities of multimodal information fusion. In turn, it provides high-quality, highly correlated, and semantically strong multimodal feature inputs for upper-layer application algorithms, providing reliable data and semantic support for complex action recognition and accurate action evaluation.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein: Figure 1 This is a schematic diagram of the structure of an action recognition algorithm platform construction system based on multi-algorithm fusion provided in an embodiment of the present invention. Detailed Implementation

[0012] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] This invention provides a system for building an action recognition algorithm platform based on multi-algorithm fusion. For example... Figure 1The diagram shows the system architecture of the action recognition algorithm platform based on multi-algorithm fusion, including: a base model inference layer, an application algorithm layer, and a scene strategy layer. The base model inference layer receives multimodal input data and processes it through multiple parallel inference links, outputting standardized inference results with unified timestamps. Multimodal input data includes visual, text, speech, and sensor data. The multiple inference links include a hand inference link for detecting and recognizing hand regions and key points, a human posture inference link for detecting and recognizing human body regions and posture key points, and a multimodal fusion inference link for fusing different modal data. Visual data consists of raw images or video frames from video capture devices (such as cameras), text data consists of natural language commands from user interfaces or speech recognition (such as "Please quickly tap your fingers 10 times consecutively"), speech data consists of voice prompts or commands from microphones, and sensor data consists of data from various sources. The IMU sensor data (accelerometer, angular velocity, etc.) of wearable devices (such as smartwatches and wristbands) are communicated between the application algorithm layer and the fundamental model inference layer. This layer receives standardized inference results with a unified timestamp from the fundamental model inference layer. After aligning the multi-source data using a time-series alignment model (such as a model based on Dynamic Time Warping (DTW) or a time-series alignment model based on an attention mechanism) with the unified timestamp, it executes multiple general-purpose algorithm functions in parallel and outputs standardized algorithm function results with a unified timestamp. These general-purpose algorithm functions include hand algorithm links, human posture algorithm links, and multimodal algorithm links. The scene strategy layer communicates with the application algorithm layer. This layer receives the standardized algorithm function results with a unified timestamp from the application algorithm layer. Based on the business requirements of the target medical assessment scenario, it combines and calls the corresponding general-purpose action recognition algorithm functions, executes action assessment logic matching the target medical assessment scenario, and finally outputs the action assessment results for that medical assessment scenario.

[0014] This invention constructs a complete multi-algorithm fusion action recognition platform by building a three-tiered architecture consisting of a base model inference layer, an application algorithm layer, and a scene strategy layer. The platform boasts outstanding and comprehensive technical advantages: The base model inference layer is compatible with heterogeneous modal input data from vision, text, speech, and sensors. It performs model inference in parallel through multiple links, including hand inference, human posture inference, and multi-modal fusion inference, simultaneously realizing the detection and recognition of hand regions and key points, human regions and posture key points, and multi-modal data fusion processing. It outputs standardized inference results with timestamps, significantly improving the parallel processing efficiency of multi-source heterogeneous data and eliminating processing obstacles caused by differences in data formats and timing through standardization and unified timestamp regulation. The application algorithm layer is interconnected with the base model inference layer. After receiving standardized inference results, it uses dynamic time warping or attention-based timing alignment models to achieve accurate timing matching of multi-source data, and then runs multiple algorithm links for hand, human posture, and multi-modal processing in parallel to execute general algorithm functions. The system continuously outputs standardized algorithm results with unified timestamps, effectively solving the problems of temporal misalignment and inconsistent feature extraction in multi-source action data, significantly improving the accuracy of action feature parsing and data synergy. The scenario strategy layer can flexibly combine and call corresponding general action recognition algorithm functions to meet the specific business needs of the target medical assessment scenario, execute dedicated action assessment logic adapted to the medical scenario, and finally output professional and practical medical action assessment results. The overall layered architecture achieves decoupling and efficient collaboration between the underlying basic inference, the middle layer general algorithm functions, and the upper layer scenario-based assessment applications. The combination of multi-algorithm fusion and multi-modal data processing greatly enhances the robustness and generalization ability of action recognition. Standardized output and temporal alignment processing ensure the stability, consistency, and reliability of recognition and assessment results in medical assessment scenarios. At the same time, the system can flexibly adapt to different medical assessment business scenarios, taking into account both platform universality and medical scenario specificity, and effectively meeting the core technical needs of the medical field for accurate human action recognition and professional quantitative assessment.

[0015] The hand inference chain includes a hand target detection module and a hand keypoint recognition module. The hand target detection module processes the input visual data (covering different lighting conditions, skin color, occlusion, background complexity, shooting angle, and hand size) using a hand target detection model (such as a YOLO-based hand detection model), and outputs the bounding box coordinates and confidence scores of the hand region according to the target detection format. The hand detection model is a target detection network optimized for small hand targets and close-range scenes. The hand target detection model uses the YOLO series detection network, adjusting the input size (e.g., 640×640) and anchor box / keypoint recognition for the characteristics of small hand targets. The model can use bounding boxes or anchor-free heads, and a lightweight module can be added. During training, the labeled bounding boxes are used as supervision signals. The model combines IoU regression loss (such as CIoU / DIoU), classification loss (such as BCE / CE), and confidence loss. Data augmentation (random scaling / cropping, color jittering, motion blur, MixUp / Mosaic, etc.) is used to improve generalization. The optimal weights are selected based on metrics such as recall. During inference, the input frames are preprocessed and then input into the model. The model outputs candidate boxes, which are then subjected to threshold filtering and non-maximum suppression to obtain the final hand bounding box. The coordinates are then mapped back to the original image to provide accurate localization for subsequent hand keypoint recognition. The hand keypoint recognition module is used to detect keypoints based on the detected hand region using a hand keypoint recognition model (such as MediaPipeHands or a custom hand keypoint model), and outputs 2D / 3D coordinate information of the hand keypoints (such as the 2D or 3D coordinates of each joint of the thumb, index finger, middle finger, ring finger, and little finger) and confidence scores; the hand keypoint recognition model is a deep learning network that outputs hand keypoints to achieve hand pose estimation. The construction, training, and processing of the hand keypoint recognition model include: cropping the region of interest (ROI) based on the detected hand region, labeling keypoint coordinates, and marking occluded points with visibility or missing markers to mask regression errors from invisible points during training; the model can employ a two-stage or direct regression network (such as HRNet), outputting heatmaps or coordinates, and can add a 3D estimation module; during training, the heatmap output uses MSE / OKS loss, coordinate regression uses L1 / L2 / Huber loss and introduces skeletal constraints, and generalization is improved through scale / rotation enhancement, evaluated using PCK / OKS; during inference, the ROI is expanded, affine transformed, and normalized before being input into the model, and post-processing obtains the keypoints of the original image by taking the peak value of the heatmap or by inverse coordinate transformation, followed by temporal smoothing and geometric verification, outputting standardized keypoints and confidence scores.

[0016] This invention, through adapting and adjusting input size and anchor boxes or adopting an anchor-free structure and incorporating lightweight design, combines IoU regression loss (CIoU / DIoU), BCE / CE classification loss, and confidence loss during the training phase. It also employs diverse data augmentation techniques such as random scaling and cropping, color jittering, and motion blur to enhance model generalization ability. Recall rate is used as the core metric to select the optimal model weights. During the inference phase, after image preprocessing, candidate box output, threshold filtering, and non-maximum suppression, it can accurately output the bounding box coordinates and confidence score of the hand region. It effectively adapts to complex visual scenes with varying lighting, skin tones, occlusion, background complexity, shooting angles, and hand size, significantly improving the localization accuracy, recall rate, and anti-interference ability of small hand target detection. This allows for precise delineation of the region of interest (ROI) for subsequent keypoint recognition, completely eliminating interference from irrelevant background areas. The hand keypoint recognition module then performs ROI cropping based on this precisely located hand region, employing dedicated deep learning networks such as MediaPipeHands or HRNet. By labeling the visibility and missing information of occluded keypoints, it masks invalid regression errors. During the training phase, the model incorporates heatmap MSE / OKS loss, coordinate L1 / L2 / Huber loss, and introduces skeletal structure constraints. Scale and rotation enhancements further optimize the model's generalization performance. PCK / OKS is used as the model evaluation metric. During the inference phase, the model is input after edge expansion, affine transformation, and normalization. Post-processing operations such as heatmap peak extraction, inverse coordinate transformation, temporal smoothing, and geometric verification are then performed to stably output the 2D / 3D coordinates and confidence scores of the joints of the thumb, index finger, and other fingers, achieving high-precision hand pose estimation and effectively solving the problem of complex poses. The system addresses the issues of large detection errors and unstable results in scenarios with key point occlusion. Overall, the two modules work seamlessly together to construct a complete and efficient hand inference chain. This ensures the accuracy and scene adaptability of hand region detection, while also enabling refined and standardized extraction of hand key points. The high-reliability hand detection and pose data output provides a precise and stable underlying visual data foundation for the temporal processing, action parsing, and action recognition in the final medical assessment scenario of the upper application algorithm layer, significantly improving the robustness and accuracy of the entire system in hand action recognition.

[0017] The human pose inference chain includes a human target detection module, a human pose estimation module, and a keypoint normalization module. The human target detection module processes the input visual data using a human target detection model (such as the YOLOv8 human detection model) and outputs the bounding box coordinates and confidence scores of the human body region. The human target detection model is a deep learning network that outputs the human body position to achieve human localization. The construction, training, and processing of the human target detection model include: acquiring human images covering multiple poses, occlusions, viewpoints, and multiple people interference in rehabilitation assessment scenarios and labeling the bounding boxes; training using target detection frameworks such as YOLOv8, combined with IoU regression, classification, and confidence loss; applying data augmentation such as Mosaic; and achieving mAP. The system selects optimal weights based on recall; during inference, the input frames are preprocessed and input into the network, then thresholded and non-maximum suppressed to output human bounding boxes, which are then mapped back to the original image coordinates to provide accurate localization for subsequent human pose estimation; the human pose estimation module is used to estimate the pose based on the detected human regions using a human pose estimation model (such as HigherHRNet), outputting 2D / 3D coordinate information of human skeletal key points (such as 2D or 3D coordinates of key points such as head, shoulder, elbow, wrist, hip, knee, and ankle) and confidence scores; the human pose estimation model outputs human skeletal key points to realize a deep learning network for human pose parsing. The construction, training, and processing of the human pose estimation model include: collecting and labeling the human region of interest. Note key points (such as COCO17 points) and visibility; 3D pose can be combined with multi-view or 2D-to-3D data; use backbone networks such as HigherHRNet / HRNet to output heatmaps, improve the accuracy of small joints such as wrists and ankles through multi-scale feature fusion, and add associative embeddings to distinguish multiple people; during training, heatmaps are used as supervision, MSE / OKS loss is used, and occlusion points are masked according to visibility masks, combined with scale / rotation / occlusion enhancement, and evaluated by AP and PCK; during inference, the human bounding box is expanded and the region of interest is cropped and affinely transformed to the input size, input into the network to obtain the heatmap, obtain key point coordinates through peak detection and map them back to the original image, and then perform temporal smoothing and skeletal geometric constraint verification to output key points and confidence scores; key point labels The standardization module is used to perform format standardization, missing and anomaly handling, and integrity verification on the output human skeletal key points. It outputs a standardized human key point data structure. The format standardization process forms a standard data structure by unifying the key point index order, coordinate system (pixel / normalized), timestamp field, and confidence field. The missing and anomaly handling process marks low-confidence points with confidence scores below a preset confidence threshold as missing. It performs interpolation smoothing on abrupt changes that differ from adjacent frame data values ​​by more than a preset deviation threshold. If there are still no abrupt changes after interpolation smoothing, reliable points from the previous frame are used for short-term completion. The integrity verification includes checking the number and range of key points (whether the coordinates are out of bounds), and whether the length of key bone segments and joint angles are within a reasonable range.Output an alarm and record the reason for data that does not meet the constraints, for use in subsequent algorithm steps (such as stability determination, joint angle determination).

[0018] In this invention, the human target detection module employs a YOLOv8 human detection model adapted for rehabilitation scenarios. It collects labeled data for training under complex conditions such as multiple poses, partial occlusion, varying shooting angles, and interference from multiple people. The model is optimized by combining IoU regression loss, classification loss, and confidence loss, and its generalization ability is improved through data augmentation techniques such as Mosaic. The optimal model weights are selected based on mAP and recall. During the inference stage, after preprocessing, threshold filtering, and non-maximum suppression, the bounding boxes of the human body region are accurately output and mapped to the original image coordinates. This effectively solves the problems of human positioning being easily interfered with and small joint region positioning in medical scenarios. To address the issue of blurriness, the module delineates precise and effective regions for subsequent pose estimation. The human pose estimation module relies on backbone networks such as HigherHRNet and HRNet, and utilizes multi-scale feature fusion technology to enhance the recognition accuracy of small joints such as the wrist and ankle. Simultaneously, it achieves multi-person pose differentiation through associative embedding. During training, keypoint visibility is annotated and masked to mitigate errors from occlusion points. The model's robustness is improved by combining MSE / OKS loss functions with scale, rotation, and occlusion enhancements. During inference, after region cropping, affine transformation, and heatmap peak detection, temporal smoothing and skeletal geometric constraints are applied for verification, resulting in a stable output of the head and torso. The 2D / 3D coordinates and confidence scores of key skeletal points such as limbs significantly improve the problems of large deviations and distorted results in human skeleton key point detection under pose occlusion, pose distortion, and multi-person overlapping scenes. The key point standardization processing module further standardizes the index order, coordinate system, timestamp, and confidence score field format of human skeleton key points, marks missing key points with low confidence, uses interpolation smoothing and short-term completion processing with reliable points from the previous frame for data mutation anomalies, and completes the integrity verification by the rationality of key point quantity, coordinate boundaries, bone segment length, and joint angles. Alarm records are output for non-compliant data, ultimately forming a complete system. The standardized and highly stable human key point data structure completely eliminates calculation errors caused by data heterogeneity between different frames, numerical jumps, and missing key points, ensuring that the data foundation of subsequent upper-level algorithm links such as joint angle determination and motion stability analysis is consistent and reliable. Overall, the three modules are interconnected and optimized layer by layer, enabling the human posture inference link to have high detection accuracy, strong anti-interference ability, and high data standardization in complex medical assessment scenarios. This significantly improves the robustness and accuracy of human posture analysis, providing a solid and reliable underlying data guarantee for the multi-algorithm fusion motion recognition platform to achieve accurate quantitative analysis in medical motion assessment.

[0019] The multimodal fusion inference pipeline includes a multimodal input data preprocessing module, a unified timestamp synchronization module, and a standardized encapsulation module. The multimodal input data preprocessing module preprocesses, performs quality control, and format standardization on visual, text, speech, and sensor data respectively, generating standardized feature data for each modality. First, integrity checks and missing data markers are performed on each modal input, and standard timestamps are generated using a unified time base. Then, the multimodal input data is encapsulated into a unified data packet structure. Specifically, visual data is decoded, frame-sampling, and normalized to generate image tensors; text data is cleaned, segmented, and encoded to generate standardized text identifiers; speech data undergoes sampling rate unification, noise reduction, endpoint detection, and feature extraction to generate acoustic features; and sensor data is filtered, denoised, resampled, aligned, and windowed normalized to generate inertial measurement unit features. Finally, all modal data is output in a structured format with accompanying quality metadata, providing standardized input for subsequent temporal alignment and multimodal fusion. The unified timestamp synchronization module is used to synchronize data from different modalities using a temporal alignment model (such as a model based on Dynamic Time Warping (DTW) or a temporal alignment model based on an attention mechanism). For example, it aligns the timestamps of sensor data with the timestamps of video frames to ensure that multimodal input data at the same moment can be correctly matched. The standardization encapsulation module uses a multimodal fusion model (such as a fusion model based on a cross-attention mechanism) to fuse multimodal input data, generating a unified multimodal feature representation, which is then output to subsequent temporal alignment and fusion processing stages. First, the preprocessed and aligned visual, audio, IMU, and text features are mapped to a unified dimension through a projection layer, and modality and position embeddings are added to form a temporal sequence. Then, a cross-attention mechanism is used for cross-modal information interaction, with the target modality as the query and the other modalities as keys to calculate the enhanced representation. Gating coefficients are calculated based on the quality indicators of each modality for weighted fusion, resulting in a unified fused feature. Finally, after layer normalization and temporal pooling, a structured result is output for downstream action determination. The visual language understanding component employs models such as CLIP / BLIP to encode text instructions and visual frames / segments into a shared space. It calculates the matching degree using cosine similarity and utilizes sliding window or attention mechanisms to locate the action segment and key regions that best match the instruction. Domain fine-tuning (contrastive learning or image-text matching loss) enhances fine-grained semantic understanding capabilities. The fused features and semantic alignment results from both components provide rich multimodal semantic input to the application algorithm layer, supporting action recognition and evaluation.

[0020] This invention, through the collaborative operation of a multimodal data preprocessing module, a unified timestamp synchronization module, and a standardized encapsulation module, achieves end-to-end optimized processing of heterogeneous data from vision, text, speech, and sensors, from standardized processing and temporal synchronization to deep semantic fusion. The overall technical effect is significant and comprehensive: The multimodal data preprocessing module first performs integrity checks and missing data marking on each modality input, generates a standard timestamp based on a unified time base, and then performs decoding, frame extraction, and image normalization on visual data; performs cleaning, word segmentation, and encoding on text data; performs unified sampling rate noise reduction and acoustic feature extraction on speech data; and performs filtering, noise reduction, and resampling on sensor data. Along with windowing and normalization, the final output is structured feature data with accompanying quality metadata, achieving format unification and quality control of heterogeneous data sources. This effectively eliminates format barriers and quality differences between different modal data, providing standardized and reliable input for subsequent time-series alignment and fusion processing. The unified timestamp synchronization module relies on Dynamic Time Warping (DTW) or attention mechanism time-series alignment models to accurately align the timestamps of sensor data with different modal data such as video frames. This ensures that the multimodal information corresponding to the same action moment can be accurately matched and corresponded, fundamentally solving the problems of time-series misalignment and data asynchrony caused by multi-device acquisition, and guaranteeing the accuracy of cross-modal data association. For accuracy and consistency, the standardized encapsulation module adopts a multimodal fusion model based on cross-attention mechanism. First, aligned features are projected onto a unified dimension, and modality and position embeddings are added to construct a temporal sequence. Cross-modal information interaction is enhanced through cross-attention. Then, weighted fusion is achieved by combining quality indicators of each modality. Simultaneously, visual language models such as CLIP / BLIP encode text instructions and visual data into a shared feature space. Cosine similarity and sliding window mechanisms are used to accurately locate and match action segments and key regions of the instructions. Domain fine-tuning improves fine-grained semantic understanding capabilities. Finally, structured unified multimodal data is output after layer normalization and temporal pooling. The link not only achieves deep complementary fusion of multimodal features, greatly enriching the information dimension of action features and effectively improving the robustness and anti-interference ability of action recognition in complex scenarios, but also achieves accurate semantic alignment between text commands and action visual data. It can accurately parse action commands related to medical assessment and lock key action regions, providing high-quality input with both multimodal perception and semantic understanding capabilities for the application algorithm layer. This further strengthens the entire action recognition platform's ability to recognize and parse commanded and refined actions in medical assessment scenarios, providing solid data and feature support for subsequent accurate action assessment.

[0021] The fundamental inference layer also includes enhancement modules, which include a data quality assessment unit, a multimodal fusion weight adaptive unit, an anomaly detection unit, a cross-scenario knowledge transfer unit, an edge-cloud collaboration unit, and a federated learning privacy protection unit. The data quality assessment unit evaluates the input data for each modality and outputs a data quality score for the corresponding modality, including video clarity scores (based on image clarity, contrast, noise level, etc.), sensor data reliability scores (based on data integrity, outlier detection, etc.), and speech clarity scores (based on signal-to-noise ratio, speech activity detection, etc.). The multimodal fusion weight adaptive unit dynamically adjusts the fusion weights of each modality in the multimodal fusion model based on the data quality scores of each modality. For example, it increases the visual weight when the video quality is high, increases the sensor weight when the sensor data is reliable, and decreases the weight of a modality or excludes the modality when the data quality of a certain modality is low. The anomaly detection unit detects anomalies in various modal data, including action interruptions, equipment malfunctions, missing data, and outlier values. When any data anomaly is detected, the anomaly modality and type are labeled, providing a basis for subsequent error-tolerant processing. The cross-scenario knowledge transfer unit identifies common action features across different medical assessment scenarios (such as common patterns of hand movements and similarities in human posture). Through transfer learning, it transfers model knowledge from trained scenarios to new scenarios, reducing the training data requirements for new scenarios. The edge-cloud collaboration unit, based on model complexity and real-time requirements, transfers lightweight... Inference models (such as hand object detection) are deployed on edge devices to perform real-time inference, while complex models (such as multimodal fusion models) are deployed in the cloud for computation. This supports data synchronization, collaborative computation, and incremental model updates between the edge and cloud. The federated learning privacy protection unit uses differential privacy technology to add noise during training and homomorphic encryption to encrypt parameters. Each medical institution trains locally, uploading only the encrypted parameters to the central server for federated averaging and aggregation. This achieves distributed training and incremental updates without data leaving the local machine, and outputs the inference results, including the federated learning model parameters, to the application algorithm layer in a unified data format. First, differential privacy technology is used to add noise during training, combined with homomorphic encryption to encrypt and aggregate model parameters, ensuring patient data privacy is not leaked. Each medical institution trains the model locally using its own data, uploading only the encrypted model parameters to the central server, achieving distributed training without data leaving the local machine. The central server generates global model parameters using federated averaging and other aggregation algorithms and distributes them to each institution, enabling continuous model optimization and improved generalization capabilities. Incremental updates are also supported, allowing the addition of new institutions or new data without retraining the entire model. The base model inference layer outputs the results generated in the above process, such as detection boxes, key point coordinates, confidence scores, multimodal fusion features, semantic understanding results, data quality scores, anomaly markers, knowledge transfer features, and federated learning model parameters, to the application algorithm layer in a unified data format for subsequent use.

[0022] This invention integrates a data quality assessment unit, a multimodal fusion weight adaptive unit, an anomaly detection unit, a cross-scenario knowledge transfer unit, an edge-cloud collaboration unit, and a federated learning privacy protection unit. It comprehensively improves the overall system performance from multiple dimensions, including data management, fusion optimization, anomaly tolerance, scenario expansion, deployment adaptation, and privacy security, providing more stable, accurate, secure, and efficient underlying inference support for upper-layer applications. The data quality assessment unit can quantitatively score multimodal input data such as video, sensor, and voice data based on dimensions such as image clarity, data integrity, and signal-to-noise ratio, providing objective and reliable quality references for subsequent data processing and fusion decisions. The multimodal fusion weight adaptive unit dynamically adjusts the weight allocation of each modality in the fusion model based on the aforementioned quality scores, strengthening the feature contribution of high-quality modalities and reducing or even eliminating low-quality data, effectively avoiding interference from invalid or inferior data in the fusion results and significantly improving the rationality and accuracy of multimodal feature fusion. The anomaly detection unit can identify various abnormal situations in real time, such as action interruption, equipment failure, data loss, and abnormal values, and accurately label the abnormal modalities and types, providing a basis for further analysis and evaluation. The system provides crucial support for fault-tolerant processing and ensuring the continuous and stable operation of the recognition process. The cross-scenario knowledge transfer unit mines common action and posture features in different medical assessment scenarios, and reuses the mature knowledge of trained models through transfer learning, significantly reducing the amount of data and training cycle required for training models in new scenarios, effectively enhancing the system's scenario expansion capabilities and generalization performance. The edge-cloud collaboration unit deploys lightweight models such as hand detection at the edge to achieve low-latency real-time inference, and places complex models such as multimodal fusion in the cloud for high-performance computing, based on the model's lightweight nature and business real-time requirements. It also supports data synchronization, collaborative inference, and incremental model updates between the edge and the cloud, balancing the system's real-time response speed and complex computing capabilities. The federated learning privacy protection unit adopts a combination of differential privacy noise addition and homomorphic encryption, and uses a federated average aggregation algorithm to achieve distributed training and model iterative updates of local data from various medical institutions without leaving the domain. While strictly protecting patient privacy data and meeting medical data compliance requirements, it continuously optimizes model accuracy and generalization capabilities, and also supports incremental updates after new institutions and new data are connected, avoiding the resource consumption of full retraining. Finally, the fundamental model inference layer encapsulates and outputs a unified format of diverse results, including detection boxes, key point coordinates, confidence scores, multimodal fusion features, semantic understanding results, data quality scores, anomaly markers, knowledge transfer features, and federated learning model parameters. This not only ensures the standardization and integrity of the underlying inference data but also comprehensively improves the system's robustness, adaptability, deployment flexibility, and privacy and security compliance in complex medical scenarios. It provides a high-quality, highly reliable, and highly secure data foundation for the application algorithm layer's temporal alignment, action parsing, and scenario strategy layer's medical evaluation.

[0023] Standardized rules are set for the inter-layer flow and timing alignment of multimodal input data. These rules include inter-layer data synchronization rules and timing alignment adaptation rules. The inter-layer data synchronization rules include: after the base model inference layer completes the inference of each link (hand, human posture, multimodal fusion), it encapsulates the output of each link into a data packet with a unified timestamp and transmits it to the application algorithm layer through the inter-layer interface. The synchronization methods include push mode (written to a shared buffer or message queue upon completion of inference, and consumed by the application algorithm layer according to the time window), pull mode (the application algorithm layer requests the inference results of a certain time period as needed), and synchronous call mode (called and transmitted the data of the current frame layer by layer when the scene strategy layer is triggered). The inference results of vision, sensor, voice and other multimodal data at the same time share the same timestamp, which facilitates the application algorithm layer to perform fusion and judgment according to time alignment. After the application algorithm layer completes its algorithmic functions, it also outputs data to the scene strategy layer in a unified timestamp format to ensure data consistency and traceability between layers, avoiding out-of-order processing and duplicate processing. Timing alignment adaptation rules include: using a unified time base as a reference, setting timestamp synchronization precision for each modal data to match business requirements: frame level (e.g., approximately 33.3ms for 30fps), millisecond level (e.g., 1ms), etc., to meet the timing consistency requirements of multimodal fusion and action evaluation; for multimodal input data with different sampling rates, multiple methods are used to achieve timing alignment, ensuring... To ensure the temporal consistency of multimodal features, various methods are employed, including interpolation, resampling, and event alignment. Interpolation involves linearly or spline-interpolating high-frequency modalities (e.g., a 100Hz IMU) at low-frequency time points (e.g., 33.3ms per frame) to obtain sensor sequences corresponding one-to-one with video frames. Resampling involves resampling each modality to a unified time grid (e.g., 100Hz) and then matching them by timestamp. Event alignment uses key events (e.g., start and end of actions) as anchor points to align local segments using dynamic time warping (DTW). After these temporal alignments, the multimodal fusion model and subsequent algorithms use the same semantic event's multimodal features at the same time, ensuring the accuracy of fusion and judgment.

[0024] This invention, through setting two standardized rules—inter-layer data synchronization rules and timing alignment adaptation rules—completely solves the problems of out-of-order, duplication, and timing misalignment that easily occur in the inter-layer transmission and cross-modal collaboration of multimodal data from two core dimensions: data flow standardization and timing consistency assurance. This significantly improves the standardization, stability, and accuracy of data processing in the entire action recognition platform, providing solid timing and flow guarantees for multi-algorithm fusion and action assessment in medical scenarios. The inter-layer data synchronization rules clarify the data transmission standards between the base model inference layer and the application algorithm layer, and between the application algorithm layer and the scene strategy layer. After completing inference for each link, such as hand, human posture, and multimodal fusion, the base model inference layer encapsulates all outputs into standardized data packets with a unified timestamp. Through three flexible transmission modes—push, pull, and synchronous call—data is transmitted to the application algorithm layer on demand. After completing general algorithm function processing, the application algorithm layer also outputs results to the scene strategy layer in a unified timestamp format. This standardized transmission method does not... By ensuring that the results of multimodal inference, such as vision, sensor, and voice, share the same timestamp at the same moment, the consistency and traceability of data between layers are achieved, effectively avoiding problems such as out-of-order data transmission and duplicate processing, and ensuring efficient and smooth data interaction between layers. The time sequence alignment adaptation rules are based on a unified time benchmark and flexibly set different precision timestamp synchronization standards such as frame level and millisecond level according to the needs of medical assessment business. This accurately matches the differentiated requirements of multimodal fusion and action assessment for time sequence consistency. For multimodal data with different sampling rates, three adaptation methods are used: interpolation processing (linear or spline interpolation of high-frequency IMU data at low-frequency video frame time points), resampling processing (unifying each modality to the same time grid), and event alignment processing (using key events such as the start and end of actions as anchor points and completing local segment alignment through algorithms such as DTW). This completely solves the problem of time sequence misalignment caused by the difference in sampling rate of multimodal data, ensuring that the aligned multimodal data corresponds to the same semantic event at the same moment. The two rules work together to standardize the flow of multimodal data between the basic model inference layer, application algorithm layer, and scenario strategy layer, achieving unified standardization of data format, timestamp, and transmission method. Through precise temporal alignment processing, the temporal consistency of multimodal features is ensured, enabling the multimodal fusion model and subsequent upper-layer algorithms to process and judge multimodal features with the same time and semantics. This effectively avoids action recognition errors caused by temporal misalignment and data disorder, significantly improving the accuracy and reliability of the entire system's action recognition. At the same time, the flexible transmission mode and temporal alignment method also enhance the system's adaptability to different medical assessment scenarios and different sampling devices, providing stable, standardized, and reliable temporal data support for accurate action assessment in medical scenarios.

[0025] The hand algorithm link, human pose algorithm link, and multimodal algorithm link of the application algorithm layer each include the following functional units: The hand algorithm link includes a finger pinching determination unit, a palm flipping determination unit, and a hand key point tracking unit; The finger pinching determination unit is used to calculate the Euclidean distance between fingers based on the coordinates of hand key points, and determine the finger pinching / opening state. When the Euclidean distance is less than a preset distance threshold (e.g., 20 pixels), it is determined to be a finger pinching state. When the Euclidean distance is not less than the preset distance threshold, it is determined to be a finger open state, and outputs the speed and duration features of the pinching action; The palm flipping determination unit is used to calculate the palm normal vector and orientation angle based on the hand key point recognition results, determine the palm flipping action, and output the flipping angle, angular velocity, and completion features. Specifically, the palm plane can be constructed by selecting "wrist point → index finger root point" and "wrist point → little finger root point", and its cross product is calculated to obtain the normal vector. The angle between the normal vector and the reference direction of the camera coordinate system (e.g., towards the lens direction) is calculated as the orientation angle. By analyzing the changes in the direction or orientation of the normal vector over time, when the angle crosses a preset threshold (configurable) and continues for several frames, it is determined that the palm has flipped (e.g., flipping from palm up to palm down). At the same time, a simple smoothing (e.g., moving average) is applied to the angle sequence to suppress jitter. The hand key point tracking unit is used to perform temporal tracking and trajectory continuation of hand key points in consecutive frames. Specifically, the position of the key point in the previous frame is used as a priori, and nearest neighbor matching or Hungarian algorithm is used to complete the inter-frame association. For short-term occlusion or loss of key points, Kalman / OneEuro filters can be used for position prediction and smoothing, and trajectory continuation is performed when the key point reappears, thereby improving the continuity and stability in fast-moving and occluded scenes. Subsequently, based on the timestamp, the velocity and acceleration of the keypoint trajectory are calculated by difference (the difference result can be subjected to moving average to suppress jitter), and the temporal features of the motion trajectory, velocity, and acceleration of the keypoint are output; the human posture algorithm link includes a limb angle determination unit, a trunk stability determination unit, and a joint motion determination unit; the limb angle determination unit is used to calculate the angle of the target limb joint based on the human skeleton keypoints, and outputs the joint angle value, angle change rate, and angle range features. In order to suppress keypoint jitter, the angle sequence can be subjected to moving average / median filtering, and outliers (caused by low confidence keypoints) can be removed or interpolated. The algorithm can calculate the angle change rate (difference / fitting slope) and angle range (maximum-minimum) within a fixed time window, and output features such as angle value, angle change rate, and angle range. The trunk stability determination unit is used to calculate the trunk tilt angle, center of gravity position, and stability index based on the human body key point recognition results, and to determine the abnormal trunk movement state. Specifically, the "midpoint of the left and right shoulders and midpoint of the left and right hips" can be used as the trunk axis vector, and the angle between the vector and the vertical direction of the image can be used as the trunk tilt angle. The center of gravity position can be approximately taken as the projection of the midpoint of the shoulder / hip or its weighted average onto the image plane.Stability metrics can be defined as the standard deviation of the tilt angle, the displacement amplitude (maximum-minimum) of the center of gravity trajectory, or the mean square of velocity within a time window. A moving average is applied to the angle and displacement sequences to suppress jitter. When the tilt angle exceeds a threshold or the stability metrics exceed a threshold for several frames, abnormal movements such as tilting or swaying of the torso are determined, and the tilt angle, center of gravity drift, and stability score are output. The joint motion determination unit is used to analyze joint motion patterns (such as flexion-extension, rotation, abduction-adduction, etc.) based on temporal data of human key points. Specifically, a joint angle / displacement sequence (e.g., knee joint angle changes over time) can be constructed from adjacent bone segments, and the sequence can be smoothed and denoised. Then, type discrimination is performed based on the sign and periodicity of the angle changes (e.g., periodic increases or decreases in angle correspond to flexion-extension, and periodic changes in hip / shoulder lateral displacement correspond to abduction-adduction). The motion amplitude can be taken as the peak-to-peak value of the angle / displacement within the time window, and the motion frequency can be obtained through peak counting (number of peaks per unit time) or FFT main frequency estimation.Coordination can be measured by indicators such as the phase difference and correlation coefficient / cross-correlation peak of the left and right symmetrical joints. When the indicators fall below a threshold, it indicates a decline in coordination. The system outputs joint movement type, amplitude, frequency, and coordination characteristics to assess joint flexibility and motor function. The multimodal algorithm chain includes a semantic understanding unit, an action alignment unit, a temporal alignment optimization unit, and a context-aware unit. The semantic understanding unit parses natural language instructions based on text and visual language understanding results. For example, it understands the meaning of "quickly and continuously tap your fingers 10 times," extracting key information: action type (tapping), action part (fingers), action requirements (quick, continuous), and number of actions (10 times). This information is then extracted and transformed into structured evaluation parameters. The data (such as tapping frequency threshold, duration, number of times required, etc.) is processed as follows: First, the instruction text is cleaned and standardized, completing synonym normalization and numerical unit parsing; then, a rule template or lightweight semantic model is used for intent recognition and slot extraction, extracting key fields such as action type, location, number of times, speed requirement, continuity requirement, left and right sides; then, the extracted slots are mapped to evaluation parameters that the algorithm can execute (such as frequency threshold, interruption duration threshold, target number of times, etc.), and if no parameters are specified, the default values ​​of the scenario are loaded; optionally, consistency verification is performed with the visual language understanding results, and when there is a conflict between the text and visual semantics, a high-confidence source is selected according to the configuration strategy and the decision basis is recorded; finally, a structured evaluation is output. Parameters are provided for subsequent action alignment and scoring logic. The action alignment unit is used to perform temporal and spatial alignment between the actual action sequence and the standard action sequence based on multimodal fusion features and standard action templates. Temporal alignment uses Dynamic Time Warping (DTW) or a deep learning-based alignment model to align the actual action sequence with the standard action sequence in time and find the corresponding time points. Spatial alignment uses keypoint matching or feature-based alignment methods to align the keypoint positions of the actual action with the keypoint positions of the standard action, eliminating differences in position, scale, and perspective. The similarity between the aligned actual action and the standard action is calculated to evaluate the action completion and quality. Similarity can be based on keypoint distance, etc. The calculation is performed across multiple dimensions, including motion trajectory and velocity curve. The temporal alignment optimization unit optimizes the temporal synchronization of multimodal input data based on the temporal alignment results of the fundamental model inference layer. It can handle issues such as inconsistent sampling rates and different delays of different modal data, ensuring adaptation to modal data with different sampling rates. For example, it aligns 30fps video frames with 100Hz sensor data to ensure that each video frame can find corresponding sensor data. The context awareness unit analyzes the continuity of actions and identifies action intentions and abnormal actions based on multimodal fusion features and temporal information. Among them, the action continuity analysis analyzes the continuity of action sequences, identifies the start, progress, and end stages of actions, and understands the complete process of actions.Intent recognition identifies the patient's action intent based on contextual information (such as previous actions, environmental state, patient status, etc.) and determines whether the action meets the assessment requirements. Anomaly detection detects abnormal actions (such as interrupted actions, incorrect actions, incomplete actions, etc.) based on contextual information and provides anomaly prompts and feedback.

[0026] This invention demonstrates how to construct parallel algorithm chains for hand recognition, human pose recognition, and multimodal recognition, and configure specialized functional units within each chain for refined processing. This transforms the standardized inference results output from the fundamental model inference layer into high-precision, multi-dimensional quantitative features and structured algorithm results for medical motion assessment. This provides the upper-level scenario strategy layer with directly callable general motion recognition capabilities, resulting in significant overall technical effectiveness and meeting the needs of medical assessment. Specifically, the hand algorithm chain relies on a finger pinching determination unit to accurately determine the pinching and opening states of fingers through Euclidean distance calculation of key hand points, outputting motion speed and duration features. The palm rotation determination unit is based on the palm surface... The normal vector and orientation angle changes enable stable determination of hand flipping motions. After suppressing jitter with a moving average, the flipping angle, angular velocity, and other features are output. The hand keypoint tracking unit completes the association of keypoints in consecutive frames through nearest neighbor matching and the Hungarian algorithm. Combined with Kalman filters, it achieves smooth position prediction and trajectory continuation in occlusion and missing point scenarios. Then, differential calculation is used to obtain the temporal features of keypoint motion trajectory, velocity, and acceleration. This effectively solves the problems of tracking breakage and data jitter in fast hand movements and keypoint occlusion, and achieves full-dimensional accurate quantitative analysis of fine hand movements. In the human posture algorithm link, the limb angle determination unit is based on the calculation of human skeletal keypoints. The joint angles are analyzed using filtering and outlier processing to output joint angles, rates of change, and range characteristics. The trunk stability assessment unit calculates tilt angles, drift, and stability indices based on the trunk axial vector and center of gravity projection, accurately identifying abnormal movements such as trunk tilting and swaying. The joint motion assessment unit analyzes joint angles and displacement sequences to identify flexion, extension, and rotation patterns, calculates motion amplitude and frequency, and measures bilateral joint coordination. This comprehensive and professional quantitative analysis of human limb posture, trunk stability, and joint motion function meets the objective and refined needs of limb function assessment in medical rehabilitation. The semantic understanding unit of the multimodal algorithm link can interpret natural language action commands. The system performs cleaning and parsing, slot extraction and parameter mapping, and combines visual semantic verification to output structured evaluation parameters, achieving accurate conversion of natural language instructions into algorithm evaluation logic. The action alignment unit eliminates the temporal, scale and perspective differences between actual actions and standard templates through DTW temporal alignment and key point spatial matching, and calculates action similarity in multiple dimensions to evaluate the completion quality. The temporal alignment optimization unit further adapts to modal data with different sampling rates to solve the problem of temporal misalignment caused by different sampling rates and transmission delays. The context awareness unit analyzes action coherence, identifies action intent and detects abnormal situations such as action interruption and errors based on temporal and fusion features, and provides intelligent prompts and feedback.The three major algorithmic pathways process standardized data with unified timestamps in parallel and collaboratively. The output quantified features, evaluation parameters, action alignment results, and anomaly markers all exhibit high consistency and traceability. This achieves full-scale action analysis, from fine hand movements to overall human posture, and completes deep matching, temporal optimization, and context awareness of multimodal semantics and actual actions. This significantly improves the fineness, accuracy, and intelligence of action recognition. Furthermore, the standardized output format can be flexibly combined and invoked by the scenario strategy layer as needed, providing robust and reliable mid-level algorithmic support for action function determination, rehabilitation scoring, and anomaly feedback in different medical assessment scenarios.

[0027] The application algorithm layer also includes an enhanced algorithm function unit, which comprises an action template management subunit, an interpretability analysis subunit, a fault tolerance processing subunit, and a federated training subunit. The action template management subunit automatically extracts action templates from standard action data, supporting incremental template updates and the maintenance of multi-scene, multi-difficulty template libraries. Specifically, it includes: Template extraction: extracting keypoint sequences, motion trajectories, velocity curves, angle changes, and other features from standard action videos as action templates; Template updates: supporting incremental template updates, automatically updating templates when new standard action data is available, improving template accuracy and diversity; Template management: maintaining multiple action template libraries, supporting action templates for different scenes and difficulty levels. The interpretability analysis subunit analyzes the contribution of each algorithm function to the evaluation results, performing feature importance analysis, decision path tracing, and output result confidence assessment. Feature importance analysis: identifying which keypoints, features, and modal data have the greatest impact on the evaluation results; Decision path tracing: tracing the decision path from input data to the final evaluation result, recording the output and impact of each algorithm function. Confidence Assessment: Evaluates the confidence level of each algorithm's output, providing a credibility index for the final evaluation result; Fault Tolerance Subunit: Implements fault tolerance mechanisms when algorithm functionality or data is abnormal. These mechanisms include backup algorithm switching, modality degradation processing, and abnormal result repair and completion operations. Backup Algorithm: When the main algorithm fails, it automatically switches to a backup algorithm (e.g., using simplified judgment rules). Failures may include: empty output, overall keypoint confidence level below the overall confidence threshold, failure to detect the target for several consecutive frames, or significant conflict between the output and physical constraints; After switching, more robust but less precise rules are adopted (e.g., judgment based on keypoint distance thresholds or angle thresholds), and the reason for this switch and the continuous frame segment are recorded; Degradation Processing: When a modality's data is abnormal, only data from other normal modalities are used for processing. Specifically, based on data quality assessment and anomaly labeling, the "set of available modalities" is determined. The weights of low-quality / abnormal modalities are reduced or set to zero, and the corresponding degradation path is selected (such as visual keypoints only / IMU only / text parameter-driven assessment only) to ensure that core indicators (number of times, frequency, amplitude, etc.) can still be output. At the same time, the "degradation mode" and the affected indicators are marked in the results. Result repair: Based on historical data and contextual information, abnormal or missing results are repaired and completed.The following methods can be employed: linear / spline interpolation for short missing segments, and moving average smoothing for jittery segments; alignment and completion of key event results (start / end, first count) combined with action templates and trends from neighboring windows; outputting missing markers and providing the missing percentage for irreparable segments to ensure traceability and interpretability; a federated training subunit to support federated learning training of various algorithm functions in the application algorithm layer, enabling cross-institutional algorithm parameter optimization and knowledge sharing while protecting data privacy. Specifically, this includes: federated training of algorithm parameters: each institution trains the parameters of its algorithm functions locally using its own data (such as the threshold for finger pinch judgment, the similarity threshold for action alignment, etc.), only uploading parameter updates to the central server for aggregation. Privacy-preserving algorithm optimization: differential privacy and homomorphic encryption technologies are used during algorithm optimization to ensure that patient data privacy is not leaked. Cross-institutional knowledge sharing: through federated learning, institutions can share knowledge of algorithm optimization, improving the generalization ability and accuracy of the algorithm without sharing original patient data. The application algorithm layer outputs the results of the above algorithm functions (judgment results, feature values, state information, semantic understanding results, action alignment results, context information, action templates, interpretability analysis results, fault tolerance processing labels, federated learning parameters, etc.) to the scene policy layer in a unified data format, without caring about which specific scenarios these functions are used in.

[0028] This invention, through the addition of an enhanced algorithm functional unit comprised of an action template management subunit, an interpretability analysis subunit, a fault-tolerant processing subunit, and a federated training subunit, further improves the mid-level algorithm processing capabilities from four dimensions: template standardization, result interpretability, operational fault tolerance robustness, and privacy and security collaborative optimization. This enables the system to possess stronger versatility, stability, and compliance in complex medical assessment scenarios, while simultaneously providing the scenario strategy layer with comprehensive, reliable, and traceable standardized algorithm outputs: the action template management subunit can automatically extract key point sequences, motion trajectories, and speeds from standard action data. The system constructs action templates based on core features such as curve and angle changes, supports incremental template updates driven by new data, and maintains a motion template library adapted to multiple scenarios and difficulty levels. This effectively improves the accuracy and adaptability of the templates, providing a unified and standardized reference for the comparison, alignment, and evaluation of actual actions. The interpretability analysis subunit accurately identifies key points, feature dimensions, and modal data that play a crucial role in the evaluation results through feature importance analysis, decision path tracing, and result confidence assessment. It fully records the entire decision-making logic from input data to final judgment, while quantifying the reliability of the output of each algorithm function. This ensures clear traceability and interpretability of the action evaluation process and results, fully meeting the stringent requirements of the medical field for rigor and transparency in evaluation. The fault-tolerant processing subunit constructs a multi-level fault-tolerant mechanism encompassing backup algorithm switching, modality degradation processing, and abnormal result repair and completion. It automatically switches to robust backup rules when the main algorithm fails, key point confidence is insufficient, target detection is lost, or data conflicts occur. For abnormal modal data, it implements weight reduction or link degradation processing. Based on historical data and contextual information, it performs interpolation smoothing and trend completion on missing and jittery data. For missing paragraph marker information that cannot be repaired, it ensures... This ensures stable output of core evaluation indicators such as the number of actions, frequency, and amplitude, while also fully recording anomalies and their handling, significantly improving the system's continuous operation capability under complex scenarios such as poor data quality and equipment fluctuations. The federated training subunit leverages federated learning to optimize functional parameters such as algorithm judgment thresholds and similarity parameters across institutions. After local training at each institution, only encrypted parameters are uploaded to the cloud for aggregation. Combined with differential privacy and homomorphic encryption technologies, the system strictly protects the privacy of patients' original data. While achieving cross-institutional knowledge sharing and improving algorithm generalization ability and recognition accuracy, it fully complies with medical data privacy compliance requirements. Finally, the application algorithm layer encapsulates and outputs diverse content such as action judgment results, feature values, status information, template data, interpretability analysis results, fault-tolerant tags, and federated learning parameters in a unified format. This decouples the mid-layer general algorithm from the upper-layer scenario application, comprehensively enhancing the system's algorithm robustness, evaluation standardization, deployment flexibility, and privacy security. It provides solid and reliable mid-layer support for the scenario strategy layer to flexibly call algorithm functions and conduct professional medical action evaluations.

[0029] The multimodal fusion inference chain also includes a multimodal cross-attention fusion module and a visual language understanding module. The multimodal cross-attention fusion module uses a multimodal fusion model (such as a fusion model based on cross-attention mechanisms) to fuse temporally aligned visual, text, speech, and sensor modal features to generate a unified multimodal feature representation. This model can learn the relationships between different modalities, such as the correspondence between the text command "rapid tapping" and the visual representation of rapid finger movement. The multimodal fusion model includes a modal projection layer, intramodal encoding units, cross-modal cross-attention units, and gated weighting units. It first maps preprocessed and aligned visual, audio, IMU, and text features to a unified dimension through the projection layer and adds modal and positional embeddings to form a temporal sequence. The core adopts a "shared fusion unit + modal interaction" structure. First, temporal dependencies are captured through optional intramodal encoding. Then, cross-modal information interaction and feature enhancement are achieved through multi-head cross-attention. At the same time, gating coefficients are calculated based on the quality indicators of each modality to complete weighted fusion, achieving robust adaptation to missing and low-quality modalities. Subsequently, the output unified fused features are subjected to layer normalization and temporal pooling to output standardized fusion results with timestamps and interpretability-related weight coefficients. During model training, multimodal synchronous data is used as samples, including video, audio, IMU, and instruction text. A multi-task joint loss of action classification, regression, and contrastive learning alignment is adopted. The model's generalization and robustness are improved by freezing the single-modal encoder in a phased training and modality discarding strategy. During the inference stage, feature fusion is completed according to a fixed process. The gating coefficients of low-quality / missing modalities can be dynamically adjusted and relevant information is retained for subsequent result traceability and interpretability analysis.The modal projection layer maps features from different modalities to a unified dimension. Cross-modal attention units capture the relationships between different modalities. Gated weighting units dynamically adjust the fusion weights based on the data quality scores of each modality, outputting standardized fusion features. The visual language understanding module uses pre-trained visual language models (such as CLIP and BLIP) to perform semantic understanding of text instructions and semantic alignment of text-visual features. It locates action patterns and action segments matching the text instructions in visual data, outputting text-visual matching scores, action segment localization results, and key region heatmaps. First, a text-visual pair is constructed: the text side consists of the cleaned and encoded instruction text, and the visual side consists of normalized image frames and optional hand / human ROI or keypoint rendering maps. A dual-tower encoder (visual...) is used... The model employs ViT / CNN for visual information and Transformer for text, encoding both into a shared space. Cosine similarity or dot product similarity is calculated and temporally aggregated to obtain the semantic-visual segment matching degree. A matching degree curve is calculated on consecutive frames using a sliding window, and peak intervals are selected for segment-level action localization. An attention mechanism can be combined to output a heatmap of key regions for region-level localization, which is cross-validated with detection results to suppress background interference. The output includes structured results such as matching degree, aligned segments, and heatmaps. The model uses pre-trained weights such as CLIP / BLIP as its foundation, fine-tuned through domain-specific text-image pairs (contrastive learning or text-image matching loss), and utilizes fine-grained negative samples to improve semantic discrimination capabilities. Ultimately, it provides semantic alignment priors for multimodal fusion and application algorithm layers.

[0030] This invention achieves refined fusion and semantic-level association of multi-source heterogeneous features from vision, text, speech, and sensors through cross-modal deep feature interaction and image-text semantic alignment. This significantly improves the system's accuracy in understanding action commands, its ability to locate action patterns, and its robustness in complex scenarios. The multi-modal cross-attention fusion module employs a fusion model including a modal projection layer, intra-modal coding units, cross-modal cross-attention units, and gated weighting units. It first maps the temporally aligned features of each modality to a unified dimension and adds modality and position embeddings to form a temporal sequence. Intramodal encoding captures temporal dependencies, and multi-head cross-attention enables cross-modal information interaction and feature enhancement. Simultaneously, it dynamically calculates gating coefficients based on data quality indicators for each modality to achieve weighted fusion, enabling adaptive handling of missing or low-quality modalities. After layer normalization and temporal pooling, standardized fusion features with unified timestamps and interpretable weight coefficients are output. During training, the model employs a multi-task joint loss aligned with action classification, regression, and contrastive learning, and utilizes staged training and modality dropout strategies to enhance generalization ability. It can autonomously learn the relationships between textual commands, visual motion, and sensor changes. The inherent correlation effectively weakens the interference of inferior or missing modalities on the fusion results. The visual language understanding module is based on visual language pre-trained models such as CLIP and BLIP. It uses a dual-tower encoder to encode the cleaned and encoded text instructions, normalized visual frames, hands, and human regions of interest into a shared feature space. It calculates cosine similarity and performs temporal aggregation to obtain the image-text matching degree. Then, it completes the localization of action segments and outputs the heat map of key regions through a sliding window and attention mechanism. At the same time, it improves semantic discrimination ability by leveraging domain fine-tuning and fine-grained negative samples, and accurately realizes the semantic parsing of text instructions and the alignment and matching of text and visual actions. The two modules work together to achieve deep complementarity and adaptive weighted fusion of multimodal features, enriching the dimensions of action features, and realize accurate matching and localization of natural language instructions to visual actions. It effectively solves problems such as weak cross-modal correlation, semantic understanding bias, and background interference, and significantly improves the system's perception and parsing level of instructional and refined actions in medical scenarios. It provides highly consistent and reliable multimodal semantic fusion feature support for action judgment, alignment evaluation, and professional medical evaluation at the scene strategy layer.

[0031] The scenario strategy layer includes a scenario-based assessment execution module and an enhanced scenario function module. The scenario-based assessment execution module, based on the target medical assessment scenario, combines and calls the corresponding general algorithm functions of the application algorithm layer to execute scenario-specific action assessment logic. Target medical assessment scenarios include finger tapping assessment scenarios, palm movement assessment scenarios, alternating hand movement assessment scenarios, leg flexibility assessment scenarios, toe tapping assessment scenarios, and natural language command-based custom assessment scenarios. The finger tapping scenario combines and calls the functions of "hand key point tracking," "finger pinching judgment," "semantic understanding," "action alignment," and "context awareness." The specific strategy is as follows: First, the "semantic understanding" function parses the natural language command (e.g., "quickly and continuously tap your fingers 10 times") to extract assessment parameters (tapping frequency threshold, number of taps, etc.). Then, it tracks the movement trajectory of the index finger key points, detecting whether the index finger is performing rapid up-and-down tapping movements within the specified area; simultaneously, it detects the finger pinching judgment result to ensure the fingers remain open during the tapping process. Next, the "action alignment" function aligns and compares the actual tapping action with a standard tapping action template to assess the standardization of the action. Finally, the "context-aware" function analyzes the continuity and completeness of the slapping action, identifying the start, progress, and end stages of the action, and determining whether the specified number of slaps has been completed. Based on features such as slapping frequency, amplitude, duration, action alignment similarity, and completion, an evaluation score for finger slapping is output. For hand movement scenarios: the "hand rotation judgment" and "hand key point tracking" functions are combined. The specific strategy is to detect whether the hand has completed the specified rotation action (e.g., rotating from palm up to palm down and then back), analyzing the completion, speed, and smoothness of the rotation; simultaneously, the movement trajectory of the hand is tracked to ensure the movement path meets the requirements. Based on the completion of the rotation and the quality of the movement, an evaluation score for the hand movement is output. For alternating action scenarios: the "finger pinching judgment," "hand rotation judgment," and "hand key point tracking" functions are combined. The specific strategy is as follows: First, based on the key point tracking results of the left and right hands, generate state sequences for the left and right hands respectively (such as "clenched fist / open", "palm up / down", etc.), and extract the time point of each "state switch" through state machine or event detection; then determine whether the alternation meets the expected pattern (such as left fist clenched and right open → left open and right fist clenched → loop), and count the number of effective alternations and the number of missed / false detections; on this basis, calculate indicators such as alternation rhythm (cycle mean and variance), synchronicity (time difference / phase difference of left and right hand state switching), and coordination (degree of complementarity of the two hand states, correlation coefficient / mutual information, etc.), and filter out jittering switching caused by short-term occlusion (minimum continuous frame threshold). Based on the comprehensive score of the effective alternation count, rhythm stability, synchronicity and coordination, output the evaluation score of the alternation action and the main reasons for deduction (such as unstable rhythm, asynchrony, and alternation interruption). Leg flexibility scenario: Combine the "limb angle determination" and "joint movement determination" functions.The specific strategy is as follows: First, extract the "leg lift-leg drop" event segment based on the hip / knee joint angle sequence and key point trajectory (e.g., determine the start / peak / end of each leg lift through angle peak detection or threshold cross-detection), and smooth the angle sequence and remove outliers; then calculate the leg lift amplitude (peak-to-trough difference / maximum lift angle), speed (peak / average rate of angle change), frequency (number of leg lifts per unit time or dominant frequency), stability (period variance, amplitude variance), and smoothness (degree of jiggling in angular velocity / angular acceleration), and evaluate the coordination of the left and right legs (phase difference, correlation coefficient, amplitude symmetry) when both legs are involved simultaneously. Based on a comprehensive score considering whether the amplitude meets the standard, whether the rhythm is stable, and the smoothness and coordination, output the evaluation score of leg flexibility and the main reasons for deduction (e.g., insufficient amplitude, unstable rhythm, obvious jiggling, and lack of coordination between left and right). For the toe-slapping scenario: combine the "joint motion determination" and "key point tracking" functions (requires the base model inference layer to support foot key point recognition). The specific strategy is as follows: track the movement trajectory of key points on the toes, detect whether the toes are making rapid up-and-down slapping movements within a designated area, and analyze the frequency, amplitude, and duration of the slapping. Based on the quality of the toe slapping, an evaluation score is output. The scenario strategy layer ultimately outputs the evaluation results for each scenario (such as evaluation score, completion rate, abnormal indicators, action alignment similarity, semantic understanding results, etc.) for use by the medical diagnostic system. Natural language instruction parsing scenarios: The scenario strategy layer supports defining evaluation strategies through natural language descriptions. For example, if a user inputs "Please evaluate whether the patient can quickly and continuously slap their fingers 10 times, with each slap amplitude greater than 5 centimeters," the system processes it through the following process. Enhanced scenario function modules include an adaptive evaluation adjustment unit, a real-time feedback guidance unit, an interpretable report generation unit, and a multilingual support unit; the adaptive evaluation adjustment unit is used to dynamically adjust evaluation parameters and evaluation difficulty based on the patient's historical evaluation data, disease progression stage, and individual characteristics to generate personalized evaluation strategies; it analyzes the patient's historical evaluation data to identify the patient's baseline motor ability, disease severity, age, and other individual characteristics. Based on patient records and current status, assessment parameters (such as tapping frequency thresholds, required movement amplitude, and completion time limits) are dynamically adjusted to achieve personalized assessment strategies. The assessment difficulty is gradually increased according to the patient's ability level, achieving progressive rehabilitation assessment. A real-time feedback guidance unit analyzes the gap between the current movement and the standard movement based on real-time assessment results, generating multimodal real-time feedback and movement guidance; it also analyzes the gap between the current movement and the target movement based on the real-time assessment results from the application algorithm layer. Combined with a semantic understanding module, the assessment results are transformed into natural language feedback that is easy for patients to understand (such as "the movement amplitude needs to be larger," "the speed can be faster," "the movement is completed very well"). Multimodal feedback supports various forms of feedback, including text, voice, and visual prompts, improving the user experience.Based on feedback, guidance strategies are continuously optimized to improve the success rate and accuracy of assessments. The interpretability report generation unit generates interpretable medical assessment reports that include visualizations, assessment criteria, natural language assessment conclusions, and improvement suggestions. Visualizations include key point annotation maps, action trajectory maps, action alignment comparison maps, and time-series feature curves. The assessment criteria, combined with the results of the interpretability analysis module, explains the assessment criteria (e.g., which key points, features, and modal data influenced the assessment results). The natural language report, combined with the semantic understanding module, generates assessment reports in natural language, including assessment results, improvement suggestions, and anomaly alerts. Multilingual reports support the generation of assessment reports in multiple languages ​​to adapt to the needs of different regions and language environments. The multilingual support unit supports multilingual natural language instruction parsing, feedback guidance output, and assessment report generation. The semantic understanding module supports multilingual instruction parsing, capable of understanding assessment instructions in different languages. The feedback generation module supports multilingual feedback output, providing guidance in a language familiar to patients. The report generation module supports multilingual report output, providing assessment results in a language familiar to physicians. The scenario strategy layer ultimately outputs the evaluation results for each scenario (such as evaluation score, completion rate, anomaly indicators, action alignment similarity, semantic understanding results, interpretability report, real-time feedback, adaptive parameter adjustment, etc.) for use by the medical diagnostic system.

[0032] This invention, through the synergistic linkage of a scenario-based assessment execution module and an enhanced scenario function module, achieves a precise transformation from general algorithm functions to professional and personalized medical action assessment. This not only meets the differentiated needs of various medical assessment scenarios but also enhances the practicality, interpretability, and adaptability of the assessment process. It provides comprehensive, accurate, and easy-to-use action assessment support for medical diagnostic systems. The overall technical effect aligns with the core requirements of medical rehabilitation assessment and offers significant advantages: The scenario-based assessment execution module can flexibly combine and call the corresponding general algorithm functions of the application algorithm layer to execute the scenario-specific assessment logic based on various target medical assessment scenarios, such as finger tapping, hand movement, alternating hand movements, leg flexibility, toe tapping, and natural language commands. For example, in the finger tapping scenario, semantic understanding and key point tracking functions are integrated to parse commands, track actions, and assess standardization and coherence; in the alternating action scenario, functions such as pinching and flipping judgments are combined to analyze alternation patterns, rhythm, and coordination; and in the leg flexibility scenario, limb angle and joint movement judgment functions are used to quantify leg lifting amplitude, speed, and coordination. Each scenario can accurately extract core assessment indicators such as action frequency, amplitude, and completion. The system outputs structured assessment results including scores, causes of anomalies, and alignment similarity, effectively addressing the issue of insufficient adaptability of general algorithms to specific medical scenarios. This achieves precision and standardization in the assessment of different medical actions. The enhanced scenario function module further improves the personalization, usability, and traceability of the assessment. The adaptive assessment adjustment unit dynamically adjusts assessment parameters and difficulty based on the patient's historical assessment data, disease progression, and individual characteristics, generating personalized and progressive rehabilitation assessment strategies to adapt to the different patients' motor ability baselines. The real-time feedback guidance unit, relying on real-time assessment results, transforms movement gaps into multimodal feedback and guidance in text, voice, and visualization, helping patients adjust their movements in a timely manner and improving the success rate of assessments and the effectiveness of rehabilitation training. The interpretable report generation unit generates assessment reports including key point annotation maps, movement trajectory maps, assessment basis, natural language conclusions, and improvement suggestions. Combined with interpretability analysis results, it clarifies the key factors affecting the assessment, meeting the stringent requirements of the medical field for the rigor and traceability of assessments. The multilingual support unit enables multilingual instruction parsing, feedback output, and report generation, adapting to the usage needs of different regions and language environments. Overall, the scenario strategy layer achieves precise alignment between general algorithm functions and medical scenario requirements through the deep integration of scenario-based accurate assessment and enhanced functions. This ensures the professionalism, accuracy, and standardization of action assessment while improving the usability, adaptability, and compliance of the assessment process through personalized adjustments, real-time feedback, interpretable reports, and multilingual support. The resulting comprehensive and reliable assessment results can be directly used by medical diagnostic systems, providing solid and reliable decision support for medical rehabilitation assessment, disease diagnosis, and rehabilitation program development. This significantly enhances the practicality and application value of the entire action recognition platform in the medical field.

Claims

1. A system for constructing an action recognition algorithm platform based on multi-algorithm fusion, characterized in that, The system includes a schema inference layer, an application algorithm layer, and a scenario strategy layer: The base model inference layer is used to receive multimodal input data, perform model inference processing on the multimodal input data through multiple inference links running in parallel, and output standardized inference results with unified timestamps. The multimodal input data includes visual, text, speech, and sensor data. The multiple inference links include a hand inference link for detecting and recognizing hand regions and key points, a human posture inference link for detecting and recognizing human body regions and posture key points, and a multimodal fusion inference link for fusing different modal data. The application algorithm layer is communicatively connected to the fundamental model inference layer. It is used to receive the standardized inference results with unified timestamps output by the fundamental model inference layer. After aligning the multi-source data based on the unified timestamps, it executes multiple general-purpose algorithm functions in parallel and outputs the standardized algorithm function results with unified timestamps. The multiple general-purpose algorithm functions include hand algorithm chain, human posture algorithm chain, and multimodal algorithm chain. The scenario strategy layer is communicatively connected to the application algorithm layer. It is used to receive the standardized algorithm function results with a unified timestamp output by the application algorithm layer, combine and call the corresponding general action recognition algorithm function according to the business requirements of the target medical assessment scenario, execute the action assessment logic that matches the target medical assessment scenario, and finally output the action assessment result under the medical assessment scenario.

2. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The hand inference chain includes a hand target detection module and a hand key point recognition module; The hand target detection module is used to process the input visual data using a hand target detection model, and output the bounding box coordinates and confidence scores of the hand region according to the target detection format; the hand detection model is a target detection network optimized for small hand targets and close-range scenes; The hand key point recognition module is used to detect key points based on the detected hand region using a hand key point recognition model, and output the 2D / 3D coordinate information and confidence level of the hand key points; the hand key point recognition model is a deep learning network that outputs hand key points to achieve hand pose estimation.

3. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The human pose inference chain includes a human target detection module, a human pose estimation module, and a key point standardization processing module; The human target detection module is used to process the input visual data using a human target detection model and output the bounding box coordinates and confidence score of the human body region. The human target detection model is a deep learning network that outputs the human body position to achieve human body localization; The human pose estimation module is used to estimate the pose based on the detected human body region using a human pose estimation model, and output the 2D / 3D coordinate information and confidence of the human skeleton key points; the human pose estimation model outputs the human skeleton key points to realize a deep learning network for human pose parsing. The key point standardization processing module is used to perform format standardization processing, missing and anomaly handling, and integrity verification on the output human skeleton key points, and output a standardized human key point data structure.

4. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The multimodal fusion inference link includes a multimodal input data preprocessing module, a unified timestamp synchronization module, and a standardized encapsulation module; The multimodal input data preprocessing module is used to preprocess, control the quality and unify the format of visual data, text data, voice data and sensor data respectively, and generate standardized feature data corresponding to each modality. The unified timestamp synchronization module is used to perform time-series synchronization of data in different modalities using a time-series alignment model. The standardized encapsulation module uses a multimodal fusion model to fuse multimodal input data, generate a unified multimodal feature representation, and output it to the subsequent time-series alignment and fusion processing stage.

5. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The basic inference layer also includes an enhancement module, which includes a data quality assessment unit, a multimodal fusion weight adaptive unit, an anomaly detection unit, a cross-scenario knowledge transfer unit, an edge-cloud collaboration unit, and a federated learning privacy protection unit. The data quality assessment unit is used to assess the quality of the input data for each modality and output the corresponding modality's data quality score, including video clarity score, sensor data reliability score, and speech clarity score. The multimodal fusion weight adaptive unit is used to dynamically adjust the fusion weight of each modality in the multimodal fusion model based on the data quality score of each modality. The anomaly detection unit is used to detect anomalies in each modal data, including action interruption, equipment failure, data loss, and abnormal data values. When any data anomaly is detected, the abnormal modality and anomaly type are marked. The cross-scenario knowledge transfer unit is used to identify common action features between different medical assessment scenarios and transfer the model knowledge of the trained scenarios to the new scenarios through transfer learning. The edge-cloud collaboration unit is used to deploy lightweight inference models on edge devices to perform real-time inference, and deploy complex models in the cloud to perform computation, according to the model complexity and real-time requirements. The federated learning privacy protection unit uses differential privacy technology to add noise during training and combines it with homomorphic encryption to encrypt parameters. Through local training at each medical institution, only the encrypted parameters are uploaded to the central server for federated averaging and aggregation, realizing distributed training and incremental updates without data leaving the local area. The inference results containing the parameters of the federated learning model are output to the application algorithm layer in a unified data format.

6. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The system has standardized rules for the inter-layer flow and timing alignment of multimodal input data. The standardized rules include inter-layer data synchronization rules and timing alignment adaptation rules. The inter-layer data synchronization rules include: after the base model inference layer completes the inference of each link, it encapsulates the output of each link into a data packet with a unified timestamp and transmits it to the application algorithm layer through the inter-layer interface. The synchronization methods include push mode, pull mode and synchronous call mode. The timing alignment and adaptation rules include: using a unified time base as a reference, setting timestamp synchronization precision for each modal data to match business requirements; and adopting multiple methods to achieve timing alignment for multimodal input data with different sampling rates to ensure the timing consistency of multimodal features. The multiple methods include interpolation processing, resampling processing, and event alignment processing.

7. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The hand algorithm link, human pose algorithm link, and multimodal algorithm link of the application algorithm layer each include the following functional units: The hand algorithm chain includes a finger pinching determination unit, a palm flipping determination unit, and a hand key point tracking unit; the finger pinching determination unit is used to calculate the Euclidean distance between fingers based on the coordinates of hand key points, determine the pinching / opening state of the fingers, and output the speed and duration characteristics of the pinching action. The palm flipping determination unit is used to calculate the palm surface normal vector and orientation angle based on the hand key point recognition results, determine the palm flipping action, and output the flipping angle, angular velocity, and completion characteristics. The hand key point tracking unit is used to perform temporal tracking and trajectory continuation of hand key points in consecutive frames, and output the motion trajectory, velocity, and acceleration temporal features of the key points. The human posture algorithm chain includes a limb angle determination unit, a trunk stability determination unit, and a joint motion determination unit; the limb angle determination unit is used to calculate the included angle of the target limb joint based on the key points of the human skeleton, and output the joint angle value, angle change rate, and angle range characteristics. The trunk stability determination unit is used to calculate the trunk tilt angle, center of gravity position and stability index based on the human key point recognition results, determine the abnormal movement state of the trunk, and output the tilt angle, center of gravity drift amount and stability score features. The joint motion determination unit is used to analyze joint motion patterns based on human key point time-series data and output joint motion type, amplitude, frequency, and coordination characteristics. The multimodal algorithm chain includes a semantic understanding unit, an action alignment unit, a temporal alignment optimization unit, and a context-aware unit. The semantic understanding unit is used to parse natural language instructions based on text and visual language understanding results, extract and convert them into structured evaluation parameters. The action alignment unit is used to perform temporal and spatial alignment between the actual action sequence and the standard action sequence based on multimodal fusion features and standard action templates, calculate the similarity between the aligned actual action and the standard action, and evaluate the action completion degree and quality. The temporal alignment optimization unit is used to optimize the temporal synchronization of multimodal input data based on the temporal alignment results of the base model inference layer, and adapt to modal data with different sampling rates. The context-aware unit is used to analyze action coherence and identify action intent and abnormal actions based on multimodal fusion features and temporal information.

8. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 7, characterized in that, The application algorithm layer also includes an enhanced algorithm function unit, which includes an action template management subunit, an interpretability analysis subunit, a fault tolerance processing subunit, and a federated training subunit. The action template management subunit is used to automatically extract action templates from standard action data, and supports incremental updates of templates and maintenance of template libraries with multiple scenarios and difficulties. The interpretability analysis subunit is used to analyze the contribution of each algorithm function to the evaluation results, and to perform feature importance analysis, decision path tracing and output result confidence evaluation. The fault tolerance processing subunit is used to execute a fault tolerance mechanism when the algorithm function is abnormal or the data is abnormal. The fault tolerance mechanism includes switching to a backup algorithm, modal degradation processing, and abnormal result repair and completion operations. The federated training subunit is used to support federated learning training of various algorithm functions in the application algorithm layer, and to achieve cross-institutional algorithm parameter optimization and knowledge sharing while protecting data privacy.

9. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The multimodal fusion inference pipeline also includes a multimodal cross-attention fusion module and a visual language understanding module; The multimodal cross-attention fusion module is used to fuse temporally aligned visual, text, speech, and sensor modal features using a multimodal fusion model to generate a unified multimodal feature representation. The multimodal fusion model includes a modal projection layer, an intramodal coding unit, a cross-modal cross-attention unit, and a gated weighting unit. The modal projection layer maps each modal feature to a unified dimension, the cross-modal cross-attention unit captures the correlation between different modalities, and the gated weighting unit dynamically adjusts the fusion weights based on the data quality scores of each modality to output standardized fusion features. The visual language understanding module is used to employ a visual language pre-trained model to complete the semantic understanding of text instructions and the semantic alignment of text-visual features, locate the action patterns and action segments that match the text instructions in the visual data, and output the text-visual matching degree, action segment localization results, and key region heatmaps.

10. The action recognition algorithm platform construction system based on multi-algorithm fusion as described in claim 1, characterized in that, The scenario strategy layer includes a scenario-based evaluation and execution module and a scenario-enhancing function module; The scenario-based assessment execution module is used to combine and call the general algorithm functions corresponding to the application algorithm layer according to the target medical assessment scenario, and execute the scenario-specific action assessment logic. The target medical assessment scenarios include finger tapping assessment scenario, palm movement assessment scenario, alternating hand movement assessment scenario, leg flexibility assessment scenario, toe tapping assessment scenario, and natural language command-customized assessment scenario. The enhanced scenario function module includes an adaptive evaluation and adjustment unit, a real-time feedback guidance unit, an interpretable report generation unit, and a multi-language support unit; The adaptive assessment adjustment unit is used to dynamically adjust the assessment parameters and assessment difficulty based on the patient's historical assessment data, disease progression stage and individual characteristics, and generate a personalized assessment strategy. The real-time feedback guidance unit is used to analyze the gap between the current action and the standard action based on the real-time evaluation results, and generate real-time feedback and action guidance in multimodal form. The interpretability report generation unit is used to generate an interpretable medical assessment report that includes visual charts, assessment criteria, natural language assessment conclusions, and improvement suggestions. The multilingual support unit is used to support the parsing of natural language instructions in multiple languages, the output of feedback guidance, and the generation of evaluation reports.

Images