Rehabilitation motion quality optimization method and system based on visible light camera device

Abstract

This invention provides a method and system for optimizing the quality of rehabilitation movements based on a visible light camera, relating to the field of image processing technology. The invention acquires visible light video and extracts three-dimensional skeletal data from it; then, based on a pre-defined rehabilitation movement quality evaluation model, it processes the three-dimensional skeletal data to obtain the rehabilitation movement quality evaluation results. This invention designs a rehabilitation movement quality optimization method based on a visible light camera, enabling skeletal-based rehabilitation movement quality evaluation using a low-cost visible light camera, thus reducing rehabilitation costs for patients.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method and system for optimizing the quality of rehabilitation movements based on a visible light camera device. Background Technology

[0002] A Home-based Exercise Program (HEP) is a personalized rehabilitation plan tailored to each patient by a rehabilitation therapist. This plan includes a set of recommended exercises, which the patient follows, records daily progress in a log, and regularly visits the clinic for progress assessments. HEPs effectively alleviate problems such as the shortage of rehabilitation physician resources, the high cost of hospital treatment, and the high costs of frequent hospital visits, thus improving the quality of rehabilitation for patients. However, HEPs developed by rehabilitation therapists often lack activity monitoring and feedback, and most patients do not adhere to or are unable to follow the prescribed activities, preventing HEPs from achieving their intended effects. With the emergence and rapid development of computer vision systems and artificial intelligence technologies, automated monitoring and decision support for rehabilitation movements can be achieved, overcoming the challenges faced by traditional clinic and home rehabilitation programs.

[0003] Existing AI-based rehabilitation movement evaluation methods can be divided into two categories: machine learning-based manual feature methods and deep learning-based deep feature methods. Machine learning-based manual feature methods can intuitively understand the key attributes of movements and reflect the quality of rehabilitation movements, but they require sophisticated expert knowledge and tedious manual work. Furthermore, for each rehabilitation movement, machine learning-based manual feature methods require designing corresponding motion features and constructing feature representations, resulting in high labor costs and poor applicability. Deep learning-based deep feature methods use deep learning models to extract deep features from 3D skeletal data end-to-end for movement quality evaluation, avoiding the need to focus on complex environmental backgrounds in rehabilitation videos that might affect the learning and reasoning abilities of rehabilitation movement evaluation models. With the advent of depth-sensing cameras and motion capture systems such as Microsoft Kinect sensors and Intel RealSense, acquiring 3D skeletal data has become easier. Therefore, skeletal data-based movement quality evaluation methods dominate this research field. However, most existing skeletal-based methods use depth cameras with SDKs, motion capture systems, or inertial sensors to acquire 3D skeletal data and use it as input to the model, but depth cameras with SDKs are costly. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for optimizing the quality of rehabilitation movements based on visible light camera equipment, which solves the technical problem of high cost in existing artificial intelligence-based rehabilitation movement evaluation methods.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a method for optimizing the quality of rehabilitation movements based on a visible light camera device, comprising:

[0009] Acquire visible light video and extract 3D skeletal data from the visible light video;

[0010] The three-dimensional skeletal data is processed based on the pre-set rehabilitation movement quality evaluation model to obtain the rehabilitation movement quality evaluation results.

[0011] The pre-defined rehabilitation movement quality evaluation model includes a batch normalization layer, an embedding layer, two stacked residual graph attention modules based on graph neural networks, a spatial Transformer encoder module based on an attention mechanism, a fully connected layer, three stacked temporal Transformer encoder modules based on an attention mechanism, an average pooling layer, and a prediction layer. Specifically, the data output from the embedding layer is encoded using spatial location encoding, and the encoded data retains spatial information before being input into the first graph attention module based on the graph neural network. Similarly, the data output from the embedding layer is encoded using temporal location encoding, and the encoded data retains temporal information before being input into the first temporal Transformer encoder module. The prediction layer is either a classification head or a regression head. The classification head is used to obtain the probability value of the quality of the i-th type of movement during a classification task, while the regression head is used to obtain the score for a regression task.

[0012] Preferably, the graph attention module based on the graph neural network includes a normalization layer, a first fully connected layer, a second fully connected layer, an attention mechanism-based order attention layer, a third fully connected layer, a batch normalization layer, and a ReLU layer.

[0013] Among them, the features input to the graph attention module The data distribution is normalized using a single normalization layer to obtain the normalized features. The superscript 'l' indicates the l-th graph attention module. There are two such modules, and the first graph attention module... Data encoded by spatial location;

[0014] After normalization Through the first fully connected layer Perform feature transformation mapping to obtain features At the same time, the features are paired using a predefined first-order neighbor matrix A. First-order neighbor node information aggregation is performed, and then processed through the second fully connected layer W. l Mapping yields aggregated first-order neighbor features.

[0015] Using an attention-based attention layer and The degree of association is calculated to obtain node features.

[0016] Use a third fully connected layer right Perform feature mapping to obtain features

[0017] The obtained features are processed using a batch normalization layer and a ReLU layer. Normalization and activation are performed.

[0018] Preferably, the attention-based spatial Transformer encoder module includes a residual self-attention layer and a residual layer with two fully connected layers.

[0019] The output of the second graph attention module is passed through a residual self-attention layer to calculate the similarity between the input nodes and then weighted to update the spatial features.

[0020] Spatial features are obtained from the output of the spatial Transformer encoder module through a residual layer with two fully connected layers.

[0021] Preferably, the attention-based temporal Transformer encoder module includes a residual self-attention layer and a residual layer with two fully connected layers;

[0022] The output of the previous module is passed through a residual self-attention layer to calculate the similarity between the input nodes and then weighted to update the time features.

[0023] The temporal characteristics are obtained from the output of the Transformer encoder module through a residual layer with two fully connected layers;

[0024] Among them, the module preceding the first attention-based temporal Transformer encoder module is the spatial Transformer encoder module; the second attention-based temporal Transformer encoder module is the first attention-based temporal Transformer encoder module; and the third attention-based temporal Transformer encoder module is the second attention-based temporal Transformer encoder module.

[0025] Preferably, the training process of the preset rehabilitation movement quality evaluation model includes:

[0026] a. Obtain datasets for 3D pose estimation and rehabilitation evaluation, respectively. The datasets include visible light videos and ground truth labels.

[0027] b. Extract 3D skeleton data from visible light videos in the dataset;

[0028] c. The skeletal-based rehabilitation movement quality assessment network is trained and optimized using 3D skeletal data and ground truth labels from visible light videos in the dataset to obtain the rehabilitation movement quality assessment model.

[0029] Preferably, during the training process, the training rehabilitation movement quality evaluation model is optimized with the goal of minimizing the loss function;

[0030] For regression tasks related to rehabilitation assessment, the expression for the loss function is as follows:

[0031]

[0032] Among them, X final Y is the final output score for evaluating the quality of rehabilitation movements during the regression task. regression For score labels based on score regression, L MSE This represents the mean squared error loss function;

[0033] For the classification task of rehabilitation assessment, the expression for its loss function is as follows:

[0034] L = aL EDL +bL cross

[0035] Where a and b represent weight parameters, L cross L represents the multi-class cross-entropy loss function; EDL This indicates that the loss function is incorporated into evidence-based deep learning.

[0036]

[0037] in, Let be the label value of the current video belonging to the i-th class of motion quality in the classification-based category labels. If it belongs to that class, the corresponding value is 1; otherwise, it is 0. log(·) is the logarithmic function. i This represents the probability that the predicted result belongs to the i-th class;

[0038]

[0039]

[0040]

[0041] Evid = ReLU(X) final )+1

[0042]

[0043] KL evid =(Evid-1)(1-Y class )+1

[0044] Where KL(·) is used to calculate the KL divergence, C2 is the total number of movement quality categories of the rehabilitation training results, and 1 is a C2-dimensional vector of all 1s.

[0045] Preferably, when performing a classification task, the method further includes:

[0046] The uncertainty of the rehabilitation movement quality evaluation model is calculated based on evidence-based deep learning, and the calculation formula is as follows:

[0047]

[0048] Where uncertainty represents the uncertainty of the rehabilitation movement quality evaluation model, and C2 is the total number of movement quality categories of the rehabilitation training results; This indicates the probability that the final output is the quality of the i-th type of action.

[0049] Secondly, the present invention provides a rehabilitation movement quality optimization system based on a visible light camera device, comprising:

[0050] The data acquisition module is used to acquire visible light video and extract three-dimensional skeletal data from the visible light video;

[0051] The evaluation module is used to process three-dimensional skeletal data based on a preset rehabilitation movement quality evaluation model to obtain rehabilitation movement quality evaluation results.

[0052] The pre-defined rehabilitation movement quality evaluation model includes a batch normalization layer, an embedding layer, two stacked residual graph attention modules based on graph neural networks, a spatial Transformer encoder module based on an attention mechanism, a fully connected layer, three stacked temporal Transformer encoder modules based on an attention mechanism, an average pooling layer, and a prediction layer. Specifically, the data output from the embedding layer is encoded using spatial location encoding, and the encoded data retains spatial information before being input into the first graph attention module based on the graph neural network. Similarly, the data output from the embedding layer is encoded using temporal location encoding, and the encoded data retains temporal information before being input into the first temporal Transformer encoder module. The prediction layer is either a classification head or a regression head. The classification head is used to obtain the probability value of the quality of the i-th type of movement during a classification task, while the regression head is used to obtain the score for a regression task.

[0053] Thirdly, the present invention provides a computer-readable storage medium storing a computer program for a method for optimizing the quality of rehabilitation movements based on a visible light camera device, wherein the computer program causes a computer to execute the method for optimizing the quality of rehabilitation movements based on a visible light camera device as described above.

[0054] Fourthly, the present invention provides an electronic device, comprising:

[0055] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing rehabilitation motion quality optimization methods based on visible light camera devices as described above.

[0056] (III) Beneficial Effects

[0057] This invention provides a method and system for optimizing the quality of rehabilitation movements based on a visible light camera. Compared with existing technologies, it has the following advantages:

[0058] This invention acquires visible light video and extracts three-dimensional skeletal data from it. Based on a pre-defined rehabilitation movement quality evaluation model, the three-dimensional skeletal data is processed to obtain rehabilitation movement quality evaluation results. This invention designs a rehabilitation movement quality optimization method based on a visible light camera, enabling skeletal-based rehabilitation movement quality evaluation using a low-cost visible light camera, thus reducing rehabilitation costs for patients. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a block diagram of a method for optimizing the quality of rehabilitation movements based on a visible light camera device, according to an embodiment of the present invention.

[0061] Figure 2 A schematic diagram of the structure of a skeletal-based rehabilitation movement quality assessment network;

[0062] Figure 3 This is a schematic diagram of the graph attention module based on a graph neural network.

[0063] Figure 4 A spatial thermal image of human joints;

[0064] Figure 5 This is a time heatmap of the video frames. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] This application provides a method and system for optimizing the quality of rehabilitation movements based on a visible light camera device. This solves the technical problem of high cost in existing artificial intelligence-based rehabilitation movement evaluation methods. It enables skeletal-based rehabilitation movement quality evaluation based on a low-cost visible light camera and provides visualized two-dimensional and three-dimensional skeletal movement feedback to assist rehabilitation therapists and patients in making rehabilitation movement quality evaluation decisions and optimizing rehabilitation movement quality.

[0067] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0068] Machine learning-based handcrafted feature extraction methods first extract local or global motion features to represent human movements through expert-designed feature engineering. Then, low-dimensional feature representations obtained from feature engineering are used to quantify the quality of the performed movements. However, this requires sophisticated expert knowledge and tedious manual work. Deep learning-based methods eliminate the need for complex manual work, but most existing skeletal data-based methods use depth cameras, motion capture systems, or inertial sensors with SDKs (Software Development Kits) to acquire 3D skeletal data and use it as model input. This results in limitations such as high equipment costs, restricted activity spaces, and inconvenient device wear. Furthermore, existing movement quality assessment methods almost entirely lack visual feedback, failing to clearly guide patients on which limbs / joints require focused attention. Therefore, designing an end-to-end, automatically learning, low-cost deep feature extraction method that effectively extracts rehabilitation movement features and provides visual feedback is essential for reducing rehabilitation costs, increasing the interpretability of rehabilitation movement quality assessment, improving the effectiveness of home rehabilitation training, and restoring the physical function of rehabilitation patients.

[0069] To achieve an end-to-end, automated, low-cost rehabilitation movement evaluation method that can effectively extract rehabilitation movement features and provide visual feedback, this invention presents a multi-stage visible light video skeletal rehabilitation movement evaluation model and optimization strategy based on a visible light camera device. The model sets rehabilitation physicians or patients as decision-making subjects, and combines the model's capabilities to evaluate the quality of patients' rehabilitation movements and provide visual feedback, thereby improving the quality of patients' rehabilitation.

[0070] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0071] This invention provides a method for optimizing the quality of rehabilitation movements based on a visible light camera device, such as... Figure 1 As shown, the method includes:

[0072] S1. Acquire visible light video and extract 3D skeleton data from the visible light video;

[0073] S2. Based on a pre-defined rehabilitation movement quality evaluation model, the three-dimensional skeletal data is processed to obtain the rehabilitation movement quality evaluation results. The pre-defined rehabilitation movement quality evaluation model includes a batch normalization layer, an embedding layer, two stacked graph attention modules based on graph neural networks, a spatial Transformer encoder module based on an attention mechanism, a fully connected layer, three stacked temporal Transformer encoder modules, an average pooling layer, and a prediction layer. Specifically, the data output from the embedding layer is encoded using spatial location encoding, and the encoded data retains spatial information before being input into the first graph attention module based on graph neural networks. Similarly, the data output from the embedding layer is encoded using temporal location encoding, and the encoded data retains temporal information before being input into the first temporal Transformer encoder module. The prediction layer is either a classification head or a regression head. The classification head is used to obtain the probability value of the quality of the i-th type of movement during a classification task, while the regression head is used to obtain the score for a regression task.

[0074] This invention presents a method for optimizing the quality of rehabilitation movements based on a visible light camera. This method can evaluate the quality of skeletal rehabilitation movements using a low-cost visible light camera, thereby reducing the rehabilitation costs for patients.

[0075] The following is a detailed explanation of each step:

[0076] In step S1, a visible light video is acquired, and three-dimensional skeletal data is extracted from the visible light video. The specific implementation process is as follows:

[0077] Acquire visible light video containing the patient's rehabilitation movements, and extract 3D skeletal data from the visible light video. Specifically, extracting 3D skeletal data from the visible light video includes:

[0078] First, the two-dimensional coordinates (x) of the human skeleton in each frame of the video are obtained using various existing 2D detectors. 2d ∈(T,N,2), specifically, the 2D human skeleton coordinates of each frame can be detected using networks such as Openpose or YOLO+CPN. The obtained 2D skeleton coordinates are then uplifted to 3D using pose enhancement networks such as MixSTE or MHFormer, resulting in the 3D human skeleton coordinates x. 3d ∈(T,N,3). Where T is the number of frames in the video and N is the predefined number of human joints.

[0079] In step S2, the three-dimensional skeletal data is processed based on a preset rehabilitation movement quality evaluation model to obtain the rehabilitation movement quality evaluation results. The specific implementation process is as follows:

[0080] The pre-defined rehabilitation movement quality evaluation model includes a batch normalization layer, an embedding layer, two stacked residual graph attention modules based on graph neural networks, a spatial Transformer encoder module based on an attention mechanism, a fully connected layer, three stacked temporal Transformer encoder modules, an average pooling layer, and a prediction layer. Specifically, the data output from the embedding layer is encoded using spatial location encoding, and the encoded data retains spatial information before being input into the first graph attention module based on graph neural networks. Similarly, the data output from the embedding layer is encoded using temporal location encoding, and the encoded data retains temporal information before being input into the first temporal Transformer encoder module.

[0081] The training process of the pre-set rehabilitation movement quality evaluation model is as follows:

[0082] a. Obtain datasets for 3D pose estimation and rehabilitation evaluation, respectively. The datasets include visible light videos and ground truth labels.

[0083] For 3D pose estimation and rehabilitation evaluation, obtain ground truth open-source datasets with data labels (open-source datasets for 3D human pose estimation such as Human3.6M and MPI-INF-3DHP, and open-source datasets for rehabilitation evaluation such as UI-PRMD) or create your own datasets.

[0084] Specifically, for 3D pose estimation, visible light video and corresponding 3D ground truth coordinates and ground truth labels are obtained; for rehabilitation evaluation, visible light rehabilitation action video and corresponding classification ground truth labels and score ground truth labels for rehabilitation action quality are obtained.

[0085] b. Extract 3D skeletal data from the visible light video. This process is the same as in step S1, and will not be described again here.

[0086] c. The skeletal-based rehabilitation movement quality assessment network is trained and optimized using 3D skeletal data and ground truth labels from visible light videos in the dataset, resulting in a rehabilitation movement quality assessment model. The implementation process is as follows:

[0087] Skeletal rehabilitation movement quality assessment network, such as Figure 2 As shown, the specific structure includes:

[0088] Batch Normalization Layer: A batch normalization (BN) layer is added at the beginning of the network to normalize the input 3D human skeleton coordinates x... 3d The distribution is normalized and integrated in (T,N,3) so that the network can better learn the data distribution, which facilitates the subsequent extraction and learning of spatiotemporal features from the normalized data.

[0089] Embedding layer: Normalized data is upscaled to a higher dimension by passing it through an embedding layer, resulting in upscaled data X. emb ∈(T,N,C), where the dimension C can be set according to trade-offs such as model efficiency, number of parameters, and inference speed, in order to meet the needs of subsequent model exploration in high-dimensional space.

[0090] Spatial Position Encoding: Before spatial feature extraction, a spatial position encoding (POS) is added. This POS is a learnable vector. s Preserve the spatial information of the data, that is,

[0091]

[0092] Two residual stacked graph attention modules based on graph neural networks:

[0093] We designed a graph attention module (GA module) based on a graph neural network and performed residual stacking to learn the spatial features of skeletal data. The specific structure of the graph attention module based on the graph neural network is as follows: Figure 3 As shown, it specifically includes: a normalization layer, a first fully connected layer, a second fully connected layer, an attention-based layer, a third fully connected layer, a batch normalization layer, and a ReLU layer.

[0094] Features input to the graph attention module The data distribution is normalized using a Layer Normalization (LN) layer to obtain normalized features. The superscript 'l' indicates the l-th graph attention module. There are two such modules, and the first graph attention module... for

[0095] After normalization Through the first fully connected layer Perform feature transformation mapping to obtain features At the same time, the features are paired using a predefined first-order neighbor matrix A. First-order neighbor node information aggregation is performed, and then processed through the second fully connected layer W. l Mapping yields aggregated first-order neighbor features.

[0096] Using an attention-based Hop Attention (HA) layer to... and X hThe association degree calculation is performed as follows: The HA layer consists of three independent fully connected layers, one softmax layer, and a matrix multiplication calculation. The HA layer can calculate the association degree between the features of all first-order neighbors and the node, and update each node with corresponding weights to obtain the updated node features. And use the third fully connected layer again. right Perform feature mapping to obtain features

[0097] The obtained features are normalized and activated using a batch normalization (BN) layer and a ReLU layer. This module effectively utilizes the graph structure and global features of the skeletal data, taking into account the joint synergistic effect of each joint on the overall motion and its correlation with other joints. The relevant calculation formulas for the attention-based attention layer, batch normalization layer, and ReLU layer are as follows:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105] in, and W l Both refer to fully connected layers. The order of the attention layer of the l-th graph attention module is shown in the following formula:

[0106]

[0107]

[0108]

[0109]

[0110] in, and These represent three fully connected layers, (·) T This represents the matrix transpose operation. This represents the square root operation, where d is the number of features in the feature channel dimension of feature Q.

[0111] Attention-based spatial Transformer encoder module:

[0112] To further learn the spatial features of the skeletal data and deepen the model, a Spatial Transformer Encoder (STE) module based on an attention mechanism is added after the stacked graph attention modules to model the relationships between nodes. First, the final output of the stacked graph attention modules is... The similarity between input nodes is calculated using a residual self-attention layer, and the features are updated using weighted summaries. The output is then obtained through a two-layer fully connected residual layer. The relevant calculation formulas are as follows:

[0113]

[0114]

[0115] X steout =X ste +FC ste2 (GELU(FC ste1 (LN(X ste ))))#

[0116] Where Softmax(·) represents the softmax operation. and These represent three fully connected layers, GELU(·) represents the Gaussian Error Linear Unit (GELU), and LN(·) represents the layer normalization operation.

[0117] A fully connected layer that transmits the output X of the spatial Transformer encoder module. steout The second and third dimensions of ∈(T,N,C) are merged and then fused into dimension C1 through a fully connected layer. The relevant calculation formula is as follows:

[0118] X emb1 =FC t (X steout )#

[0119] Before modeling temporal features, similar to spatial location encoding, a temporal location encoding is added. This is a learnable vector, defined as POS. t The time information of the data is retained, that is,

[0120]

[0121] Similar to the spatial Transformer encoder module, the three stacked temporal Transformer Encoder (TTE) modules model temporal information, and the relevant calculation formulas are as follows:

[0122]

[0123]

[0124]

[0125] Where the superscript m represents the m-th time Transformer encoder module, there are three in total, and the first time Transformer encoder module... for

[0126] Average pooling layer: To obtain the output, first process the output of the last time Transformer encoder module. Average pooling is used in the time dimension to obtain the feature X under global average. tte_final ∈(C1). And for classification tasks or score regression tasks, the classification head or regression head is used in the prediction layer to obtain the rehabilitation movement quality assessment result X. final ∈(C2). Here, the classification head is a fully connected layer that maps the input dimension C1 to the output classification dimension, i.e., C2 is the number of categories to be classified; the regression head is also a fully connected layer that maps the input dimension C1 to one dimension of the output, i.e., C2 is 1.

[0127] During training, the quality of model training is evaluated using a loss function. For regression tasks, this embodiment of the invention uses the mean squared error loss function L. MSE To make it closer to the true score.

[0128]

[0129] Among them, X final For the final output of the rehabilitation movement quality assessment, Y regression The score labels are based on score regression.

[0130] For classification tasks, this study employs the multi-class cross-entropy loss function L... cross In addition, it also incorporates the evidence deep learning loss function L based on evidence deep learning. EDL Finally, the loss function is a weighted sum of the two functions.

[0131]

[0132] in, Let be the label value of the current video belonging to the i-th class in the classification-based category label. If it belongs to that class, the corresponding value is 1; otherwise, it is 0. log(·) is the logarithmic function. i Let X represent the probability that the predicted result belongs to the i-th class. final The result after applying softmax.

[0133] Evid = ReLU(X) final )+1

[0134]

[0135]

[0136] KL evid =(Evid-1)(1-Y class )+1

[0137]

[0138]

[0139] Where KL(·) is used to calculate the KL divergence, C2 is the total number of action quality categories, and 1 is a C2-dimensional vector of all 1s.

[0140] In practical implementation, considering model interpretability and providing patients with visual guidance and feedback, this embodiment of the invention also outputs the uncertainty of the model based on evidence-based deep learning when performing classification tasks. Simultaneously, it visualizes the nodes and frames that should be emphasized in the current action based on the attention mechanism designed in the model, providing patients with visual feedback and guidance, thereby optimizing the quality of their rehabilitation movements. The uncertainty of the model is calculated as follows:

[0141]

[0142] C2 represents the total number of categories of movement quality resulting from rehabilitation training. This indicates the probability that the final output is the quality of the i-th type of action.

[0143] The visualization feedback and guidance calculates the softmax function results in the attention layer and the temporal Transformer encoder module, takes the average of these results, and obtains the final importance vectors (N) and (T), which are then visualized as heatmaps. Figure 4 , Figure 5 As shown, where, Figure 4The spatial heatmap shown represents the average attention level of each joint across all frames of the rehabilitation exercise video. The horizontal axis represents the joint index, and each colored block represents the corresponding attention level, with its color indicating the intensity of attention. The intensity value is reflected in the color bar on the right. The spatial heatmap allows observation of which joints require more attention during the current rehabilitation exercise. Figure 5 The time heatmap shown represents the average attention received per frame in the rehabilitation exercise video. The horizontal axis represents the frame index, and each colored block represents the corresponding attention level, with its color indicating the intensity of attention. The intensity value is reflected in the color bar on the right. The time heatmap allows observation of which frames in the current rehabilitation training exercise require more attention, and the execution status of the corresponding frame can be viewed through the corresponding frame index to obtain feedback.

[0144] Furthermore, the index of the maximum value of the (T) importance vector obtained from the softmax function calculated in the temporal Transformer encoder module is obtained, and the corresponding 2D human skeleton coordinates are then used. and 3D human skeleton coordinates It is visualized using plotting functions.

[0145] This invention also provides a rehabilitation movement quality optimization system based on a visible light camera device, the system comprising:

[0146] The visible light video processing module is used to acquire visible light video and extract three-dimensional skeletal data from the visible light video.

[0147] The evaluation module processes 3D skeletal data based on a pre-defined rehabilitation movement quality evaluation model to obtain rehabilitation movement quality evaluation results. This pre-defined model includes a batch normalization layer, an embedding layer, two stacked graph attention modules based on graph neural networks, a spatial Transformer encoder module based on an attention mechanism, a fully connected layer, three stacked temporal Transformer encoder modules, an average pooling layer, and a prediction layer. Specifically, spatial location encoding is used to encode the data output from the embedding layer, retaining spatial information before inputting it into the first graph attention module based on the graph neural network; temporal location encoding is used to encode the data output from the embedding layer, retaining temporal information before inputting it into the first temporal Transformer encoder module. The prediction layer is either a classification head or a regression head. The classification head is used to obtain the probability value of the quality of the i-th type of movement during a classification task, while the regression head is used to obtain the score for a regression task.

[0148] It is understood that the rehabilitation movement quality optimization system based on visible light camera equipment provided in this embodiment of the invention corresponds to the rehabilitation movement quality optimization method based on visible light camera equipment described above. The explanation, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the rehabilitation movement quality optimization method based on visible light camera equipment, and will not be repeated here.

[0149] This invention also provides a computer-readable storage medium storing a computer program for optimizing the quality of rehabilitation movements based on a visible light camera device, wherein the computer program causes a computer to execute the above-described method for optimizing the quality of rehabilitation movements based on a visible light camera device.

[0150] This invention also provides an electronic device, comprising:

[0151] One or more processors;

[0152] Memory; and

[0153] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing rehabilitation motion quality optimization methods based on visible light camera devices as described above.

[0154] In summary, compared with existing technologies, it has the following beneficial effects:

[0155] 1. This invention presents a method for optimizing the quality of rehabilitation movements based on a visible light camera. This method can evaluate the quality of skeletal rehabilitation movements using a low-cost visible light camera, thereby reducing the rehabilitation costs for patients.

[0156] 2. This invention can visualize the nodes and frames that should be emphasized in the current action based on multiple attention mechanisms designed in the model, so as to provide patients with visual feedback and guidance, thereby optimizing the quality of their rehabilitation actions.

[0157] 3. The embodiments of the present invention combine the evidence deep learning loss function to additionally obtain the uncertainty of the model, thereby reducing the risk of prediction errors and giving the model interpretability to a certain extent.

[0158] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for rehabilitation motion quality optimization based on a visible light camera device, characterized in that, include: Acquire visible light video and extract 3D skeletal data from the visible light video; The three-dimensional skeletal data is processed based on the pre-set rehabilitation movement quality evaluation model to obtain the rehabilitation movement quality evaluation results. The pre-defined rehabilitation movement quality evaluation model includes a batch normalization layer, an embedding layer, two stacked residual graph attention modules based on graph neural networks, a spatial Transformer encoder module based on an attention mechanism, a fully connected layer, three stacked temporal Transformer encoder modules based on an attention mechanism, an average pooling layer, and a prediction layer. Specifically, the data output from the embedding layer is encoded using spatial location encoding, retaining spatial information before being input into the first graph attention module based on the graph neural network; the data output from the embedding layer is encoded using temporal location encoding, retaining temporal information before being input into the first temporal Transformer encoder module; the prediction layer is either a classification head or a regression head, where the classification head is used to obtain the first regression head for classification tasks. i The probability value of the quality of the class of actions is used by the regression head to obtain the score of the regression task; The graph attention module based on the graph neural network includes a normalization layer, a first fully connected layer, a second fully connected layer, an attention mechanism-based order attention layer, a third fully connected layer, a batch normalization layer, and a ReLU layer. For the classification task of rehabilitation assessment, the expression for its loss function is as follows: in, a , b Represents the weight parameters. This represents the cross-entropy loss function for multi-class classification. This indicates that the loss function is incorporated into evidence-based deep learning. in, To determine the category to which the current video belongs in the category-based labeling. The label value for the quality of the action class; if it belongs to that class, the corresponding value is 1, otherwise it is 0. It is a logarithmic function; This indicates that the predicted result belongs to the first... The probability of a class; in, To calculate the KL divergence, The total number of movement quality categories representing the results of rehabilitation training. for A vector of all 1s in dimension. The final output score is used to evaluate the quality of rehabilitation movements during the regression task.

2. The method for optimizing the quality of rehabilitation movements based on a visible light camera as described in claim 1, characterized in that, The attention-based spatial Transformer encoder module includes a residual self-attention layer and a residual layer with two fully connected layers. The output of the second graph attention module is passed through a residual self-attention layer to calculate the similarity between the input nodes and then weighted to update the spatial features. Spatial features are obtained from the output of the spatial Transformer encoder module through a residual layer with two fully connected layers.

3. The method for optimizing the quality of rehabilitation movements based on a visible light camera as described in claim 1, characterized in that, The attention-based temporal Transformer encoder module includes a residual self-attention layer and a residual layer with two fully connected layers. The output of the previous module is passed through a residual self-attention layer to calculate the similarity between the input nodes and then weighted to update the time features. The temporal characteristics are obtained from the output of the Transformer encoder module through a residual layer with two fully connected layers; Among them, the module preceding the first attention-based temporal Transformer encoder module is the spatial Transformer encoder module; the second attention-based temporal Transformer encoder module is the first attention-based temporal Transformer encoder module; and the third attention-based temporal Transformer encoder module is the second attention-based temporal Transformer encoder module.

4. The method for optimizing the quality of rehabilitation movements based on a visible light camera device as described in any one of claims 1 to 3, characterized in that, The training process of the preset rehabilitation movement quality evaluation model includes: a. Obtain datasets for 3D pose estimation and rehabilitation evaluation, respectively. The datasets include visible light videos and ground truth labels. b. Extract 3D skeleton data from visible light videos in the dataset; c. The skeletal-based rehabilitation movement quality assessment network is trained and optimized using 3D skeletal data and ground truth labels from visible light videos in the dataset to obtain the rehabilitation movement quality assessment model.

5. The method for optimizing the quality of rehabilitation movements based on a visible light camera device as described in any one of claims 1 to 3, characterized in that, When performing classification tasks, the method also includes: The uncertainty of the rehabilitation movement quality evaluation model is calculated based on evidence-based deep learning, and the calculation formula is as follows: in, This indicates the uncertainty of the rehabilitation movement quality evaluation model. The total number of categories for movement quality resulting from rehabilitation training; This indicates that the final output is the [number]. i Probability of action quality.

6. A rehabilitation movement quality optimization system based on a visible light camera device, characterized in that, include: The data acquisition module is used to acquire visible light video and extract three-dimensional skeletal data from the visible light video; The evaluation module is used to process three-dimensional skeletal data based on a preset rehabilitation movement quality evaluation model to obtain rehabilitation movement quality evaluation results. The pre-defined rehabilitation movement quality evaluation model includes a batch normalization layer, an embedding layer, two stacked residual graph attention modules based on graph neural networks, a spatial Transformer encoder module based on an attention mechanism, a fully connected layer, three stacked temporal Transformer encoder modules based on an attention mechanism, an average pooling layer, and a prediction layer. Specifically, the data output from the embedding layer is encoded using spatial location encoding, retaining spatial information before being input into the first graph attention module based on the graph neural network; the data output from the embedding layer is encoded using temporal location encoding, retaining temporal information before being input into the first temporal Transformer encoder module; the prediction layer is either a classification head or a regression head, where the classification head is used to obtain the first regression head for classification tasks. i The probability value of the quality of the class of actions is used by the regression head to obtain the score of the regression task; The graph attention module based on the graph neural network includes a normalization layer, a first fully connected layer, a second fully connected layer, an attention mechanism-based order attention layer, a third fully connected layer, a batch normalization layer, and a ReLU layer. For the classification task of rehabilitation assessment, the expression for its loss function is as follows: in, a , b Represents the weight parameters. This represents the cross-entropy loss function for multi-class classification. This indicates that the loss function is incorporated into evidence-based deep learning. in, To determine the category to which the current video belongs in the category-based labeling. The label value for the quality of the action class; if it belongs to that class, the corresponding value is 1, otherwise it is 0. It is a logarithmic function; This indicates that the predicted result belongs to the first... The probability of a class; in, To calculate the KL divergence, The total number of movement quality categories representing the results of rehabilitation training. for A vector of all 1s in dimension. The final output score is used to evaluate the quality of rehabilitation movements during the regression task.

7. A computer-readable storage medium, characterized in that, It stores a computer program for a method of optimizing the quality of rehabilitation movements based on a visible light camera device, wherein the computer program causes a computer to execute the method of optimizing the quality of rehabilitation movements based on a visible light camera device as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing rehabilitation motion quality optimization methods based on visible light camera devices as described in any one of claims 1 to 5.

Images