Exercise ability assessment method

Abstract

The invention provides an athletic ability assessment method. The athletic ability assessment method comprises the following steps: acquiring exercise data of a to-be-assessed person; extracting original motion posture data of the to-be-assessed person from the motion data of the to-be-assessed person; according to a preset data alignment requirement, preprocessing the original motion posture data of the to-be-assessed person to obtain preprocessed motion posture data of the to-be-assessed person; and inputting the preprocessed motion posture data of the to-be-assessed person into the trained motion ability assessment model obtained according to the training method provided by the invention so as to assess the motion ability of the to-be-assessed person.

Description

Technical Field

[0001] The present invention relates to the field of ubiquitous computing, in particular to the field of artificial intelligence, and more particularly to a method for evaluating athletic ability. Background Art

[0002] Stroke, also known as stroke, causes brain cells to be damaged or die within minutes due to lack of oxygen, thus affecting other parts of the body. According to research, the number of stroke patients worldwide exceeds 100 million, and about 40%-50% of stroke patients will experience long-term upper limb dysfunction, which will seriously affect the patient's daily life and social participation ability. Rehabilitation is crucial for the functional recovery of patients after stroke. Studies have found that if active rehabilitation treatment is given to stroke patients, 90% of surviving patients can walk and take care of themselves again, and 30% of surviving patients can do some lighter work. On the contrary, without rehabilitation treatment, the recovery percentages of the above two aspects are only 6% and 5% respectively. Motor ability assessment is the basis of stroke rehabilitation. However, traditional assessment methods rely on scales to assess patients' motor ability, such as the Burkholderia staging scale, the Fulger Meyer scale, the Berg scale, etc., which have the disadvantages of low efficiency of the assessment process, strong subjectivity of the assessment, and reliance on expert experience.

[0003] With the continuous development of artificial intelligence and deep learning models, motion assessment methods based on machine learning are constantly being explored in order to achieve convenient, efficient and objective motor ability assessment. There are many attempts to use machine learning to evaluate the motion of stroke patients. Some methods extract manual features, including sequence length, sequence variance, and average speed and acceleration of each measurement value, and use support vector machines (SVM) and random forest algorithms (RF) to score the patient's movement. Some methods extract features such as joint angles and joint distances, and use LSTM networks and GCN networks to process features and bone point data to evaluate the quality of movement of patients during home rehabilitation.

[0004] However, in real medical scenarios, the motor function assessment of stroke patients faces the following challenges:

[0005] 1) Due to the privacy and collection costs of medical data, the amount of medical data is very limited;

[0006] 2) In addition, unlike the 3D evaluation devices such as Kinect used in scientific research, RGB cameras are mostly used in clinical and daily applications, and then the coordinate axes (X, Y) of the skeleton points are obtained through posture recognition algorithms, so only 2D data can be obtained, which further compresses the source of data;

[0007] 3) At the same time, in the process of stroke motion assessment based on skeleton points, some stroke patients have severe motor impairment and it is difficult for them to complete the corresponding assessment movements in a standard manner. This phenomenon leads to occlusion of the patient's limbs, further causing inaccurate posture recognition of the patient and inducing skeleton point noise.

[0008] Existing methods have difficulty learning generalizable features in such a situation where the amount of data is limited and there is noise, resulting in low accuracy in motor function assessment. Moreover, blindly increasing the complexity of the model is not a good choice in such a situation where the amount of data is limited and there is severe noise in the data, as it can easily lead to overfitting of the model.

[0009] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. The relevant information is submitted and disclosed together with the present invention solution. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0010] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a training method for a sports ability evaluation model.

[0011] The objective of the present invention is achieved through the following technical solutions:

[0012] According to a first aspect of the present invention, a training method for a motion ability assessment model is provided, comprising: obtaining initial training data, which comprises a plurality of samples, each sample comprising motion data of a sample person and a label thereof, wherein the motion data is a video of a recorded object responding to a preset action requirement, and the label is a true value of the object's motion ability category; extracting the original motion posture data of the sample person from the motion data of each sample; preprocessing the original motion posture data of the sample person according to a preset data alignment requirement to obtain preprocessed motion posture data; replacing the motion data of the sample person with the preprocessed motion posture data to obtain a classification training set; performing data enhancement on the preprocessed motion posture data to obtain enhanced motion posture data; constructing a comparative learning training set using the preprocessed motion posture data and the enhanced motion posture data, which comprises a positive sample pair and a negative sample pair, wherein the motion posture data in the positive sample pair originates from the same sample person, and the motion posture data in the negative sample pair originates from different sample persons; and jointly training a preset motion ability assessment model using the classification training set and the comparative learning training set to obtain a trained motion ability assessment model. This solution can at least achieve the following beneficial technical effects: by constructing positive and negative sample pairs through data enhancement and calculating contrast loss, the model can learn time-invariant features, thereby improving its generalization ability.

[0013] Optionally, data enhancement is to perform overall random translation of the preprocessed motion posture data along the frame direction, and, when jointly training a preset motion ability evaluation model, use a contrast loss sub-function to guide the model to close the distance between the features of the motion posture data in the positive sample pair and push away the distance between the features of the motion posture data in the negative sample pair. This solution can at least achieve the following beneficial technical effects: This solution performs translation enhancement on the data in the time dimension, and uses contrast learning to help the model learn differential features in a limited number of samples, so that it can learn time-invariant features, thereby improving its generalization ability.

[0014] Optionally, the preset sports ability assessment model includes: a spatiotemporal feature extractor for extracting the spatiotemporal features of the input motion posture data from the spatial dimension and the temporal dimension; a bone point spatial enhancement module for mapping the spatiotemporal features of the motion posture data according to a learnable matrix, and then processing through a convolution layer, a pooling layer and a first fully connected layer to obtain spatial enhancement features; a prediction layer for determining the sports ability category of the sample person according to the spatial enhancement features. This scheme can at least achieve the following beneficial technical effects: the scheme can mine the spatiotemporal correlation between the bone points, thereby reducing the impact of noise in the bone sequence on the sports ability stage, and adopt a method based on the spatial enhancement of the bone points to construct a learnable matrix, which can improve the weight of important bone points and reduce the weight of the bone point features of irrelevant and interfered nodes according to the importance of the sports ability assessment on the data of each bone point, and learn the feature dependency of the global bone points at the same time, thereby improving the performance of the model in evaluating the sports ability of the sample person.

[0015] Optionally, the skeleton point spatial enhancement module includes: a spatial weight enhancement layer, which is used to calculate the product of the spatiotemporal features of the motion posture data and a learnable matrix to obtain an enhanced feature matrix; a modeling layer, which is used to convolve the enhanced feature matrix according to the one-dimensional convolution layer to obtain a mapped feature matrix; a pooling layer, which is used to pool the mapped feature matrix to obtain pooled features; and a first fully connected layer, which is used to process the pooled features to obtain spatially enhanced features. This solution can at least achieve the following beneficial technical effects: this solution can further increase the weight of data related to athletic ability assessment through the spatial weight enhancement layer, use the one-dimensional convolution layer of the modeling layer for one-dimensional mapping, use the pooling layer to reduce the feature dimension, and use the fully connected layer to process the pooled features to obtain spatially enhanced features, thereby improving the performance of the model.

[0016] Optionally, the spatiotemporal feature extractor uses a spatiotemporal graph convolutional neural network with a depth of 3 or 4 layers. This solution can at least achieve the following beneficial technical effects: when the spatiotemporal graph convolutional neural network uses 3 or 4 layers, the performance of the sports ability evaluation model is better.

[0017] Optionally, when training the model, the parameters of the athletic ability assessment model are updated according to the total loss function to minimize the total loss function value, wherein the total loss function includes a contrast loss sub-function and a classification loss sub-function.

[0018] Optionally, the patient's motor ability includes upper limb motor ability, lower limb motor ability, sitting balance ability and standing balance ability, and each motor ability is divided into multiple categories to distinguish the strength of the motor ability.

[0019] According to a second aspect of the present invention, there is provided a method for evaluating athletic ability, comprising: acquiring athletic data of a person to be evaluated; extracting original athletic posture data of the person to be evaluated from the athletic data of the person to be evaluated; preprocessing the original athletic posture data of the person to be evaluated according to preset data alignment requirements to obtain preprocessed athletic posture data of the person to be evaluated; and inputting the preprocessed athletic posture data of the person to be evaluated into a trained athletic ability evaluation model obtained by the method described in the first aspect to evaluate the athletic ability of the person to be evaluated.

[0020] According to a third aspect of the present invention, there is provided an electronic device comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the first and / or second method by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0022] Figure 1 is a schematic diagram of the structure of a sports ability assessment model according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a preset action requirement according to an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a flow chart of a training method for a sports ability assessment model according to an embodiment of the present invention;

[0025] Figure 4 The figure is a schematic flowchart of a method for evaluating motor ability of stroke patients according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] As mentioned in the background technology section, existing methods are difficult to learn generalizable features in such a situation where the amount of data is limited and there is noise, resulting in low accuracy in motor function assessment. Moreover, in such a situation where the amount of data is limited and there is severe noise in the data, blindly increasing the complexity of the model is not a good choice, and it is easy to cause overfitting of the model.

[0028] When the inventors were conducting research on stroke motor ability assessment based on computer vision, they found that in real medical scenarios, due to the impaired motor ability of stroke patients, it was difficult to complete the corresponding assessment actions in a standard manner. This phenomenon leads to occlusion of the patient's limbs, resulting in inaccurate patient posture recognition and bone point noise. At the same time, medical data is very limited, which makes the model prone to overfitting on the training data and the model does not have generalization. Therefore, it is considered to enhance the model's ability to extract features, thereby improving the generalization ability of the model, and then improving the accuracy of the model's assessment of the motor ability of stroke patients.

[0029] In order to solve the above challenges and improve the generalization of the skeleton-based stroke motor ability assessment model in an open environment, a spatiotemporal scale enhancement technology based on contrastive learning, namely STSAug, is proposed. This model cleverly sets the contrastive learning training set and uses contrastive learning to enable the model to learn time-independent features, thereby improving the generalization ability of the model.

[0030] The following introduces the scheme of the present invention from five aspects: model structure, training samples, training process, application scenarios and experimental data.

[0031] 1. Model Structure

[0032] The sports ability assessment model is constructed using a neural network and is used to assess the category of a subject's sports ability.

[0033] According to an embodiment of the present invention, the sports ability evaluation model can adopt the model structure in the prior art, such as the model structure in ST-GCN, 2s-AGCN or CTR-GCN. The relevant technical documents are as follows:

[0034] ST-GCN: Sijie Yan, Yuanjun Xiong, and Dahua Lin. “Spatial temporal graph convolutional networks for skeleton- based action recognition”. In: Proceedings of the AAAI conference on artificial intelligence. Vol. 32. 1.2018.

[0035] 2s-agcn: Lei Shi et al. “Two-stream adaptive graph convolutional networks for skeleton-based action recognition”. In: Proceedings of the IEEE / CVF conference on computer.

[0036] CTR-GCN: Yuxin Chen et al. “Channel-wise topology refinement graphconvolution for skeleton-based action recognition”. In: Proceedings of the IEEE / CVF international conference on computer vision. 2021, pp. 13359–13368.

[0037] When evaluating the motor ability of stroke patients, giving higher weights to the features of points that are not related to the staging (i.e., skeletal points that are not related to the staging task) will weaken the generalization ability of the model and reduce the accuracy of the evaluation stage. Therefore, in order to improve the effect of motor ability, it is possible to consider improving the model structure.

[0038] According to one embodiment of the present invention, the sports ability assessment model includes: a spatiotemporal feature extractor, which is used to extract the spatiotemporal features of the input motion posture data from the spatial dimension and the temporal dimension; a skeletal point spatial enhancement module, which is used to map the spatiotemporal features of the motion posture data according to a learnable matrix, and then obtain the spatial enhancement features after processing through a convolution layer, a pooling layer and a first fully connected layer; a prediction layer, which is used to determine the sports ability category of the sample person according to the spatial enhancement features. Among them:

[0039] Preferably, a spatiotemporal graph convolutional neural network is used as a spatiotemporal feature extractor. The spatiotemporal graph convolutional network represents the human skeleton as a graph structure, wherein the bone points are represented as nodes and the connection relationship between the bones is represented as edges. The spatiotemporal graph convolutional network comprises a plurality of stacked spatiotemporal convolutional layers, each of which comprises a spatial convolutional module and a temporal convolutional module, wherein the spatial convolutional module is used to extract spatial features from the input data, and the temporal convolutional module is used to extract spatiotemporal features from the input spatial features. The input of the first spatial convolutional module is the preprocessed motion posture data, and the output of the last temporal convolutional module is the final spatiotemporal features. The input of each intermediate module is the output of the previous module. In the process of motor ability assessment, the relative spatial position of the bone points is crucial. The spatial convolutional module can deeply explore the positional relationship between adjacent bone points, thereby extracting the spatial features of the bone points. The temporal convolutional module is responsible for extracting the features of the bone points in the time series, including the position changes of the bone points in the time dimension. Especially when the skeleton points generate noise due to occlusion, the skeleton points may jump or cannot be accurately identified. The temporal convolution module can smooth these abnormal frames, thereby reducing the impact of noise on motion ability evaluation.

[0040] Indicatively, see Figure 1 , a set of GCN and TCN represents a spatiotemporal convolutional layer, where GCN represents the spatial convolutional module and TCN represents the temporal convolutional module. In addition, residual connections may also be added between the input and output of some spatiotemporal convolutional layers, that is, Figure 1 The arrows crossing GCN and TCN are generally set at intervals. Through these two modules, the temporal and spatial features of the skeleton point posture data of stroke patients are extracted respectively, and the temporal and spatial correlation between the skeleton points is mined, thereby reducing the impact of noise in the skeleton sequence on the motor ability stage.

[0041] Preferably, the spatiotemporal feature extractor uses a spatiotemporal graph convolutional neural network with a depth of 3 or 4 layers, that is, the number of spatiotemporal convolutional layers is 3 or 4. In addition, if the spatiotemporal graph convolutional neural network contains 3 layers of spatiotemporal convolutional layers, a residual connection can be set in the second layer; if the spatiotemporal graph convolutional neural network contains 4 layers of spatiotemporal convolutional layers, residual connections may not be set in layers 1 and 3, and residual connections are set in layers 2 and 4.

[0042] According to an example of the present invention, a spatiotemporal feature extractor (STFE) is provided, which represents human skeleton points as a graph ,in, It is a picture The adjacency matrix of Represents a collection of body joints, is the set of connections between the skeleton points. Indicates the number of bone points. Contains multiple matrices , special, , I represents the diagonal unit matrix, when k is not 0, is the normalized adjacency matrix. In order to extract spatiotemporal features, the skeleton point data Input to the spatiotemporal feature extractor, where is the number of frames, is the number of bone points, Represents the coordinate dimension. Schematically, the extraction of spatial features is shown in the following formula:

[0043]

[0044] in, It is a weight matrix formed by stacking the weight vectors of multiple output channels;

[0045] Next, the spatial features Input into the time convolution module to extract spatiotemporal features. Schematically, the extraction of spatiotemporal features is shown in the following formula:

[0046]

[0047] in, represents the spatiotemporal features extracted from the skeleton points, represents the activation function, represents the convolution operation, Represents the convolution kernel.

[0048] In the process of motor ability assessment, different skeletal points have different degrees of influence on the final staging results. For example, in the assessment of upper limb ability in stroke, the feature importance of the hand skeletal points is significantly higher than the spatiotemporal features of the lower limbs. At the same time, when the amount of data is limited, the disturbance of irrelevant features can easily interfere with the feature extraction process of the model, thereby affecting the generalization of the model. In addition, the relative position between skeletal points, whether adjacent nodes (such as elbows and shoulders) or non-adjacent nodes (such as palms and heads), is crucial to motor ability staging. Therefore, an embodiment of the present invention proposes a skeletal point spatial enhancement module, which aims to increase the weights of important skeletal points, reduce the weights of irrelevant or interfered node features, and learn the feature dependencies between global skeletal points.

[0049] Preferably, the skeleton point spatial enhancement module includes: a spatial weight enhancement layer, which is used to calculate the product of the spatiotemporal features of the motion posture data and the learnable matrix to obtain an enhanced feature matrix; a modeling layer, which is used to convolve the enhanced feature matrix according to the one-dimensional convolution layer to obtain a mapped feature matrix; a pooling layer, which is used to pool the mapped feature matrix to obtain pooled features; and a first fully connected layer, which is used to process the pooled features to obtain spatially enhanced features. When evaluating and staging the motor ability of stroke patients, the prior art assigns higher weights to the features of points that are not related to the staging (i.e., skeleton points that are not related to the staging task), which weakens the generalization ability of the model and reduces the accuracy of the evaluation and staging. This solution can at least achieve the following beneficial technical effects: a learnable matrix is ​​constructed based on a method of skeleton point spatial enhancement to learn the importance of skeleton point features for different motor ability assessments (such as upper limbs, lower limbs and motor balance assessments), increase the weights of important skeleton points, reduce the weights of skeleton point features of irrelevant and interfered nodes, and learn the feature dependencies of global skeleton points at the same time, thereby improving the generalization ability of the model and the accuracy of motor ability assessment.

[0050] According to an example of the present invention, a skeleton point space enhancement (SSA) module is constructed to increase the weights of important skeleton points, reduce the weights of irrelevant or disturbed nodes, and learn the global dependencies between skeleton features. For example, a learnable matrix , which is multiplied with the final spatiotemporal features to adaptively learn the importance of each skeleton point. Global Modeling For further feature extraction, as shown in the following formula:

[0051]

[0052] After the features are pooled, they are processed in a fully connected layer to obtain spatially enhanced features, ensuring that the impact of irrelevant features on the generalization ability of the model is minimized.

[0053] The prediction layer is used to determine the athletic ability category of the sample person based on the spatial enhancement features.

[0054] According to one embodiment of the present invention, the prediction layer includes a second fully connected layer and a Softmax layer. The second fully connected layer is used to process the features output by the first fully connected layer to output the logical value (Logit) of each category under each athletic ability. The Softmax layer is used to determine the confidence of each category under each athletic ability based on the logical value (Logit) of each category under each athletic ability. Under each athletic ability, the category with the highest confidence is selected as the evaluation result of the athletic ability.

[0055] 2. Training Samples

[0056] Before starting training, you need to collect data sets to provide samples required for training so that you can train the model later.

[0057] According to one embodiment of the present invention, the data set may adopt an existing data set for evaluating athletic ability. The data set includes multiple samples, each sample includes the athletic data of the sample person and its label, wherein the athletic data is a video of the object responding to a preset action requirement, and the label is the true value of the object's athletic ability category. For example: based on the athletic ability, multiple sample persons may be selected evenly, and then the athletic data of each sample person may be collected. Instructions are issued to the sample person (object) to act according to the preset action requirement, and a video of the sample person acting is collected. Subsequently, labels may be added, i.e., the true value of the athletic ability category of the sample person corresponding to the athletic data (i.e., the actual athletic ability category) may be marked. Optionally, see Figure 2 , the preset action requirements include: the first action requirement: let the subject move the hand on the affected side from the knee of the healthy side to behind the ear, the second action requirement: let the subject do shoulder flexion (standing or sitting, let the arm hang naturally first, then slowly raise the arm forward, and finally until the arm is facing up, and the arm rotates about 180° during the whole process), the third action requirement: let the subject do sitting knee flexion (start with sitting position, legs perpendicular to the ground, then move the leg on the side affected by stroke backward), and the fourth action requirement: let the subject do standing knee flexion (start with vertical standing position, gradually bend the knee until the leg forms a 90-degree angle with the thigh). It should be understood that the implementer can actually adjust the action requirements according to the actual situation to achieve similar effects. For example, in the fourth action requirement, let the patient flex the knee on one leg first, and then slightly bend the other leg.

[0058] The athletic ability may be an overall athletic ability, or may be divided into multiple types (multiple aspects) so as to assess the strength of the various athletic abilities of the subject.

[0059] According to one embodiment of the present invention, the patient's motor ability includes the motor ability of the upper limbs, the motor ability of the lower limbs, the sitting balance ability and the standing balance ability, and each motor ability is divided into multiple categories to distinguish the strength of the motor ability. For example: the motor ability of the upper limbs, the motor ability of the lower limbs, the sitting balance ability and the standing balance ability are divided into 4 categories, respectively, expressed as: 0-very weak, almost no motor ability; 1-weak, with weak motor ability, obvious muscle compensation phenomenon; 2-has good motor ability, can perform the assessment action irregularly; 3-has strong motor ability. Of course, the implementer can also set other numbers of categories, such as 3 categories, namely: 0-poor, 1-medium, 2-good. Or, set it according to the categories set in some scales. Among them, when the preset action requirements are designed according to the previous embodiment, the evaluation of the upper limbs' motor ability can be performed by using a video of the subject performing actions according to the first action requirement and the second action requirement, the evaluation of the lower limbs' motor ability can be performed by using a video of the subject performing actions according to the third action requirement and the fourth action requirement, the evaluation of the sitting balance ability can be performed by using a video of the subject performing actions according to the third action requirement, and the evaluation of the sitting balance ability can be performed by using a video of the subject performing actions according to the fourth action requirement. Of course, the implementer can also use other correspondences between action requirements and various balance abilities.

[0060] 3. Training Process

[0061] During training, a subset can be extracted from the data set as an initial training set, which contains the initial training data.

[0062] According to one embodiment of the present invention, see Figure 3 , provides a training method for a sports ability evaluation model, including steps: A1-A7. Each step is described below.

[0063] Step A1: Obtain initial training data, which includes multiple samples.

[0064] According to an embodiment of the present invention, the data set may be divided into a training set and a test set (or a training set, a validation set, and a test set) according to a preset ratio, and the samples in the training set are used as initial training data.

[0065] Step A2: Extract the original motion posture data of the sample person from the motion data of each sample.

[0066] According to an embodiment of the present invention, the motion data is a video, and a key point (set as a skeleton point in the embodiment of the present invention) detection algorithm can be used to extract skeleton key points from the video. For example: Mediapipe algorithm, skeleton point detection algorithm in OpenPose or skeleton point detection algorithm in PoseC3D. Of course, a key point detection algorithm customized by the implementer can also be used.

[0067] According to an embodiment of the present invention, step A3: preprocessing the original motion posture data of the sample person according to the preset data alignment requirement to obtain the preprocessed motion posture data;

[0068] According to one embodiment of the present invention, in order to better evaluate, in the data preprocessing stage, data frame length alignment operation and skeleton point coordinate alignment operation are performed. Of course, it can also be set according to the needs of the implementer, for example, only data frame length alignment is performed without skeleton point coordinate alignment; or other alignment operations are added additionally.

[0069] Among them, data frame length alignment is to unify all the original motion posture data in the time dimension. For example, the last frame of data can be used for length alignment. In the stroke patient motion data set, the data length is aligned to a preset number of frames (such as 250 frames, or other values, such as 240 frames, 300 frames, etc.). For example, the last frame of data is copied and added to the end of the data until the data length is aligned. Of course, other methods can also be used to fill in the gaps, such as intercepting as many frames from the last frame of data as the preset number of frames is missing and adding them to the end of the data.

[0070] Bone point coordinate alignment is to align the origin of the coordinate system of all frames with the origin of the coordinate system preset in the first frame. For example, in order to unify the distribution of data and reduce the impact of the position of the human body in the image, the 0th bone point of the first frame of each data is used as the origin of the coordinate system, and each data frame is aligned to this coordinate system. That is, the value obtained by subtracting the original coordinate of the 0th bone point from the original coordinate of each bone point is used as the coordinate of the aligned bone point.

[0071] Step A4: replacing the motion data of the sample person with the preprocessed motion posture data to obtain a classification training set;

[0072] According to one embodiment of the present invention, the video in the initial training data is used to replace the corresponding pre-processed motion posture data, thereby obtaining a classification training set.

[0073] Step A5: performing data enhancement on the preprocessed motion posture data to obtain enhanced motion posture data;

[0074] Since it is difficult to collect samples in the field of stroke, the amount of sample data is limited. When the number of samples is limited, contrastive learning can help the model learn differentiated features from limited samples. In order to further improve the effect of contrastive learning, data enhancement technology can be used.

[0075] According to one embodiment of the present invention, in the evaluation of athletic ability, the key skeletal posture plays a decisive role in the judgment of athletic ability, i.e., the maximum athletic posture that the patient can complete. The inventor believes that the change of this key posture in the time dimension will not change the patient's evaluation result. Therefore, the present embodiment performs enhanced processing on the time dimension on the data. Enhanced athletic posture data is generated by performing random time offset on the frames in the pre-processed athletic posture data. The athletic posture data is actually a skeleton point frame sequence, which is equivalent to generating a random number, and the frames in the skeleton point frame sequence are translated as a whole according to the random number to obtain the enhanced skeleton point data; for example, if the random number is 5 and the setting is right shift, the frames in the skeleton point frame sequence are shifted right by 5 frames as a whole, which is equivalent to the last 5 frames becoming the first 5 frames.

[0076] Preferably, in order to avoid the impact of excessive random time offset on the accuracy of the evaluation, a random number range can be set (for example, [-15, 15] or [-20, 20], where a negative random number indicates a left shift and a positive random number indicates a right shift), and a random number is generated from the random number range to perform random time offset.

[0077] Step A6: Use the preprocessed motion posture data and the enhanced motion posture data to construct a comparative learning training set, which includes positive sample pairs and negative sample pairs, wherein the motion posture data in the positive sample pairs come from the same sample person, and the motion posture data in the negative sample pairs come from different sample persons.

[0078] According to one embodiment of the present invention, when constructing positive sample pairs and negative sample pairs, it is necessary to satisfy that the motion posture data in the positive sample pairs come from the same sample person, and the motion posture data in the negative sample pairs come from different sample persons. The actual construction method can be set according to the needs of the implementer.

[0079] Optionally, training is generally carried out in batches. In each batch, the preprocessed motion posture data of each sample person and each enhanced motion posture data are used as a positive sample pair; and the preprocessed motion posture data of each sample person in the same batch is used as a positive sample, and the preprocessed motion posture data and each motion posture data in the enhanced motion posture data of other sample persons are used as negative samples respectively, and each positive sample is combined with a negative sample to form a negative sample pair.

[0080] Optionally, in each batch, the preprocessed motion posture data of each sample person and each enhanced motion posture data thereof can be used as a positive sample pair; and a certain enhanced motion posture data of each sample person in the same batch can be used as a positive sample, and each motion posture data in the preprocessed motion posture data and / or enhanced motion posture data of other sample persons can be used as a negative sample, and each positive sample and a negative sample can be combined to form a negative sample pair.

[0081] Step A7: Use the classification training set and the comparative learning training set to jointly train the preset sports ability assessment model to obtain a trained sports ability assessment model.

[0082] According to one embodiment of the present invention, the contrastive learning training set includes enhanced motion posture data. Therefore, when the classification training set and the contrastive learning training set are used to jointly train the model, both the preprocessed motion posture data and the enhanced motion posture data of the batch are input into the model.

[0083] The preprocessed motion posture data will be processed in sequence by the model's spatiotemporal feature extractor, skeleton point spatial enhancement module and prediction layer, wherein the skeleton point spatial enhancement module obtains the spatial enhancement features corresponding to the preprocessed motion posture data, and the prediction layer obtains the sports ability category of the sample person.

[0084] For the enhanced motion posture data, since it is mainly for comparative learning and there is no need to predict the classification results, in order to save computing resources, it is processed in turn by the model's spatiotemporal feature extractor and the bone point spatial enhancement module to obtain the spatial enhancement features corresponding to the enhanced motion posture data.

[0085] Subsequently, the model's parameter update is guided by the total loss function, the spatial enhancement features corresponding to the preprocessed motion posture data, the spatial enhancement features corresponding to the enhanced motion posture data, and the motion ability categories and labels of the sample persons.

[0086] During training, the total loss function includes a contrast loss sub-function and a classification loss sub-function, wherein during training, the total loss function is used to guide the parameter update of the motor ability evaluation model to minimize the total loss function value.

[0087] According to one embodiment of the present invention, the total loss function is calculated as follows:

[0088]

[0089] in, represents the total loss function, represents the classification loss sub-function, represents the contrast loss subfunction, express The weighting coefficient of express The weighting coefficient of . is a hyperparameter used to adjust , The weight of can be set according to the needs of the implementer. The technical solution can at least achieve the following beneficial technical effects: the solution adopts a time enhancement method based on contrastive learning, which can help the model learn differential features in limited samples. The model can learn time-invariant features, thereby improving its generalization ability.

[0090] Schematically, the classification loss subfunction is calculated as follows:

[0091]

[0092] in, Indicates the number of samples in the current batch, The output of the model is The sports ability categories of the sample personnel in the sample, Indicates The labels in the samples.

[0093] Schematically, the contrast loss sub-function adopts InfoNCE loss, which can be calculated as follows:

[0094]

[0095] Among them, sim(·) represents the similarity function, which is used to calculate the similarity between two spatial enhancement features in a positive sample pair or a negative sample pair. Represents the spatial enhancement features corresponding to the preprocessed motion posture data of a sample person, express The spatial enhancement features corresponding to the enhanced motion posture data of the corresponding sample personnel, express The preprocessed motion posture data or the spatial enhancement features corresponding to the enhanced motion posture data of other sample persons other than the corresponding sample persons, Represents a batch of data. The similarity function sim(·) can be implemented by calculating the dot product of two vectors. and Corresponding to a positive sample pair, and Corresponding to a negative sample pair. Schematically speaking, the preprocessed motion posture data of the input Perform random offset in the time dimension to obtain enhanced motion posture data Then the original data and augmented data The input models are respectively processed by the spatiotemporal feature extractor and the skeleton point spatial enhancement module to extract the corresponding spatial enhancement features. and , and according to the setting of positive sample pairs and negative sample pairs in the contrastive learning training set, the corresponding contrastive learning sub-loss can be calculated.

[0096] IV. Application Scenarios

[0097] According to one embodiment of the present invention, a method for evaluating athletic ability is provided, comprising: acquiring athletic data of a person to be evaluated; extracting original athletic posture data of the person to be evaluated from the athletic data of the person to be evaluated; preprocessing the original athletic posture data of the person to be evaluated according to preset data alignment requirements to obtain preprocessed athletic posture data of the person to be evaluated; and inputting the preprocessed athletic posture data of the person to be evaluated into a trained athletic ability evaluation model obtained according to a training method for the athletic ability evaluation model to evaluate the athletic ability of the person to be evaluated.

[0098] According to an example of the present invention, a schematic flow chart of a method for evaluating motor ability of a stroke patient is provided, such as Figure 4 As shown, the method includes:

[0099] K1. Obtain the motion data of the object;

[0100] K2, using computer vision algorithm to obtain the original motion posture data of the object;

[0101] K3, preprocessing the original motion posture data;

[0102] K4, perform data enhancement on the preprocessed motion posture data;

[0103] K5, respectively obtain the original motion posture data and the enhanced motion posture data;

[0104] K6, respectively using a spatiotemporal feature extractor to extract the spatiotemporal features of the preprocessed motion posture data and the spatiotemporal features of the enhanced motion posture data;

[0105] K7, respectively enhancing the spatial features of the preprocessed spatiotemporal features of the motion posture data and the enhanced spatiotemporal features of the motion posture data to obtain spatial enhanced features;

[0106] K8, use the total loss function to constrain model training, including contrastive learning loss and classification loss sub-functions;

[0107] K9. Model training is completed, and the trained model is used to output the sports ability assessment results of the person to be assessed.

[0108] 5. Experimental Data

[0109] In order to verify the effect of the present invention, the inventors also conducted comparative experiments and ablation experiments.

[0110] Part I: Comparative Experiment

[0111] (1) Experimental evaluation

[0112] The method of the present invention is designed for the assessment of skeletal point-based stroke motor ability in real medical scenarios, and is based on the skeletal point-based stroke motor ability assessment data collected in a real hospital environment.

[0113] The stroke patient motion data set is the motion data of real stroke patients collected in a hospital. A total of 100 stroke patients were recruited, and they were instructed to simulate the four assessment actions in the Brunner's staging scale, and the motion data of the patients were collected using the motion video data collection system HealthySrs. Each patient contains 7 motion video data, and a total of 700 data. The Mediapipe algorithm is used to identify the patient's bone point data, and the bone point data only includes 2D bone points. At the same time, we recruited 5 doctors to annotate the patient's motion data according to the Brunner's staging scale, and divide the patient's upper limb and lower limb motor function, sitting balance ability and standing balance ability into 4 categories, namely: 0-very weak, almost no motor ability; 1-weak, with weak motor function, obvious muscle compensation phenomenon; 2-with good motor ability, can perform the assessment action irregularly; 3-with strong motor ability. The stroke patient motion data set is acquired in the real world, and there is a large amount of bone point noise in the data set due to occlusion, background factors, etc. The participating patients and rehabilitation therapists were mainly from Guiyang Jinyang Hospital. Patients were informed of all data collection and evaluation processes in advance, and all data and processes were ethically reviewed by the WMA Declaration of Helsinki and the CIOMS International Ethical Guidelines for Biomedical Research Involving Human Subjects.

[0114] (2) Experimental setup

[0115] During the training process, the SGD optimizer is used to optimize the learning rate. On the N stroke patient movement dataset, the learning rate was initially set to 0.1 and 0.005, Momentum was and 0.005, the decay rate was 0.05, the batch size was 16, the number of training rounds was 100, and the number of spatiotemporal graph convolution modules was 4.

[0116] (3) Data preprocessing

[0117] In the data preprocessing stage, data frame length alignment and skeleton point coordinate alignment were performed.

[0118] Data frame length alignment: All data are unified in the time dimension. We use the last frame of data for length alignment. In the stroke patient movement dataset, we align the data length to 250 frames. The specific operation is to copy the last frame of data and add it to the end of the data until the data length is aligned.

[0119] Bone point coordinate alignment: In order to unify the data distribution and reduce the impact of the human body position in the image, we use the bone point 0 of the first frame of each data as the origin of the coordinate system and align other data frames to this coordinate system. The specific operation is to subtract the coordinate of the bone point 0 from the coordinate of each bone point.

[0120] (4) Comparison method

[0121] ST-GCN: A graph convolutional approach to learning spatial and temporal features from data as a baseline for skeleton-based motion estimation.

[0122] 2s-AGCN: Based on graph convolutional neural network, an adaptive topology map is added to each graph convolutional layer to enhance the long-distance spatial modeling capability of the graph convolutional layer.

[0123] CTR-GCN: A channel topology refinement module is proposed, which compresses and aggregates the channel dimensions and uses different graph convolutional networks to extract features for each channel.

[0124] It is worth mentioning that because the LSTM-based method experienced gradient explosion when verified on the motion data of stroke patients, the above GCN-based methods were mainly compared.

[0125] (5) Comparison method results

[0126] The methods were compared on the stroke patient movement dataset for upper and lower limb motor ability staging and sitting and standing balance staging. Accuracy, precision, recall and F1 score were used as evaluation indicators. The relevant experimental results are shown in Tables 1 and 2.

[0127] Table 1

[0128]

[0129] Table 1 shows the comparison results of the upper and lower limb motor ability assessment tasks of stroke patient movement datasets with other methods. In the upper limb assessment and lower limb assessment tasks, compared with other baseline methods, STSAug has improved the accuracy by at least 6% and 3%, respectively. At the same time, STSAug is also superior to other methods in terms of precision, recall, and F1 score indicators. STSAug is 7% and 4.6% higher than other methods in upper limb assessment tasks and lower limb assessment tasks, respectively. This shows the effectiveness of STSAug in upper and lower limb ability assessment.

[0130] Table 2

[0131]

[0132] Table 2 shows the comparison results of the sitting and standing balance ability assessment tasks of the stroke patient movement dataset with other methods. In the sitting balance assessment task, the method STSAug of the present invention is superior to other recognition methods in terms of accuracy, recall and F1 score, and is 2% and 0.4% higher than the baseline method in terms of accuracy and F1 score, respectively. In the standing balance assessment task, the method of the present invention is 4% higher than other methods in terms of accuracy, but is lower than the baseline method in terms of other evaluation indicators. This may be due to the imbalanced distribution of categories in the dataset.

[0133] In summary, the STSAug of the present invention is effective in improving other existing methods. The method of the present invention can be applied to the situation of small data volume and low data quality in real medical scenarios, and is superior to other motion assessment methods.

[0134] Part II: Ablation Experiment

[0135] In order to further explore the effectiveness of the method proposed in the present invention, the inventors also conducted ablation experiments. The method of the present invention consists of two parts: a skeleton point spatial enhancement module, referred to as SSA, and a temporal and spatial enhancement training process based on contrastive learning, referred to as TA-CL. The results of the upper limb function assessment task on stroke motion data are shown in Table 3.

[0136] Table 3 Ablation experiment results

[0137]

[0138] As can be seen from Table 3, in terms of accuracy, SSA and TA-CL have improved by 2% and 3% respectively, while precision, recall and F1-Score are also better than the ST-GCN method.

[0139] In general, the method of the present invention can enhance features in both time and space dimensions, improve the feature extraction ability and generalization of the model, and is suitable for situations where the data volume is small and the data quality is low in real medical scenarios. In addition, experiments show that the main contribution comes from the skeleton point space enhancement module and the space-time enhancement module based on contrastive learning, which can solve the problems of poor model generalization and low motion evaluation accuracy caused by data volume limitations and data noise in real medical scenarios. Among them, the SSA module adaptively learns and enhances the skeleton features related to motion evaluation, and by enhancing the skeleton point space features related to classification, it weakens the impact of irrelevant features on the model and reduces interference from irrelevant features. TA-CL uses data enhancement and contrastive learning techniques to help the model learn features that are invariant to space-time changes, so that the model learns time-independent features, thereby improving its generalization ability.

[0140] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0141] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0142] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0143] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a sports ability assessment model, characterized in that: include: Acquire initial training data, which includes multiple samples, each sample includes motion data of the sample person and its label, wherein the motion data is a video recorded of the object responding to a preset action requirement, and the label is the true value of the object's motion ability category; Extract the original motion posture data of the sample person from the motion data of each sample; According to the preset data alignment requirements, the original motion posture data of the sample personnel is preprocessed to obtain the preprocessed motion posture data; The preprocessed motion posture data replaces the motion data of the sample personnel to obtain a classification training set; Performing data enhancement on the preprocessed motion posture data to obtain enhanced motion posture data; Using the preprocessed motion posture data and the enhanced motion posture data, a contrastive learning training set is constructed, which includes positive sample pairs and negative sample pairs, wherein the motion posture data in the positive sample pairs are from the same sample person, and the motion posture data in the negative sample pairs are from different sample persons; The preset sports ability assessment model is jointly trained by using the classification training set and the comparative learning training set to obtain a trained sports ability assessment model.

2. The method according to claim 1, characterized in that Data augmentation is to perform an overall random translation of the preprocessed motion posture data along the frame direction, and when jointly training a preset motion ability evaluation model, the contrast loss sub-function is used to guide the model to shorten the distance between the features of the motion posture data in the positive sample pair and to extend the distance between the features of the motion posture data in the negative sample pair.

3. The method according to claim 1, characterized in that The preset sports ability assessment models include: A spatiotemporal feature extractor, used for extracting spatiotemporal features of input motion posture data from spatial and temporal dimensions; The skeleton point spatial enhancement module is used to map the spatiotemporal features of the motion posture data according to the learnable matrix, and then process it through the convolution layer, pooling layer and the first fully connected layer to obtain the spatial enhancement features; The prediction layer is used to determine the athletic ability category of the sample person based on the spatial enhancement features.

4. The method according to claim 3, characterized in that The skeleton point space enhancement module includes: The spatial weight enhancement layer is used to calculate the product of the spatiotemporal features of the motion posture data and the learnable matrix to obtain the enhanced feature matrix; The modeling layer is used to convolve the enhanced feature matrix according to the one-dimensional convolution layer to obtain the mapped feature matrix; The pooling layer is used to pool the mapped feature matrix to obtain the pooled features; The first fully connected layer is used to process the pooled features to obtain spatial enhanced features.

5. The method according to claim 3, characterized in that: The spatiotemporal feature extractor uses a spatiotemporal graph convolutional neural network with a depth of 3 or 4 layers.

6. The method according to any one of claims 1 to 5, characterized in that: When training the model, the parameters of the athletic ability evaluation model are updated according to the total loss function to minimize the total loss function value, wherein the total loss function includes a contrast loss sub-function and a classification loss sub-function.

7. The method according to claim 6, characterized in that The patient's motor ability includes upper limb motor ability, lower limb motor ability, sitting balance ability and standing balance ability, and each motor ability is divided into multiple categories to distinguish the strength of the motor ability.

8. A method for evaluating athletic ability, comprising: Obtaining movement data of the person to be evaluated; Extracting original motion posture data of the person to be evaluated from the motion data of the person to be evaluated; According to the preset data alignment requirements, the original motion posture data of the person to be evaluated is preprocessed to obtain the preprocessed motion posture data of the person to be evaluated; The pre-processed movement posture data of the person to be evaluated is input into the trained movement ability evaluation model obtained by the method according to any one of claims 1 to 6 to evaluate the movement ability of the person to be evaluated.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1-8 by executing the executable instructions.

Images