Abnormal motion detection method, system, terminal and storage medium

Abstract

The present invention provides a method, system, terminal, and storage medium for detecting motion anomalies. The method comprises: obtaining sample data, inputting the sample data into a motion anomaly detection model for spatiotemporal graph convolution, and obtaining sample spatiotemporal graph features; performing spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupled features, and performing category prediction on the sample decoupled features to obtain motion prediction results; determining a model loss based on the motion prediction results and the sample decoupled features, and updating parameters of the motion anomaly detection model based on the model loss until the motion anomaly detection model converges; and inputting the data to be detected into the converged motion anomaly detection model for motion anomaly detection, and obtaining a motion anomaly detection result. In an embodiment of the present invention, the converged motion anomaly detection model can effectively detect motion anomalies in the data to be detected, thereby improving the accuracy of motion anomaly detection.

Description

Technical Field

[0001] The present invention relates to the field of motion detection technology, and in particular to a motion abnormality detection method, system, terminal and storage medium. Background Art

[0002] Attention deficit hyperactivity disorder (ADHD), commonly known as ADHD, is the most common neurodevelopmental disorder in children and adolescents. Its clinical manifestations are difficulty concentrating, hyperactivity, impulsivity, emotional instability, learning difficulties, etc.

[0003] In the prior art, whether a patient has abnormal movements due to ADHD is generally determined mainly through subjective judgment of doctors and questionnaires, resulting in low accuracy in abnormal movement detection. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method, system, terminal and storage medium for detecting abnormal motion, so as to solve the problem of low accuracy of abnormal motion detection in the prior art.

[0005] The embodiment of the present invention is implemented as follows: a method for detecting abnormal motion, the method comprising:

[0006] Obtaining sample data, and inputting the sample data into the motion anomaly detection model for spatiotemporal graph convolution to obtain sample spatiotemporal graph features;

[0007] Performing spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupled features, and performing category prediction on the sample decoupled features to obtain an action prediction result;

[0008] Determining a model loss according to the action prediction result and the sample decoupling feature, and updating parameters of the action anomaly detection model according to the model loss until the action anomaly detection model converges;

[0009] The data to be detected is input into the converged motion anomaly detection model to perform motion anomaly detection to obtain a motion anomaly detection result.

[0010] Preferably, the sample data is input into the motion anomaly detection model for spatiotemporal graph convolution to obtain sample spatiotemporal graph features, including:

[0011] Calculating the neighborhood node index of the feature node in the sample data, determining the local domain feature according to the neighborhood node index, and calculating the edge convolution feature according to the local domain feature;

[0012] Performing graph convolution on the sample data to obtain graph convolution features, and splicing the graph convolution features and the edge convolution features to obtain splicing features;

[0013] Performing representative spatial average pooling on the spliced ​​features to obtain average pooled features, and performing hierarchical edge convolution on the average pooled features to obtain an attention map;

[0014] The attention map is combined with the graph convolution feature to obtain a combined feature, and the combined feature is summed according to the hierarchical dimension to obtain the sample spatiotemporal graph feature.

[0015] Preferably, the formula used to calculate the neighborhood node index of the characteristic node in the sample data includes:

[0016]

[0017] in, Feature nodes i of n Neighborhood node indexes, Indicates the i The characteristics of the feature nodes, Indicates the j The characteristics of the feature nodes, represents the neighbor algorithm function, n Indicates the number of neighbors set in the neighbor algorithm function;

[0018] The formula used to calculate the edge convolution feature based on the local area feature includes:

[0019]

[0020] in, represents a linear transformation, represents feature splicing, Indicates that the feature is in dimension n Perform maximum pooling on Indicates that the feature is copied T times, represents the edge convolution feature, X is the feature of the feature node, X n is the local area feature;

[0021] The formula used to perform graph convolution on the sample data includes:

[0022]

[0023] in, Indicates the k Layer-wise graph convolutional features, represents the input sample data, Represents point-by-point convolution operation, T represents the timing window size, Indicates the kThe adjacency matrix of the level, S represents the total edge set of the human joint points, and s represents the edge subset of the human joint points;

[0024] The formula used for performing representative spatial average pooling on the spliced ​​features includes:

[0025]

[0026] in, Represents the RSAP function, N k Representation level H k The number of joint points in v Indicates the joint point, t Represents a timing frame, Represents the joint point at the kth level v In the timing frame t The eigenvalues ​​on represents the average pooling feature, and max represents the maximum value function;

[0027] The formula used for performing hierarchical edge convolution on the average pooling feature includes:

[0028]

[0029] in, M represents the attention map, represents the layer-wise convolution, σ represents the sigmoid activation function, L Represents the total set of hierarchical points of the human body.

[0030] Preferably, performing spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupling features includes:

[0031] Performing temporal pooling on the sample spatiotemporal graph features to obtain first pooled features, and performing convolution on the first pooled features to obtain sample spatial features;

[0032] Performing spatial pooling on the sample spatiotemporal graph features to obtain a second pooling feature, and performing convolution on the second pooling feature to obtain a sample time feature;

[0033] The sample decoupling feature includes the sample time feature and the sample space feature.

[0034] Preferably, obtaining sample data includes:

[0035] Obtaining fuzzy label samples, and performing sample enhancement on the fuzzy label samples to obtain enhanced samples;

[0036] Performing label prediction on the fuzzy label sample and the enhanced sample according to the action anomaly detection model to obtain a first predicted label and a second predicted label;

[0037] Calculating an average of the first predicted label and the second predicted label to obtain a label average, and sharpening the label average to obtain a pseudo label;

[0038] Labeling the blurred label samples and the enhanced samples according to the pseudo labels, and obtaining clear label samples;

[0039] The clear label sample, the fuzzy label sample after label marking, and the enhanced sample are combined to obtain the sample data.

[0040] Preferably, the formula used to determine the model loss based on the action prediction result and the sample decoupling feature includes:

[0041]

[0042] in, represents the model loss, W CL represents the first weight hyperparameter, W u represents the second weight hyperparameter, represents the cross entropy loss, represents the mean square error loss, represents contrast loss;

[0043]

[0044] in, represents the number of samples of the fuzzy label samples, u represents the fuzzy label sample, q represents the pseudo label, represents the category probability of the fuzzy label sample prediction in the action prediction result, represents the data of the fuzzy label sample, Representing model parameters of the motion anomaly detection model;

[0045]

[0046] in, represents the sample number of the clear label sample, x Representing the clear label sample, p represents the true label of the clear label sample, represents the category probability of the clear label sample prediction in the action prediction result, L' represents the data of the clear label sample;

[0047]

[0048] in, represents the loss weight, represents the sample space characteristics, represents the sample time feature, represents the contrastive learning loss function, n Indicates the number of contrastive learning constructions.

[0049] Preferably, the contrastive learning loss function is:

[0050]

[0051] in, Indicates that in the action prediction result, the sample i For categories k The predicted probability of τ represents the sensitivity hyperparameter, Represents the compensation item, represents the penalty term, Representation category k The prototype representation of Representation category The prototype representation of F i Represents a sample i The sample time feature or the sample space feature;

[0052]

[0053] in, Representation category k The central feature representation of the false positive sample is, represents the set of confidence samples, represents the amount of data in the confidence sample set, Representation category k The amount of false positive samples in the dataset;

[0054]

[0055] in, Representation category k The central feature representation of the false negative sample is, Representation category k The amount of false negative samples in the dataset;

[0056]

[0057] in,F j Represents a sample j The sample time feature or the sample space feature;

[0058]

[0059] in, α represents the momentum term.

[0060] Another object of an embodiment of the present invention is to provide a motion abnormality detection system, the system comprising:

[0061] A spatiotemporal graph convolution module is used to obtain sample data and input the sample data into the motion anomaly detection model for spatiotemporal graph convolution to obtain sample spatiotemporal graph features;

[0062] A category prediction module is used to perform spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupled features, and perform category prediction on the sample decoupled features to obtain action prediction results;

[0063] a model training module, configured to determine a model loss based on the action prediction result and the sample decoupling feature, and update parameters of the action anomaly detection model based on the model loss until the action anomaly detection model converges;

[0064] The motion anomaly detection module is used to input the data to be detected into the converged motion anomaly detection model to perform motion anomaly detection and obtain a motion anomaly detection result.

[0065] The embodiments of the present invention can effectively extract sample spatiotemporal graph features from the sample data by inputting sample data into a motion anomaly detection model for spatiotemporal graph convolution, effectively extract sample decoupling features by performing spatiotemporal decoupling on the sample spatiotemporal graph features, effectively obtain motion prediction results for the sample data by performing category prediction on the sample decoupling features, and effectively obtain motion prediction results for the sample data by performing parameter updates on the motion anomaly detection model based on the model loss, so that the converged motion anomaly detection model can effectively perform motion anomaly detection on the data to be detected, eliminating the need for subjective judgment and questionnaire surveys to perform motion anomaly detection for ADHD, thereby improving the accuracy of motion anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of a method for detecting abnormal motion provided by a first embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of a motion anomaly detection model provided by the first embodiment of the present invention;

[0068] Figure 3is a schematic diagram of a feature refinement module provided in the first embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of a feature mixing module provided by the first embodiment of the present invention;

[0070] Figure 5 is a structural diagram of a motion abnormality detection system provided by a second embodiment of the present invention;

[0071] Figure 6 It is a structural diagram of a terminal device provided by the third embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0073] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0074] Example 1

[0075] See also Figure 1 , is a flow chart of a method for detecting abnormal motion provided by a first embodiment of the present invention. The method for detecting abnormal motion can be applied to any device or system. The method for detecting abnormal motion includes the following steps:

[0076] Step S10: acquiring sample data, and inputting the sample data into the motion anomaly detection model to perform spatiotemporal graph convolution to obtain sample spatiotemporal graph features;

[0077] See also Figure 2 The motion anomaly detection model adopts the spatiotemporal graph convolution semi-supervised ADHD recognition model (ADHD-RM) based on the temporal features of the human skeleton. The structure of the motion anomaly detection model consists of ten layers of nested spatiotemporal graph convolution modules (STGCM), feature refinement modules (FRM), feature blend modules (FBM), linear classification layers and softmax activation functions.

[0078] The model uses 3D human pose sequences as input. For the first 20 iterations of motion anomaly detection model training, only clearly labeled sample data is used for pre-training in conjunction with the spatiotemporal graph convolution module and the feature refinement module. Due to the label noise problem, the fuzzy labeled sample data is considered unlabeled data.

[0079] In this step, the sample data is convolved with the spatiotemporal graph based on the spatiotemporal graph convolution module to obtain the sample spatiotemporal graph features. The spatiotemporal graph convolution module represents the human skeleton sequence as a graph network consisting of several layers of joint nodes and skeletal edges. Through unified spatiotemporal modeling, it extracts joint temporal and spatial information. To enable the model to focus on local and global actions in the action stimulation paradigm, a weighted summation strategy is applied to the outputs of different levels in combination with the attention-guided hierarchical aggregation mechanism. This allows the model to focus on key local or global features, thereby helping the model distinguish subtle differences between ADHD subjects and normal subjects under similar action stimuli.

[0080] Optionally, the sample data is input into a motion anomaly detection model for spatiotemporal graph convolution to obtain sample spatiotemporal graph features, including:

[0081] Calculating the neighborhood node index of the feature node in the sample data, determining the local domain feature according to the neighborhood node index, and calculating the edge convolution feature according to the local domain feature;

[0082] Edge convolution is used to construct a local neighborhood graph and extract local domain features from the graph structure. Using the K-Nearest Neighbor (K-NN) algorithm based on the Euclidean distance between two nodes in the feature space, the n adjacent nodes with the smallest distance are selected to obtain the neighborhood node index. The neighborhood node index of each node is calculated to obtain the local domain features of each node.

[0083] Performing graph convolution on the sample data to obtain graph convolution features, and splicing the graph convolution features and the edge convolution features to obtain splicing features;

[0084] Among them, in the human skeleton structure, the human spine node is selected as the central node, and the human skeleton structure is expanded into a hierarchical tree structure to obtain a hierarchical node set. In the human skeleton structure, the central node is the first level, the nodes connected to the central node are the second level, and the nodes connected to the second level are the third level. The hierarchical node set contains the hierarchical information of the graph, which is represented as an adjacency matrix , N L Indicates the number of levels, N SThe number of edge subset types is defined as 3. Through hierarchical decomposition, the edges between all nodes in the same semantic space are obtained by connecting all nodes in the edge sets of adjacent levels, forming a hierarchical graph structure with fully connected edges;

[0085] Adjacency Matrix Defined as:

[0086]

[0087] in, H k Indicates the k A set of hierarchical nodes of levels, H k+1 Indicates the k+ A set of hierarchical nodes at one level, NL represents the number of hierarchical sets, s id 、 s cp 、 s cf represent the same edge subset, centripetal edge subset and eccentric edge subset respectively. Represents the concatenation of three edge subsets. The same edge subset represents all joints in the previous and next levels, and the set of edges connected to their own joints. For example, the first level contains a joint e1, and the second level contains e2 and e3. Then the same edge subset between the first and second levels includes the set of edges connecting e1, e2, and e3 to their own joints respectively. The centripetal edge subset represents the set of directed edges connecting the joints of the previous level to the joints of the next level. The eccentric edge subset represents the set of directed edges connecting the joints of the next level to the joints of the previous level.

[0088]

[0089] S represents the total set of the same edge subset, centripetal edge subset, and centrifugal edge subset.

[0090] Performing representative spatial average pooling on the spliced ​​features to obtain average pooled features, and performing hierarchical edge convolution on the average pooled features to obtain an attention map;

[0091] The Attention-Guided Hierarchy Aggregation (A-HA) module is used to perform Representative Spatial Average Pooling (RSAP) on the concatenated features. RSAP extracts the time series frames with the maximum score from each layer and calculates weighted values ​​based on the features of these frames to avoid scaling bias caused by inconsistent numbers of node connections.

[0092] After the RSAP layer, all hierarchical features are regarded as nodes on the graph, and the similarities between levels are learned through hierarchical edge convolution to obtain the attention map;

[0093] Combining the attention map with the graph convolution feature to obtain a combined feature, and summing the combined feature according to the hierarchical dimension to obtain the sample spatiotemporal graph feature;

[0094] The attention map is multiplied by the graph convolution features, and the hierarchical dimensions are summed to obtain the sample spatiotemporal graph features. Compared with the traditional temporal convolution method with a fixed convolution kernel size, this step convolves the output features of the graph convolution in the time dimension through convolution kernels of different sizes, thereby extracting temporal features at different time scales. The output dimension is the input dimension of the next layer of graph convolution. This step does not simply rely on larger convolution kernels, but combines dilated convolution to expand the receptive field, so that while improving the computational efficiency of the model, it can also extract time series with longer time spans. Ultimately, the sample spatiotemporal graph features contain richer temporal information, which can effectively improve the model's ability to extract features from skeleton data.

[0095] Furthermore, the formula used to calculate the neighborhood node index of the characteristic node in the sample data includes:

[0096]

[0097] in, Feature nodes i of n Neighborhood node indexes, Indicates the i The characteristics of the feature nodes, Indicates the j The characteristics of the feature nodes, represents the neighbor algorithm function, n Indicates the number of neighbors set in the neighbor algorithm function;

[0098] The formula used to calculate the edge convolution feature based on the local area feature includes:

[0099]

[0100] in, represents a linear transformation, represents feature splicing, Indicates that the feature is in dimension n Perform maximum pooling on Indicates that the feature is copied T times, represents the edge convolution feature, X is the feature of the feature node, X n is the local area feature;

[0101] The formula used to perform graph convolution on the sample data includes:

[0102]

[0103] in, Indicates the k Layer-wise graph convolutional features, represents the input sample data, Represents point-by-point convolution operation, T represents the timing window size, Indicates the k The adjacency matrix of the level, S represents the total edge set of the human joint points, and s represents the edge subset of the human joint points;

[0104] The formula used for performing representative spatial average pooling on the spliced ​​features includes:

[0105]

[0106] in, Represents the RSAP function, N k Representation level H k The number of joint points in v Indicates the joint point, t Represents a timing frame, Represents the joint point at the kth level v In the timing frame t The eigenvalues ​​on represents the average pooling feature, and max represents the maximum value function;

[0107] The formula used for performing hierarchical edge convolution on the average pooling feature includes:

[0108]

[0109] in, M represents the attention map, represents the layer-wise convolution, σ represents the sigmoid activation function, L Represents the total set of hierarchical points of the human body.

[0110] Going further, get sample data, including:

[0111] Obtaining fuzzy label samples, and performing sample enhancement on the fuzzy label samples to obtain enhanced samples;

[0112] Performing label prediction on the fuzzy label sample and the enhanced sample according to the action anomaly detection model to obtain a first predicted label and a second predicted label;

[0113] Calculating an average of the first predicted label and the second predicted label to obtain a label average, and sharpening the label average to obtain a pseudo label;

[0114] The blurred label sample is rotated along the y-axis at a random angle to obtain an enhanced sample. The blurred label sample and its enhanced sample are predicted using a motion anomaly detection model to perform pseudo-label prediction. The pseudo-labels of the two generated samples are averaged and sharpened to serve as the common pseudo-label of the two samples.

[0115] Labeling the blurred label samples and the enhanced samples according to the pseudo labels, and obtaining clear label samples;

[0116] The clear label sample, the fuzzy label sample after label marking, and the enhanced sample are combined to obtain the sample data.

[0117] Step S20, performing spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupled features, and performing category prediction on the sample decoupled features to obtain an action prediction result;

[0118] Among them, see Figure 3 The feature refinement module consists of a spatiotemporal decoupling module and a contrastive learning module. The spatiotemporal decoupling module extracts temporal and spatial features through pooling and convolution operations, respectively, thereby reducing the interference of similar actions in time and space. The decoupled features can represent independent changes in time and space, improving the precision of feature expression. At the same time, the contrastive learning module optimizes the decoupled feature expression to maximize the cosine distance between the action features of ADHD subjects and normal subjects under the same paradigm, thereby improving classification accuracy and enhancing the model's ability to discriminate difficult samples.

[0119] Optionally, performing spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupling features includes:

[0120] Performing temporal pooling on the sample spatiotemporal graph features to obtain a first pooled feature, and performing convolution on the first pooled feature to obtain a sample spatial feature;

[0121] Performing spatial pooling on the sample spatiotemporal graph features to obtain a second pooling feature, and performing convolution on the second pooling feature to obtain a sample temporal feature;

[0122] The sample decoupling feature includes the sample time feature and the sample space feature.

[0123] Step S30, determining a model loss according to the action prediction result and the sample decoupling feature, and updating parameters of the action anomaly detection model according to the model loss until the action anomaly detection model converges;

[0124] Among them, if the sample is correctly classified, it is considered to be a confident sample. The sample space features and sample time features of the confident sample are collected, and the collected sample space features and sample time features are updated to the global feature representation of the corresponding category through exponential moving average, which is called prototype representation.

[0125] In this embodiment, fuzzy label samples are divided into two categories:

[0126] False negative (0 Negative, FN): For example, it should belong to the category k , but is misclassified as a sample of other categories; False Positive (0 Positive, FP): For example, it should belong to other categories, but is misclassified as a category k Sample.

[0127] Optionally, the formula used to determine the model loss based on the action prediction result and the sample decoupling feature includes:

[0128]

[0129] in, represents the model loss, W CL represents the first weight hyperparameter, W u represents the second weight hyperparameter, represents the cross entropy loss, represents the mean square error loss, represents contrast loss;

[0130]

[0131] in, represents the number of samples of the fuzzy label samples, u represents the fuzzy label sample,q represents the pseudo label, represents the category probability of the fuzzy label sample prediction in the action prediction result, represents the data of the fuzzy label sample, Representing model parameters of the motion anomaly detection model;

[0132]

[0133] in, represents the sample number of the clear label sample, x Representing the clear label sample, p represents the true label of the clear label sample, represents the category probability of the clear label sample prediction in the action prediction result, L ' represents the data of the clear label sample;

[0134]

[0135] in, represents the loss weight, represents the sample space characteristics, represents the sample time feature, represents the contrastive learning loss function, n Indicates the number of times the model performs comparative learning on the intermediate features of spatiotemporal graph convolution at different levels. In this embodiment, it is set to 4.

[0136] Furthermore, the contrastive learning loss function is:

[0137]

[0138] in, Indicates that in the action prediction result, the sample i For categories k The predicted probability of τ represents the sensitivity hyperparameter, Represents the compensation item, represents the penalty term, Representation category k The prototype representation of Representation category The prototype representation of F i Represents a sample i The sample time feature or the sample space feature;

[0139]

[0140] in, Representation category k The central feature representation of the false positive sample is, represents the set of confidence samples, represents the amount of data in the confidence sample set, Representation category k The set of false positive samples, The amount of data representing the set of false positive samples.

[0141] In order to calibrate the prediction of fuzzy label samples, the category k The confidence samples of are used as anchor points, and compensation and penalty terms are introduced in the feature space to adjust the feature representation of FN and FP samples. Used to make FN samples closer to the category k Confidence sample of . Used to keep FP samples away from the category k Confidence sample of .

[0142]

[0143] in, Representation category k The central feature representation of the false negative sample is, Representation category k The set of false negative samples, The amount of data representing the set of false negative samples;

[0144]

[0145] in, F j Represents a sample j The sample time feature or the sample space feature;

[0146]

[0147] in, α represents the momentum term.

[0148] In this embodiment, the loss function encourages samples F i The feature vector of is close to the category to which it belongs k Prototype representation p k ,punish F i The feature vector of is close to the prototype representation of the non-belonging category By using compensation and penalty terms, we provide additional guidance for fuzzy label samples, and through contrastive learning, we use confident samples to strongly supervise those TP samples with weaker confidence, thereby optimizing and calibrating the classification ability of the model.

[0149] Step S40, inputting the data to be detected into the converged motion anomaly detection model to perform motion anomaly detection and obtain a motion anomaly detection result;

[0150] Among them, when the data to be detected in the abnormal motion detection result is identified as abnormal motion data, it is determined that the person to be detected corresponding to the data to be detected may have ADHD, effectively achieving the ADHD screening effect.

[0151] The Azure-Kinect camera array is used to perform three-dimensional human posture estimation to obtain the data to be detected of the person to be detected. Since the human body may be blocked by limbs when in motion, the accuracy of posture estimation is reduced. This embodiment is based on a multi-view solution of three Azure-Kinect depth camera arrays, which can combine color information and depth information from multiple angles to provide richer depth information and more comprehensive three-dimensional human posture data, and use a checkerboard calibration plate to align the spatial coordinate systems of the three depth cameras, and transform the human posture estimation coordinates of the sub-camera into the main camera coordinate system. Combined with the posture coordinate median fusion method, the accuracy and robustness of posture estimation can be improved.

[0152] Azure-Kinect cameras align their time by daisy-chaining the synchronization ports of multiple cameras using 3.5mm audio cables. Synchronization works by calculating the timestamp difference between the master and slave devices and setting a fixed threshold. If the timestamp difference exceeds the threshold, the device with the smaller timestamp will retake the image to achieve time synchronization again.

[0153] A homemade 60cm*60cm hard camera calibration board was used, where each checkerboard square was 5cm*5cm in size. The person to be tested stood in front of the Azure-Kinect depth camera array holding the calibration board. The cameras started synchronous shooting and the calibration board was slowly moved to ensure that the video sequence captured by each camera contained 200 frames of calibration board images.

[0154] definition w is the world coordinate system, the origin is set to the upper left corner of the calibration plate, and the camera coordinate system is defined as o. The intrinsic parameter matrix of each camera is The intrinsic matrix is ​​responsible for converting the coordinate points in the camera coordinate system to the pixel coordinate system. m For the reference camera, in the camera coordinate system, the Euclidean transformation from sub-camera k to reference camera m is for:

[0155]

[0156] in, ok Indicates sub-camerak The camera coordinate system, om Indicates the reference camera m The camera coordinate system. w Represents the world coordinate system. R represents the rotation matrix in the transformation matrix, Describing sub-cameras k Camera coordinate system and reference camera m Rotation relationship between camera coordinate systems; translation vector t Indicates that the slave camera k The coordinate system origin to the reference camera m The translation of the coordinate system origin. By using the world coordinate system as the intermediate coordinate system, the rotation relationship and translation vector are converted into the world coordinate system and then converted into the reference camera. m The camera coordinate system is expressed as follows:

[0157]

[0158]

[0159] in, T represents the matrix transpose, Indicates sub-camera k The estimated human body point coordinates to the reference camera m The camera coordinate system transformation matrix, is a series of 3D points in the coordinate system of the chessboard calibration plate, It is the 2D coordinate projected into the pixel coordinate system. The PnP algorithm calculates the Euclidean transformation based on the correspondence between 3D points and 2D points. The rest of the calculations are the same as above. k To the reference camera m After the transformation matrix is ​​obtained, the spatial alignment of the three cameras can be achieved, and the 3D skeleton information predicted by the sub-camera is converted to the reference camera m 's coordinate system.

[0160] In the RGB-D camera array, each camera can estimate the position of 32 joints of the human body. For the j-th joint estimated by the k-th camera, its three-dimensional coordinates can be expressed as:

[0161]

[0162] in, j Represents the node index of the 32 joint points in the human body, Indicates that the joint points of 32 human bodies are in the sub-camera k in x Axis coordinate position, Indicates that the joint points of 32 human bodies are in the sub-camerak in y Axis coordinate position, Indicates that the joint points of 32 human bodies are in the sub-camera k in z Axis coordinate position, Indicates that the joint points of 32 human bodies are in the sub-camera k A set of three-dimensional coordinate values ​​in .

[0163] In order to fuse the estimation results of multiple cameras, it is necessary to obtain the three-dimensional coordinate points of sub-camera k Switching to the reference camera m Based on the Euclidean transformation formula between cameras, the point can be transformed from the camera k The coordinate system of the reference camera is transformed to m Coordinate system:

[0164]

[0165] For each joint j , convert from sub-camera to base camera m The estimated result after the coordinate system is a set , in order to obtain the final three-dimensional coordinate estimate, the formula is as follows:

[0166]

[0167] in, Indicates the coordinates of the human body estimated point of the first Azure-Kinect camera in the reference camera m Coordinate representation in the coordinate system, Indicates the coordinates of the second Azure-Kinect camera's human body estimation point on the reference camera m Coordinate representation in the coordinate system, Indicates the coordinates of the human body estimated point of the third Azure-Kinect camera in the reference camera m Coordinate representation in the coordinate system, Represents the final estimate after the coordinates of the three Azure-Kinect cameras are fused.

[0168] All points in the set are 3D estimates of the same point in the camera coordinate system of the reference camera m. Considering that the human body may block each other when in motion, which may cause the coordinates of the Azure-Kinect camera to jitter when estimating the human pose, we define a function , take the median value of (x, y, z) estimated by all cameras for the joint points to obtain a relatively stable and accurate human posture estimation structure, that is, to obtain the data to be detected.

[0169] See also Figure 4 In order to solve the problem of limited data volume in the dataset, this embodiment designs a feature mixing module and adopts a semi-supervised learning strategy for model training. Since fuzzy label samples may have certain noise labels, relying directly on the initial labels will have a negative impact on model training. To solve this problem, the model is used to generate pseudo labels for training based on the prediction results of fuzzy label samples and their data augmented samples, instead of using the initial labels. Afterwards, the feature mixing module mixes the high-dimensional features of the clear label samples and the fuzzy label samples, and mines the potential information in the fuzzy label samples, so that the model can learn more diverse sample features, which helps the model better adapt to unknown or fuzzy label samples and improves the classification accuracy.

[0170] The Mixup method is a data enhancement technique. Its basic idea is to perform linear interpolation on two samples and their corresponding labels according to weights. The formula is as follows:

[0171]

[0172] in, x i and x j are two sample features, y i and y j They are x i 、 x j The corresponding label, For Beta The mixing coefficient of the samples from the distribution, Control the weight distribution of sampling, Indicates based on x i and x j Mixed features after feature mixing, Indicates based on y i and y j Mixed labels after label mixing.

[0173] In pose sequence research, because 3D pose data contains rich spatiotemporal information, directly performing a Mixup operation on the sequence data can cause the sample's spatiotemporal information to become distorted, leading to the loss of key features. Therefore, in this embodiment, a Mixup operation is performed on the high-dimensional feature space of the clean and fuzzy label samples after feature extraction to generate mixed features and corresponding mixed pseudo-labels. The new mixed features and labels are treated as new training sample features for subsequent classification training.

[0174] Compared to the original Mixup method, this implementation leverages the semantic expressiveness and smooth distribution of hidden layer features to generate more semantically sound interpolated samples in the feature space. This not only generates new samples through interpolation in the feature space but also directly influences the intermediate representation of the neural network, providing the model with more effective training information, thereby enabling the model to learn more robust feature representations and decision boundaries.

[0175] In this embodiment, clear label samples, fuzzy label samples and their enhanced samples are forward propagated through the model. For clear label sample data, after passing through the intermediate spatiotemporal graph convolution unit, its features are passed into the feature refinement module, the features are decoupled in time and space, and the time features and spatial features are extracted respectively. Then, the contrastive learning mechanism optimizes the decoupled feature expression so that the cosine distance of the action features of ADHD subjects and normal subjects under the same paradigm is increased as much as possible, making it easier to distinguish difficult samples. For fuzzy label samples, after the fifth layer of spatiotemporal graph convolution, the data features and labels of the clear label samples are mixed with the features and labels of the fuzzy label samples to generate the features and labels of the new sample data, which are then trained subsequently. Finally, by calculating the cross entropy loss and contrast loss between the predicted probability and label of the clear label sample and the mean square error loss between the predicted label and the pseudo label of the fuzzy label sample, back propagation is performed to update the model parameters.

[0176] This embodiment effectively extracts sample spatiotemporal graph features from the sample data by inputting them into a motion anomaly detection model for spatiotemporal graph convolution. By performing spatiotemporal decoupling on the sample spatiotemporal graph features, sample decoupling features can be effectively extracted. By performing category prediction on the sample decoupling features, motion prediction results for the sample data can be effectively obtained. The motion anomaly detection model parameters are updated based on the model loss, enabling the converged motion anomaly detection model to effectively detect motion anomalies in the data being tested. This eliminates the need for subjective judgment and questionnaires to detect motion anomalies in ADHD, thereby improving the accuracy of motion anomaly detection. By combining a deep learning model with high-precision posture data, children with ADHD can be quickly and objectively identified, effectively addressing issues with ambiguous labels and behavioral features. The introduced feature refinement module and contrastive learning mechanism enhance the classification capability of samples with ambiguous labels, improving the model's recognition accuracy. Furthermore, the proposed preprocessing method significantly improves data reliability through coordinate fusion and normalization to address occlusion issues in posture data. The ADHD screening based on this embodiment can not only be effectively applied in areas with scarce medical resources, but also improve the efficiency and accuracy of ADHD screening, providing strong technical support for medical institutions.

[0177] Example 2

[0178] See also Figure 5 , is a schematic diagram of the structure of an abnormal motion detection system 100 provided in a second embodiment of the present invention, comprising:

[0179] The spatiotemporal graph convolution module 10 is used to obtain sample data and input the sample data into the motion anomaly detection model to perform spatiotemporal graph convolution to obtain sample spatiotemporal graph features.

[0180] Optionally, the spatiotemporal graph convolution module 10 is further configured to: calculate a neighborhood node index of a feature node in the sample data, determine a local domain feature based on the neighborhood node index, and calculate an edge convolution feature based on the local domain feature;

[0181] Performing graph convolution on the sample data to obtain graph convolution features, and splicing the graph convolution features and the edge convolution features to obtain splicing features;

[0182] Performing representative spatial average pooling on the spliced ​​features to obtain average pooled features, and performing hierarchical edge convolution on the average pooled features to obtain an attention map;

[0183] The attention map is combined with the graph convolution feature to obtain a combined feature, and the combined feature is summed according to the hierarchical dimension to obtain the sample spatiotemporal graph feature.

[0184] Furthermore, the spatiotemporal graph convolution module 10 is further configured to: obtain fuzzy label samples, and perform sample enhancement on the fuzzy label samples to obtain enhanced samples;

[0185] Performing label prediction on the fuzzy label sample and the enhanced sample according to the action anomaly detection model to obtain a first predicted label and a second predicted label;

[0186] Calculating an average of the first predicted label and the second predicted label to obtain a label average, and sharpening the label average to obtain a pseudo label;

[0187] Labeling the blurred label samples and the enhanced samples according to the pseudo labels, and obtaining clear label samples;

[0188] The clear label sample, the fuzzy label sample after label marking, and the enhanced sample are combined to obtain the sample data.

[0189] The category prediction module 11 is configured to perform spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupled features, and perform category prediction on the sample decoupled features to obtain action prediction results.

[0190] Optionally, the category prediction module 11 is further configured to: perform temporal pooling on the sample spatiotemporal graph features to obtain first pooled features, and perform convolution on the first pooled features to obtain sample spatial features;

[0191] Performing spatial pooling on the sample spatiotemporal graph features to obtain a second pooling feature, and performing convolution on the second pooling feature to obtain a sample time feature;

[0192] The sample decoupling feature includes the sample time feature and the sample space feature.

[0193] The model training module 12 is used to determine the model loss according to the action prediction result and the sample decoupling feature, and update the parameters of the action anomaly detection model according to the model loss until the action anomaly detection model converges.

[0194] The motion anomaly detection module 13 is configured to input the data to be detected into the converged motion anomaly detection model to perform motion anomaly detection and obtain a motion anomaly detection result.

[0195] In this embodiment, by inputting sample data into the motion anomaly detection model for spatiotemporal graph convolution, the sample spatiotemporal graph features in the sample data can be effectively extracted, and by performing spatiotemporal decoupling on the sample spatiotemporal graph features, the sample decoupling features can be effectively extracted, and by performing category prediction on the sample decoupling features, the motion prediction results of the sample data can be effectively obtained. The parameters of the motion anomaly detection model are updated based on the model loss, so that the converged motion anomaly detection model can effectively perform motion anomaly detection on the detection data, and there is no need to use subjective judgment and questionnaire surveys to detect motion anomalies in ADHD, thereby improving the accuracy of motion anomaly detection.

[0196] Example 3

[0197] Figure 6 This is a block diagram of a terminal device 2 provided in the third embodiment of the present application. Figure 6 As shown, the terminal device 2 of this embodiment includes: a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for the abnormal motion detection method. When the processor 20 executes the computer program 22, the steps of each embodiment of the abnormal motion detection method described above are implemented.

[0198] Exemplarily, the computer program 22 may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 20 to implement the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 22 in the terminal device 2. The terminal device may include, but is not limited to, a processor 20 and a memory 21.

[0199] The processor 20 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0200] The memory 21 can be an internal storage unit of the terminal device 2, such as a hard drive or memory of the terminal device 2. The memory 21 can also be an external storage device of the terminal device 2, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped with the terminal device 2. Furthermore, the memory 21 can include both an internal storage unit of the terminal device 2 and an external storage device. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 can also be used to temporarily store data that has been output or is about to be output.

[0201] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0202] If the integrated module is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be either non-volatile or volatile. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunications signals, and software distribution media. It should be noted that the content contained in computer-readable storage media can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunications signals.

[0203] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for detecting abnormal motion, characterized in that: The method comprises: Obtaining sample data, and inputting the sample data into the motion anomaly detection model for spatiotemporal graph convolution to obtain sample spatiotemporal graph features; Performing spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupled features, and performing category prediction on the sample decoupled features to obtain an action prediction result; Determining a model loss according to the action prediction result and the sample decoupling feature, and updating parameters of the action anomaly detection model according to the model loss until the action anomaly detection model converges; Inputting the data to be detected into the converged motion anomaly detection model to perform motion anomaly detection and obtain the motion anomaly detection result; The sample data is input into the action anomaly detection model for spatiotemporal graph convolution to obtain sample spatiotemporal graph features, including: Calculating the neighborhood node index of the feature node in the sample data, determining the local domain feature according to the neighborhood node index, and calculating the edge convolution feature according to the local domain feature; Performing graph convolution on the sample data to obtain graph convolution features, and splicing the graph convolution features and the edge convolution features to obtain splicing features; Performing representative spatial average pooling on the spliced ​​features to obtain average pooled features, and performing hierarchical edge convolution on the average pooled features to obtain an attention map; Combining the attention map with the graph convolution feature to obtain a combined feature, and summing the combined feature according to the hierarchical dimension to obtain the sample spatiotemporal graph feature; Performing spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupling features includes: Performing temporal pooling on the sample spatiotemporal graph features to obtain a first pooled feature, and performing convolution on the first pooled feature to obtain a sample spatial feature; Performing spatial pooling on the sample spatiotemporal graph features to obtain a second pooling feature, and performing convolution on the second pooling feature to obtain a sample temporal feature; The sample decoupling feature includes the sample time feature and the sample space feature.

2. The method for detecting abnormal motion according to claim 1, wherein: The formula used to calculate the neighborhood node index of the characteristic node in the sample data includes: in, Feature nodes i of n Neighborhood node indexes, Indicates the i The characteristics of the feature nodes, Indicates the j The characteristics of the feature nodes, represents the neighbor algorithm function, n Indicates the number of neighbors set in the neighbor algorithm function; The formula used to calculate the edge convolution feature based on the local area feature includes: in, represents a linear transformation, represents feature splicing, Indicates that the feature is in dimension n Perform maximum pooling on Indicates that the feature is copied T times, represents the edge convolution feature, X is the feature of the feature node, X n is the local area feature; The formula used to perform graph convolution on the sample data includes: in, Indicates the k Layer-wise graph convolutional features, represents the input sample data, Represents point-by-point convolution operation, T represents the timing window size, Indicates the k The adjacency matrix of the level, S represents the total edge set of the human joint points, and s represents the edge subset of the human joint points; The formula used for performing representative spatial average pooling on the spliced ​​features includes: in, Represents the RSAP function, N k Representation level H k The number of joint points in v Indicates the joint point, t Represents a timing frame, Represents the joint point at the kth level v In the timing frame t The eigenvalues ​​on represents the average pooling feature, and max represents the maximum value function; The formula used for performing hierarchical edge convolution on the average pooling feature includes: in, M represents the attention map, represents the layer-wise convolution, σ represents the sigmoid activation function, L Represents the total set of hierarchical joints of the human body.

3. The method for detecting abnormal motion according to claim 1, wherein: Get sample data, including: Obtaining fuzzy label samples, and performing sample enhancement on the fuzzy label samples to obtain enhanced samples; Performing label prediction on the fuzzy label sample and the enhanced sample according to the action anomaly detection model to obtain a first predicted label and a second predicted label; Calculating an average of the first predicted label and the second predicted label to obtain a label average, and sharpening the label average to obtain a pseudo label; Labeling the blurred label samples and the enhanced samples according to the pseudo labels, and obtaining clear label samples; The clear label sample, the fuzzy label sample after label marking, and the enhanced sample are combined to obtain the sample data.

4. The method for detecting abnormal motion according to claim 3, wherein: The formula used to determine the model loss based on the action prediction result and the sample decoupling feature includes: in, represents the model loss, W CL represents the first weight hyperparameter, W u represents the second weight hyperparameter, represents the cross entropy loss, represents the mean square error loss, represents contrast loss; in, represents the number of samples of the fuzzy label samples, u represents the fuzzy label sample, q represents the pseudo label, represents the category probability of the fuzzy label sample prediction in the action prediction result, represents the data of the fuzzy label sample, Representing model parameters of the motion anomaly detection model; in, represents the sample number of the clear label sample, x Representing the clear label sample, p represents the true label of the clear label sample, represents the category probability of the clear label sample prediction in the action prediction result, L ' represents the data of the clear label sample; in, represents the loss weight, represents the sample space characteristics, represents the sample time feature, represents the contrastive learning loss function, n Indicates the number of contrastive learning constructions.

5. The method for detecting abnormal motion according to claim 4, wherein: The contrastive learning loss function is: in, Indicates that in the action prediction result, the sample i For categories k The predicted probability of τ represents the sensitivity hyperparameter, Represents the compensation item, represents the penalty term, Representation category k The prototype representation of Representation category The prototype representation of F i Representation sample i The sample time feature or the sample space feature; in, Representation category k The central feature representation of the false positive sample is, represents the set of confidence samples, represents the amount of data in the confidence sample set, Representation category k The amount of false positive samples in the dataset; in, Representation category k The central feature representation of the false negative sample is, Representation category k The amount of false negative samples in the dataset; in, F j Representation sample j The sample time feature or the sample space feature; in, α represents the momentum term.

6. A motion abnormality detection system, characterized in that: The system comprises: A spatiotemporal graph convolution module is used to obtain sample data and input the sample data into the motion anomaly detection model for spatiotemporal graph convolution to obtain sample spatiotemporal graph features; A category prediction module is used to perform spatiotemporal decoupling on the sample spatiotemporal graph features to obtain sample decoupled features, and perform category prediction on the sample decoupled features to obtain action prediction results; a model training module, configured to determine a model loss based on the action prediction result and the sample decoupling feature, and update parameters of the action anomaly detection model based on the model loss until the action anomaly detection model converges; An action anomaly detection module, configured to input the data to be detected into the converged action anomaly detection model to perform action anomaly detection and obtain an action anomaly detection result; The spatiotemporal graph convolution module is also used to: Calculating the neighborhood node index of the feature node in the sample data, determining the local domain feature according to the neighborhood node index, and calculating the edge convolution feature according to the local domain feature; Performing graph convolution on the sample data to obtain graph convolution features, and splicing the graph convolution features and the edge convolution features to obtain splicing features; Performing representative spatial average pooling on the spliced ​​features to obtain average pooled features, and performing hierarchical edge convolution on the average pooled features to obtain an attention map; Combining the attention map with the graph convolution feature to obtain a combined feature, and summing the combined feature according to the hierarchical dimension to obtain the sample spatiotemporal graph feature; The category prediction module is further configured to: Performing temporal pooling on the sample spatiotemporal graph features to obtain a first pooled feature, and performing convolution on the first pooled feature to obtain a sample spatial feature; Performing spatial pooling on the sample spatiotemporal graph features to obtain a second pooling feature, and performing convolution on the second pooling feature to obtain a sample temporal feature; The sample decoupling feature includes the sample time feature and the sample space feature.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images