A two-stage repetitive motion counting and motion completeness evaluation method based on space-time features

Abstract

The application discloses a two-stage repeated action counting and action completeness evaluation method based on space-time features, comprising the following steps: obtaining video frames from a video containing two-stage repeated actions; extracting the space-time features of each video frame through a space feature extraction network; extracting the space-time features from the space features of the video frames through a time sequence feature extraction network; obtaining a prediction value sequence corresponding to each video frame according to the space-time features through a prediction head; and identifying the two-stage repeated actions and counting the two-stage actions according to the repeated mode of the prediction values in the prediction value sequence. The method based on deep learning only needs to label the starting frame and the ending frame of each stage of the two-stage repeated actions and the current action completeness, and the artificial labeling cost is low; the use of the convolutional neural network effectively avoids the interference of camera shaking and is beneficial to the two-stage repeated action counting and the action completeness evaluation.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically to a two-stage repetitive action counting and action integrity evaluation method based on spatiotemporal features. Background Technology

[0002] Counting repetitive movements is widespread in human activities such as physical exercise, and counting some physical activities is helpful for health monitoring and planning. Traditional methods of counting repetitive movements rely on sensor detection. While this method is relatively accurate, it depends on specialized motion equipment with sensors, and different specialized equipment is usually required for different sports, resulting in high design costs and limited applicability.

[0003] Computer vision-based repetitive motion counting methods can be easily deployed on mobile clients, are user-friendly, and have good scalability. Currently, most repetitive motion counting methods on the market are based on skeletal keypoints, calculating the angles of human joints to count motions.

[0004] In previous studies, Xu Jie et al. proposed a human motion counting method based on skeletal key points (patent application number 202110239387.6), which uses a deep learning network to obtain the coordinates of human skeletal points and calculates the changes in the angle between human skeletal points based on mathematical methods to complete the motion counting. This method requires the human body to be completely inside the camera and relies too much on the performance of the human skeletal point recognition network. When the skeletal point recognition is incorrect, it is easy to cause false detection. Yin Jianqin et al. proposed an unconstrained repetitive motion counting method based on depth features (patent application number 201910874723.7). (a) This method uses optical flow maps to obtain the temporal features of the motion, which can only obtain the temporal information between the current frame and the previous frame. The lack of temporal information affects the prediction performance. (b) This method extracts spatial features and temporal features independently, ignoring the correlation between the two. (c) This method uses traditional methods to process the obtained depth features after dimensionality reduction. Compared with the method based entirely on deep learning, it has poor robustness. (d) In addition, this method does not evaluate the completeness of repetitive motions. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a two-stage repetitive motion counting and motion completeness evaluation method based on spatiotemporal features. To obtain robust motion features, a convolutional neural network is used to acquire the spatial features of each video frame, a temporal network is used to acquire spatiotemporal features, a feedforward network is used to obtain frame prediction values, and the prediction values ​​are filtered for counting and motion completeness evaluation. Completeness is crucial information related to repetitive motions. Completeness describes the degree to which a repetitive motion is performed. In applications such as fitness training, completeness directly helps in evaluating the quality of re-performed movements and estimating training effects.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for counting and evaluating the completeness of two-stage repetitive actions based on spatiotemporal features is characterized by comprising: acquiring video frames from a video containing two-stage repetitive actions; extracting spatial features of each video frame using a spatial feature extraction network; extracting spatiotemporal features from the spatial features of the video frames using a temporal feature extraction network; obtaining a prediction value sequence corresponding to each video frame based on the spatiotemporal features using a prediction head; identifying the two-stage repetitive actions based on the repetition pattern of the prediction values ​​in the prediction value sequence and counting the two-stage actions; and taking the maximum prediction value in the prediction value sequence corresponding to the identified two-stage actions as the completeness of the identified two-stage actions.

[0008] A two-stage repetitive motion counting and motion integrity assessment method based on spatiotemporal characteristics includes the following steps:

[0009] Step 1: Construct a two-stage repetitive action dataset D = {(x1,(y1)} 1,cls ,y 1,reg ),...,(x m ,(y m,cls ,y m,reg ))};

[0010] Step 2: Extract spatial features x for each frame using a deep feature extraction network. spatial ;

[0011] Step 3: Obtain spatiotemporal features x through a temporal feature extraction network. spatial-temporal ;

[0012] Step 4: Obtain frame prediction values ​​through a feedforward network

[0013] Step 5: Filter the count and evaluate the completeness of the action based on the predicted value.

[0014] Preferably, in the above-mentioned two-stage repetitive action counting and action integrity evaluation method based on spatiotemporal characteristics, step one specifically includes the following:

[0015] S11: Mark the start and end frames of the first stage of the two-stage repetitive action; mark the start and end frames of the second stage action; mark the completeness of the current action.

[0016] S12: Generate category labels using manually labeled tags. i,cls The action frames in the first stage are labeled with a classification label of 1; the action frames in the second stage are labeled with a classification label of 2; and the remaining frames are labeled with a classification label of 0.

[0017] S13: Generate regression labels y using manually labeled labels. i,reg The auxiliary label for the first stage action start frame is marked as 0, and the auxiliary label for the first stage end frame is marked as 1. The auxiliary labels for the remaining frames of the first stage are completed using linear interpolation. The auxiliary label for the second stage action start frame is marked as 1, and the auxiliary label for the second stage end frame is marked as 0. The auxiliary labels for the remaining frames of the second stage are completed using linear interpolation. The auxiliary labels for the transition frames from the first stage end frame to the second stage start frame are all marked as 1. The regression labels for the final trained model are obtained by multiplying the first stage action frames, the transition frames between the first and second stages, and the second stage action frames with the completeness of the action. The regression labels for non-action frames are all marked as 0.

[0018] Preferably, in the above-mentioned two-stage repetitive action counting and action integrity evaluation method based on spatiotemporal characteristics, steps two and three are specifically as follows:

[0019] The i-th frame RGB image in the frame sequence Spatial features x are extracted using a spatial feature extraction network based on a convolutional neural network. i,spatial Based on the characteristics of the spatial feature extraction network, an appropriate method is selected to extract spatial features. Transform into spatial feature vectors For real-time detection, the spatial feature vectors of the current i-th frame and the frames before and after t are used. Input the temporal feature extraction network to obtain spatiotemporal features For offline detection, the spatial feature vectors of all frames of the video are... Input the temporal feature extraction network to obtain spatiotemporal features

[0020] Preferably, in the above-mentioned two-stage repetitive action counting and action integrity evaluation method based on spatiotemporal characteristics, steps four and five are specifically as follows:

[0021] The regression prediction values ​​for each frame are obtained through the prediction head. The predicted values ​​are then filtered, and the peak value after filtering is used for two-stage repeated action counting; finally, the maximum value of the regression predicted value of the response frame sequence is taken. As a measure of the completeness of the action.

[0022] As can be seen from the above technical solution, compared with the prior art, this invention discloses a two-stage repetitive action counting and action completeness evaluation method based on spatiotemporal features. This method uses a concatenated spatial feature extraction network, temporal feature extraction network, and feedforward neural network to predict the motion stage and action completeness through a classification head and a regression head, enabling end-to-end training. This method only requires simple annotation information and can complete the two-stage repetitive action counting and completeness evaluation tasks through transfer learning. Compared with methods based on human skeleton point recognition, this invention removes the restriction that the entire human body must appear completely in the image; compared with methods based on mathematical techniques, this invention has better generalization performance, eliminating the need for researchers to mine the specific mathematical patterns in the features of each action; compared with methods that independently acquire the spatial and temporal features of an image, this invention correlates spatial and temporal features, with the temporal features having a larger temporal receptive field. This invention can greatly reduce the design cost for researchers and achieve considerable detection results. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the dataset annotation method of the present invention;

[0025] Figure 2 This is a model structure diagram of the method design of the present invention;

[0026] Figure 3 This is a flowchart of the method of the present invention;

[0027] Figure 4 This is a schematic diagram of the prediction results of the present invention. Detailed Implementation

[0028] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0029] The embodiments of the present invention will be described more clearly below with reference to the accompanying drawings.

[0030] See Figure 1Example: A two-stage repetitive action counting and action integrity assessment method based on spatiotemporal features, where the repetitive action dataset D = {(x1,(y1)} 1,cls ,y 1,reg ),...,(x m ,(y m,cls ,y m,reg The method of making ))}, where x i Represents a frame or image in a video, y i,cls Represents category tags, y i,reg Representing regression labels. According to embodiments of this application, the identified repetitive action includes two phases (referred to as the first phase and the second phase, respectively). For example, in a sit-up exercise, the first phase includes the process of rising from a lying position to a sitting position, while the second phase includes the process of returning from a sitting position to a lying position. The repetitive action can be performed by a human, other biological entity, or machine. Videos depicting the repetitive action are annotated. The repetitive action phases described in the video frames are annotated. The start and end frames of the first phase of the two-stage repetitive action are annotated; the start and end frames of the second phase are annotated; the completeness of each frame of the current action is annotated; and a classification label y is generated using manually annotated labels. cls ∈{0,1,2} and regression label y reg ∈[0,1].

[0031] Category tag y cls The specific generation process is as follows:

[0032] a) The classification label for the first-stage action frames is marked as 1;

[0033] b) The classification label for the second-stage action frames is marked as 2;

[0034] c) The classification labels for the remaining frames are 0.

[0035] Regression label y reg The specific generation process is as follows:

[0036] a) The auxiliary label for the starting frame of the first-stage action is set to 0, and the auxiliary label for the ending frame of the first-stage action is set to 1. The auxiliary labeling for the remaining frames of the first stage is completed using linear interpolation; see [link to documentation]. Figure 1 For each frame between the start and end frames of the first phase of the action, its auxiliary labels (see also...) Figure 1 The first row of the label processing table is set to a value between 0 and 1 using interpolation.

[0037] b) Label the starting frame of the second-stage action with an auxiliary label of 1, and the ending frame of the second-stage action with an auxiliary label of 0. The remaining frames of the second-stage action are labeled using linear interpolation (see also...). Figure 1 (The first row of the label processing table);

[0038] c) Mark all auxiliary labels of the transition frames between the first-stage termination frame and the second-stage start frame as 1;

[0039] d) Multiply the completeness of each frame of the first-stage action, the transition frame between the first and second-stage actions, and each frame of the second-stage action by the labeled completeness of each frame of the repeated action to obtain the regression label y of the final trained model. reg (See also) Figure 1 (The second row of the label processing table);

[0040] e) Mark the regression labels of all frames that do not belong to a single repeated action as 0.

[0041] It should be stated that, although in Figure 1 In the example, the annotation method used linear interpolation and manual completeness annotation. However, by using the deep learning method based on the embodiments of this application, the predicted values ​​of each frame that weigh the influence factors of multiple labels and objectively describe the repetitive actions can be learned from the linear interpolation data and subjective completeness annotation data.

[0042] See Figure 2 Figure 2 illustrates the structure of a neural network model used in a two-stage repetitive action counting and action integrity assessment method based on spatiotemporal features. The input to the neural network is a sequence of frames from a video. In Figure 2, each rectangle in the "Frame Sequence Data" represents a frame extracted from the video. The RGB image of the i-th frame in the frame sequence is shown. (Where H is the height of the frame image, and W is the width of the frame image,) (Representing the number field) extracts spatial feature vectors through a spatial feature extraction network. Where C s′ Represents the length of the spatial eigenvectors. Represents from image frame x i The extracted spatial feature vectors are then fed into a temporal feature extraction network to obtain spatiotemporal features. in It corresponds The spatiotemporal features are then processed by the prediction head (including the feedforward network, classification head, and regression head) to obtain classification predictions. Compared with regression predicted values

[0043] Optionally, the spatial feature extraction network can be selected based on the characteristics of the video frames being processed and the recognition task. For example,

[0044] a) For general two-stage repetitive fitness activities (such as squats, which can be fully described by the 17 human keypoints used in the COCO dataset (Lin TY, Maire M, Belongie S, et al. Microsoft coco: Common objects in context[C] / / Eur opean conference on computer vision. Springer, Cham, 2014:740-755.), HRNet (Sun K, Xiao B, Liu D, et al. Deep high-resolution representation learning for human pose estimation[C] / / Proceedingsof the IEEE / CVF conference on computer vision and pattern recognition. 2019:5693-5703.) is used as the spatial feature extraction network. Spatial features were obtained using HRNet. Here, 17 represents the 17 keypoints of the human body output by HRNet, 2 represents the coordinates of each keypoint, and 1 represents the prediction confidence of each keypoint. It should be noted that the method provided in Chinese patent application number 20211 0239387.6 does not use prediction confidence. This method uses deep learning, and the model can learn the impact of location confidence on the output results.

[0045] b) For more generalized two-stage repetitive activities (such as tides rising and falling, chopping vegetables, etc., which cannot be fully described by the 17 human keypoints used in the COCO dataset (Lin TY, Maire M, Belongie S, et al. Microsoft coco: Common objects in context[C] / / Eur opean conference on computer vision. Springer, Cham, 2014:740-755.), ResNet (He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-778.) and VGG (Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J].arXiv preprint) are used. Networks such as arXiv:1409.1556,2014. are used as spatial feature extraction networks. An example is shown below: Using an input image of size 224×224... The output space characteristics of V GG are spatial features x i,spatial Transform into spatial feature vectors Where C s′ =512×7×7.

[0046] Furthermore, the temporal feature extraction network is selected based on the task characteristics, and the preferred implementation method is as follows:

[0047] a) For scenarios involving real-time detection of video streams, a sliding window approach is used, where multiple spatial feature vectors within the sliding window are used as input to a temporal feature extraction network. The preferred network architecture is a bidirectional LSTM (Cu i Z, Ke R, Pu Z, et al. Deep bidirectional and unidirectional LSTM recurrent neural network for network-wide traffic speed prediction[J]. arXiv preprint arXiv:1801.02143,2018.) or a network with temporal feature extraction capabilities such as a Self-Attention Transformer (Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems,2017,30.). For the i-th frame, the spatial feature vectors of the current i-th frame and the preceding and following t frames are used as input to the temporal feature extraction network. Input the temporal feature extraction network to obtain spatiotemporal features

[0048] b) For offline detection, the spatial feature vectors of all frames of the video are used as the input to the temporal feature extraction network. The preferred network architecture is Local Attention. S, Zamani H, Mitra B, et al. Local self-attention over long text for efficient document retrieval[C] / / Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.2020:2021-2024.) Networks that can reduce computational cost and have the ability to extract sequential features can extract the spatial feature vectors of all frames of a video. Input the temporal feature extraction network to obtain spatiotemporal features

[0049] Furthermore, the prediction head comprises three parts: a feedforward network, a classification head, and a regression head, with spatiotemporal features. Auxiliary classification prediction values ​​are obtained through the prediction head. Compared with regression predicted values

[0050] a) The feedforward network has K layers (preferably K=2). For ease of description, let the spatiotemporal feature be denoted as x. i,spatial-temporal For x (0) The superscript 0 represents the input to the feedforward network. The specific operation of the feedforward network is x. (i) =ReLU(W (i) x (i-1) +b (i) for i = 1, ..., K, where W (i) Let b be the weight of the i-th fully connected layer. (i) Here are the parameters of the i-th fully connected layer, and ReLU() is the linear rectified function.

[0051] b) The classification head includes a Softmax operation, which implements... The specific operation is as follows: The classification head model assists in training but does not participate in the model's prediction.

[0052] c) The regression header includes a Sigmoid operation, which implements... The specific operation is as follows: The counting and completeness assessment of two-stage repeated actions are performed based on regression predictions. For example, the occurrence of one two-stage repeated action is based on the 0→1 and 1→0 cycles formed in the regression prediction sequence, where the boundaries of the regression prediction sequence may be numbers between 0 and 1, used to assess the completeness of the two-stage repeated action. Optionally, the cycles formed by the regression prediction sequence representing one two-stage repeated action may be incomplete (e.g., due to reasons such as action completeness or prediction accuracy). By filtering the regression prediction sequence and identifying the repetition patterns representing each cycle from the filtered results (see also...),... Figure 4 This method identifies two-stage repetitive actions. Optionally, the peak value in the repetition pattern is used as the basis for counting the two-stage repetitive actions, and this peak value is also used as the completeness of the corresponding two-stage repetitive action. Still, optionally, the difference between the peaks and troughs of the repetition pattern (the difference between the maximum and minimum values) is used as the completeness. Optionally, the obtained completeness is normalized.

[0053] Furthermore, the training strategy for the neural network model according to the embodiments of this application is as follows:

[0054] a) Calculate the loss using the output of the prediction head and the ground truth labels, then perform gradient backpropagation and parameter update iterations to complete network training. For the classification head, calculate the loss using the classification prediction value output by the classification head and the classification values ​​labeled on the training samples. For the regression head, calculate the loss using the regression prediction value output by the regression head and the regression values ​​labeled on the training samples. The loss L for the classification head... clsWhen there is an imbalance in the ratio of positive to negative samples, the Focal Loss function is used. The loss function for the classification head; the loss L for the regression head. reg Using the MSE function The loss function is used as the regression head; the total model loss is controlled by the hyperparameter β, L=(1-β)L cls +βL reg .

[0055] b) For model parameter initialization, transfer learning is used for spatial feature extraction network and temporal feature extraction network, and the parameters of other well-performing models are used for initialization. The parameters of the prediction head are randomly initialized.

[0056] c) Regarding the model training strategy, since the features of the spatial feature extraction network are relatively low-level in this method, for networks that use human keypoints as spatial feature vectors (e.g., HRNet), the parameters of the spatial feature extraction network are fixed and not updated during neural network training. For more generalized two-stage repetitive actions, networks such as ResNet and VGG are used as spatial feature extraction networks. Compared to temporal feature extraction networks, a smaller learning rate is used for the spatial feature extraction network during training. Since the parameters of the prediction head are randomly initialized, a larger learning rate is used for the prediction head compared to the spatial and temporal feature extraction networks.

[0057] See Figure 3 This example illustrates a flowchart of a two-stage repetitive action counting and action integrity assessment method based on spatiotemporal features. The classification head plays a role in training the neural network. However, the classification head does not participate in the prediction process when predicting video frames.

[0058] For the video to be detected, extract the images of each frame from the video. The RGB image of the i-th frame in the frame sequence. Spatial features x are extracted using a spatial feature extraction network based on a convolutional neural network. i,spatial Based on the characteristics of the spatial feature extraction network, an appropriate method is selected to extract spatial features. Transform into spatial feature vectors For example, multiple spatial features extracted from an image frame, along with their confidence scores, can be concatenated or flattened to form a spatial feature vector.

[0059] Then, spatiotemporal features are obtained through a temporal feature extraction network. Spatiotemporal features x i,spatial-temporal The regression prediction values ​​for each frame are obtained through the prediction head (which includes the feedforward network and the regression head). The regression prediction sequence is filtered, and the peak value after filtering is used to count the two-stage repeated actions; then, the maximum value of the regression prediction value of each frame sequence in this two-stage repeated action is taken. As a measure of the completeness of the action.

[0060] See Figure 4 This example illustrates a regression prediction value sequence in a two-stage repetitive action counting and action integrity assessment method based on spatiotemporal features. The regression prediction value is output by the regression header for each frame. The resulting sequence of regression predictions is filtered (the classification head only plays an auxiliary role in model training and does not participate in the final counting and completeness evaluation) to obtain a filtered <frame, regression prediction> sequence. The two-stage repeated action count uses the peak values ​​of the filtered sequence for statistical analysis (e.g., each peak triggers a count); the maximum value of the regression prediction value corresponding to the two-stage repeated action sequence at that count is then taken. As a measure of the completeness of the action. For comparison, such as... Figure 4 As shown, our method produces smoother predictions than patent application number 201910874723.7.

[0061] The UCFRep dataset (Zhang H, Xu X, Han G, et al. Context-aware and scale-insensitive temporal repetition counting[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2020:670-678.) contains 526 annotated videos of repetitive actions. This dataset was constructed from the action recognition dataset UCF101 (Soomro K, Zamir AR, Shah M. UCF101: A dataset of 101 human actions classes from videos in the wild[J]. arXiv preprint arXiv:1212.0402,2012.). The performance of the embodiments of this application was evaluated using a subset of the UCFRep dataset, specifically the jumping jacks. A conventional and more easily understood evaluation criterion was used as the metric for this task, where the percentage of the absolute difference between the counts between the true value G and the predicted value R was used as the evaluation result. For N videos, calculate the mean absolute error (MAE) ± standard deviation σ, where The experimental results are compared below:

[0062] The method used MAE Daochang Liu's 0.286±0.680 Huaidong Zhang's 0.147±0.243 Yunhua Zhang's 0.143±0.800 This invention 0.097±0.004

[0063] In the above table, for Daochang Liu's method, see Liu D, Jiang T, Wang Y.Completenessmodeling and context separation for weakly supervised temporal actionlocalization[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2019:1298-1307. For Huaidong Zhang's method, see Zhang H, Xu X, Han G, et al.Context-aware and scale-insensitive temporal repetition counting[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2020:670-678.Yunhua Zhang's method can be found in Zhang Y, Shao L, Snoek C GM.Repetitive activity counting by sight and sound[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:14070-14079.

[0064] Since the two-stage repetitive actions in the UCFRep dataset do not include annotation information on action completeness, additional dataset labels for action completeness are added to the two-stage repetitive actions in the UCFRep dataset, and mean absolute error is used. As an evaluation of the prediction results, among which For the truth value of completeness, The completeness prediction value is MAE = 5.02%, indicating that the completeness prediction result is relatively accurate.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made to the present invention should be included within the scope of protection of the present invention.

Claims

1. A two-stage repetitive action counting and action integrity evaluation method based on spatiotemporal characteristics, characterized in that, include: Extract video frames from a video containing two-stage repetitive motion; Spatial features of each video frame are extracted using a spatial feature extraction network; Spatiotemporal features are extracted from the spatial features of video frames using a temporal feature extraction network; The prediction head obtains the sequence of predicted values ​​corresponding to each video frame based on spatiotemporal features; Two-stage repetitive actions are identified based on the repetition pattern of predicted values ​​in the predicted value sequence, and the two-stage actions are counted. The maximum predicted value in the predicted value sequence corresponding to the identified two-stage actions is taken as the completeness of the identified two-stage actions. The method further includes: constructing a two-stage repetitive action dataset. ,in This represents a video frame in a video that contains a two-stage repetitive motion. Category labels not marked The regression labels are not specified. Among them, the construction of a two-stage repetitive action dataset include: In a video containing a two-stage repetitive action, the start and end frames of the first stage of the two-stage repetitive action are labeled; the start and end frames of the second stage are labeled; and the completeness of the current action shown in each video frame relative to the two-stage repetitive action to be identified is labeled. The video frames that recorded the first stage of the action were labeled as 1; the video frames that recorded the second stage of the action were labeled as 2; and the video frames that recorded neither the first stage nor the second stage of the action were labeled as 0. The auxiliary label for the starting frame of the continuous video frames that recorded the first stage of the action was marked as 0, and the auxiliary label for the ending frame of the continuous video frames that recorded the first stage of the action was marked as 1. The remaining frames of the continuous video frames that recorded the first stage of the action were generated by interpolation with values ​​between 0 and 1 as auxiliary labels. The auxiliary label of the starting frame of the continuous video frames that recorded the second stage action was marked as 1, the auxiliary label of the ending frame of the continuous video frames that recorded the second stage action was marked as 0, and the auxiliary labels of the remaining frames of the continuous video frames that recorded the second stage action were interpolated to values ​​between 1 and 0. The auxiliary labels for the transition frames between the end frame of the continuous video frames recording the first stage of action and the start frame of the continuous video frames recording the second stage of action are all marked as 1. The auxiliary label of each video frame in a video containing two-stage repetitive actions is multiplied by the completeness of the annotation to obtain the regression label of the video frame. The regression labels of video frames that do not contain two-stage repetitive actions are marked as 0; And wherein, the identification of two-stage repetitive actions based on the repetition pattern of predicted values ​​in the predicted value sequence is based on the pattern formed in the predicted value sequence. as well as The loop serves as the basis for the occurrence of a two-stage repeated action.

2. The method according to claim 1, wherein Using a spatial feature extraction network, from the first video frame... Spatial features extracted from video frames are ; The method further includes, for example, placing the first Spatial features of video frames R Convert to spatial feature vector R ; The spatial feature vectors of multiple consecutive video frames are provided to the temporal feature extraction network, which outputs the spatiotemporal features of each of the multiple consecutive video frames, wherein the first... The spatiotemporal characteristics of a video frame are R .

3. The method according to claim 2, wherein To perform real-time extraction of spatiotemporal features, the consecutive multi-frame video frames provided to the temporal feature extraction network include those from the first... Frame to the Frames Frame video frame, If it is a positive integer, then the first... Frame to the Frames The spatial feature vectors of each video frame R Input a temporal feature extraction network to obtain the spatiotemporal features of each of the consecutive video frames; and / or For offline detection, the consecutive multi-frame video frames provided to the temporal feature extraction network include the video from the first... Frame to the All video frames of the frame. It is a positive integer, and , will the Frame to the The spatial feature vectors of each video frame. R The input temporal feature extraction network obtains the spatiotemporal features of each of the video frames. R .

4. The method according to claim 3, wherein The step of identifying two-stage repetitive actions based on the repetition pattern of predicted values ​​in the predicted value sequence includes: The spatiotemporal features corresponding to each frame of the consecutive multi-frame video are provided to the prediction head, wherein the first... The spatiotemporal characteristics of a video frame are R The prediction head outputs the predicted values ​​corresponding to the spatiotemporal features of each frame of the consecutive multi-frame video, wherein the first frame... The predicted value corresponding to the spatiotemporal features of a video frame is The predicted values ​​corresponding to the spatiotemporal features of each frame of the consecutive multi-frame video output by the prediction head are filtered, and the two-stage repetitive action is identified based on the peaks or troughs of the curve formed by the predicted values ​​corresponding to the spatiotemporal features of each frame of the consecutive multi-frame video output by the prediction head after filtering.

5. The method according to any one of claims 1-4, further comprising: The prediction head includes a feedforward network, a classification head, and a regression head; it uses a constructed two-stage repetitive action dataset. The feedforward network, the regression head, and the classification head are trained, wherein a two-stage repetitive action dataset is constructed during the training process. The spatial feature extraction network is provided, and a loss is calculated based on the difference between the predicted value output by the regression head and the regression label, and the difference between the predicted value output by the classification head and the classification label.

6. The method according to claim 5, wherein The loss of the regression head is calculated based on the difference between the predicted value output by the regression head and the regression label. ; The loss of the classification head is calculated based on the difference between the classification prediction value output by the classification head and the classification label. ; according to Calculate the loss of the prediction head, where It's a hyperparameter.

7. The method of claim 6, wherein Using the constructed two-stage repetitive action dataset During the training of the feedforward network, the regression head, and the classification head, the parameters of the spatial feature extraction network and / or the temporal feature extraction network are not updated.

8. A two-stage repetitive motion counting and motion integrity evaluation method based on spatiotemporal characteristics, characterized in that, include: Step 1: Construct a two-stage repetitive action dataset ,in This represents a video frame in a video that contains a two-stage repetitive motion. Category labels not marked The regression labels are not specified. Among them, the construction of a two-stage repetitive action dataset include: Specifically, in video frames containing two-stage repetitive actions, the start and end frames of the first stage of the two-stage repetitive action are marked; the start and end frames of the second stage are marked; and the completeness of the current action shown in each video frame relative to the two-stage repetitive action to be identified is marked. The video frames that recorded the first stage of the action were labeled as 1; the video frames that recorded the second stage of the action were labeled as 2; and the video frames that recorded neither the first stage nor the second stage of the action were labeled as 0. The auxiliary label for the starting frame of the continuous video frames that recorded the first stage of the action was marked as 0, and the auxiliary label for the ending frame of the continuous video frames that recorded the first stage of the action was marked as 1. The remaining frames of the continuous video frames that recorded the first stage of the action were generated by interpolation with values ​​between 0 and 1 as auxiliary labels. The auxiliary label of the starting frame of the continuous video frames that recorded the second stage action was marked as 1, the auxiliary label of the ending frame of the continuous video frames that recorded the second stage action was marked as 0, and the auxiliary labels of the remaining frames of the continuous video frames that recorded the second stage action were interpolated to values ​​between 1 and 0. The auxiliary labels for the transition frames between the end frame of the continuous video frames recording the first stage of action and the start frame of the continuous video frames recording the second stage of action are all marked as 1. The auxiliary label of each video frame in a video containing two-stage repetitive actions is multiplied by the completeness of the annotation to obtain the regression label of the video frame. The regression labels of video frames that do not contain two-stage repetitive actions are marked as 0; Step 2: Extract spatial features for each frame using a deep feature extraction network. ; Step 3: Obtain spatiotemporal features through a temporal feature extraction network. ; Step 4: Obtain frame prediction values ​​through a feedforward network ; Step 5: Filter count based on predicted values ​​and action integrity assessment, wherein filtering count based on predicted values ​​includes filtering the predicted value sequence and identifying the repetition patterns of each cycle formed by the predicted value sequence from the filtered results to identify two-stage repetitive actions.

9. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1-8.

Images