Repetitive motion counting method and framework based on deep learning and temporal decision method

Abstract

The application discloses a motion counting framework based on deep learning and time sequence determination method, which comprises a model detection module and a time sequence determination module, and the time sequence determination module determines based on an input video segment containing specific repetitive motion. The motion counting framework based on deep learning and time sequence determination method and the construction method solve the problem that the start and end information cannot be effectively predicted in video understanding, improve the prediction performance and efficiency, and solve the problem that the counting value has an upper limit because the distribution of sampling frames is too sparse relative to the video length in the motion counting prediction of a long-time video by the previous repetitive motion counting model, thereby improving the accuracy performance of the repetitive motion counting model.

Description

Technical Field

[0001] This invention belongs to the field of video detection technology, specifically relating to a method and framework for counting repetitive actions based on deep learning and temporal determination. Background Technology

[0002] In real-world industrial production environments, there are numerous scenarios involving long-duration, repetitive operations. These repetitive actions often require manual counting, impacting production efficiency. Currently, deep learning models are primarily used to process video data to address the need for counting repetitive actions in industrial production scenarios. These deep learning models are mainly video understanding models. Existing video understanding models include CRNN and LSTM based on 2D convolutional neural networks and recurrent neural networks, C3D and I3D based on 3D convolutional neural networks, and TimeSformer and 3D-swinTransformer based on self-attention transformers. These models can perform video classification, spatiotemporal action recognition, and action counting.

[0003] Existing technologies, such as the Chinese invention patent application with publication number CN 114898241 A, disclose a video repetitive action counting system based on computer vision. This invention application adopts a convolutional neural network framework based on temporal autocorrelation, including a video feature extractor, a temporal autocorrelation layer, an encoder, and an action predictor. Although it has achieved the detection, localization, and counting of repetitive actions in videos to a certain extent, it has the following shortcomings: (1) It cannot identify the start and end information of the process flow of specific repetitive actions in actual industrial production. It will also detect video segments that do not perform specific repetitive actions, resulting in false detections and reducing the predictive performance and operating efficiency of the model; (2) In the sampling process of multi-scale temporal frames, the distribution of sampled frames will be too sparse in long-term videos, resulting in an upper limit to the repetitive action count value, which is affected by the number of sampled frames. Summary of the Invention

[0004] The purpose of this invention is to provide a method and framework for counting repetitive actions based on deep learning and temporal determination.

[0005] This repetitive action counting method and framework based on deep learning and temporal decision-making solves the problem of ineffective prediction of repetitive actions in video understanding, improving prediction performance and efficiency. At the same time, it also solves the problem that previous repetitive action counting models have an upper limit on the count value when predicting action counts in long-term videos because the distribution of sampling frames is too sparse relative to the video length, thus improving the accuracy performance of the repetitive action counting model.

[0006] To achieve the above-mentioned objectives, the technical solution of this invention is as follows:

[0007] A method for counting repeated actions based on deep learning and temporal determination includes a temporal determination module, a trained object detection model, and a repeated action counting model. The method further includes:

[0008] A. The object detection model determines whether action markers exist in each video frame of the video to be detected;

[0009] B. The timing determination module determines whether to perform repeated action behavior counting on the video segment of the video to be detected based on the judgment result of the target detection model;

[0010] C. The repetitive action counting model counts the number of actions in video segments that are determined to be performing repetitive actions.

[0011] D. Accumulate the number of action counts for all video segments corresponding to repetitive actions in the video to be detected based on the repetitive action counting model. The accumulated result is the total number of repetitive actions in the video to be detected.

[0012] The order of A, B, C, and D above is not strict.

[0013] Currently, deep learning-based video understanding networks have achieved good performance in video classification and semantic understanding, but they still suffer from poor performance in repeated action counting. While these networks can learn the overall information of actions within a video, their ability to understand the semantics of actions in the temporal dimension is still immature. They cannot effectively identify the start and end information of actions. When the start and end behaviors of an action are highly similar, existing models cannot make high-performance distinctions between such similar actions, leading to a large number of false positives and false negatives during the counting process, thus reducing the overall performance of video understanding models on action counting tasks.

[0014] This invention introduces a relatively mature task in image processing, object detection, into video understanding based on a temporal determination method. The advantage of object detection models is that they can achieve stable performance and fast speed for data in specific scenes. Based on the relatively excellent detection performance of object detection models, the existence of action markers is determined, and then the temporal determination module is used to determine repetitive actions. Therefore, repetitive actions can be identified from complete video segments. At the same time, this invention separates the video segments corresponding to the repetitive actions from the complete video segments. The repetitive action counting model predicts the action count for the video segment where the repetitive action begins. Therefore, the problem of the distribution of sampling frames being too sparse relative to the video length is avoided, and the action count value of the action counting framework is not limited by the video understanding model.

[0015] In the above-mentioned repetitive action counting method based on deep learning and temporal determination, each repetitive action counting process includes determining whether it is a start state based on temporal determination.

[0016] In B, the determination of repetitive actions includes determining whether the repetitive action is in the initial state;

[0017] In C, the repeat action counting model starts counting repeat actions on the corresponding video segment based on the start state of the repeat action.

[0018] By using a timing determination module and action markers, the starting point of an action in a complete video segment can be accurately and effectively described using a supervised labeling system, thus effectively distinguishing repetitive actions in the video segment.

[0019] In the above-mentioned repetitive action counting method based on deep learning and temporal determination, in step A, the object detection model determines the presence or absence of action markers in the following way:

[0020] ojb t =length(f det (Img t ))

[0021] Img t f represents the frame corresponding to the t-th frame of the video to be detected; det This represents the trained object detection model;

[0022] length indicates quantity statistics; ojb z Indicates the image Img t Based on target detection model f det The predicted target quantity;

[0023] If ojb t If the value is greater than 0, it means that the object detection model predicts that there are action markers in the image; otherwise, it means that the object detection predicts that there are no action markers in the image.

[0024] This invention utilizes a temporal decision method to improve the performance of the model's prediction results for counting repetitive actions. The target detection model predicts whether the action marker appears or not, preparing for subsequent temporal decision modules, etc.

[0025] In the above-mentioned repetitive action counting method based on deep learning and temporal determination, the temporal determination module determines the start state of the repetitive action behavior as follows:

[0026] When ojb t >0, take Img t Add to set C objIn the set C obj Let C represent the set of images predicted to contain motion markers. obj The number of elements in the data is compared with the cumulative threshold K1 to determine the value.

[0027] When length(C) obj When K ≥ K1, the video to be detected is determined to be in set C. obj An action marker appears in a series of K1 frames, thus indicating the start state.

[0028] That is, when set C obj The video segment is determined to have started repetitive actions when it contains at least K1 elements. In practical applications, the cumulative threshold K1 is set as needed. The timing determination module judges the video segment frame by frame. When the cumulative frame count reaches the threshold, it is the video segment corresponding to the start of the repetitive action. Thus, the model detection module can effectively identify the start of the action. Based on the repetitive action counting model, the set C is calculated for each consecutive K frames. obj Action counting prediction is performed on frames within the video, thus effectively avoiding the limitation of video length imposed by repetitive action counting models.

[0029] In the above-mentioned repetitive action counting method based on deep learning and temporal determination, the temporal determination module determines the end state of the repetitive action behavior as follows:

[0030] When ojb t =0, Img t Add to set C zero In the set C zero Let C represent the set of images predicted to have no motion markers. zero The number of elements in the data is compared with the cumulative threshold K2 to determine the value.

[0031] When length(C) zero When K ≥ K2, the video to be detected is determined to be in set C. zero The video segment corresponding to consecutive K2 frames within the frame does not contain any action markers, thus it is determined to be a non-starting state.

[0032] In this invention, the cumulative thresholds K1 and K2 can be the same or different, and can be adjusted according to the actual application scenario. This invention utilizes a temporal determination method based on the prediction results of the target detection model. By determining the start and non-start states of repetitive actions based on the detection of action markers, it solves to some extent the problem of ineffective prediction of the start information of a process in video understanding. Therefore, the model detection module can effectively identify the start bundle information of actions, thereby improving prediction performance and efficiency.

[0033] In the aforementioned repetitive action counting method based on deep learning and temporal determination, in step D, after each repetitive action count is completed, the current set C is reset to zero. obj The cumulative frame count will be used to advance to the next round of prediction;

[0034] Throughout the entire initial state process, whenever length(C) obj For each state ≥ K1, a round of repeated action counting in C above will be performed until the state is changed to a non-starting state.

[0035] This invention effectively shortens the duration of video segments corresponding to repetitive actions by clearing the accumulated frame count after each round of action counting prediction. The repetitive action counting model predicts actions on specific video segments, thus avoiding the problem of an upper limit on the count value caused by the distribution of sampled frames being too sparse relative to the video length, thereby improving the accuracy performance of the repetitive action counting model.

[0036] In the aforementioned repetitive action counting method based on deep learning and temporal determination, step B further includes confirmation of the repetitive action behavior state.

[0037] When legth(C) obj When K ≥ K1, set the start state to Yes.

[0038] When length(C) zero When )≥K2, the start state is set to no, i.e., non-start state.

[0039] To ensure the smooth operation of this repetitive action counting method framework, set C obj or set C zero When the cumulative number of frames is greater than or equal to the corresponding threshold, it does not directly proceed to the next step. It is necessary to first confirm the status of repetitive action behavior, giving the framework a transition process. At the same time, confirming the status of repetitive action behavior can also reduce the occurrence of errors during the framework operation.

[0040] In the aforementioned repetitive action counting method based on deep learning and temporal determination, the temporal determination module further includes a false detection prevention patch.

[0041] When ojb t >0, take Img t Add to set C obj In the middle, the patch to prevent false positives is to clear set C. zero The cumulative number of frames in

[0042] When ojb t =0, first confirm the status of the repetitive action behavior.

[0043] After confirming that the repetitive action behavior is in the start state, Imgt Simultaneously add to set C obj and set C zero middle,

[0044] After confirming that the repetitive action is not in the initial state, administer Img. t Add to set C zero In the middle, the set C is cleared. obj The cumulative number of frames in the data.

[0045] False detection prevention patches are processing methods used to prevent false detections from causing the accumulation of discontinuous frames or the omission of consecutive frames.

[0046] When the object detection model detects action markers, it clears set C. zero The cumulative frame count in the data can prevent the set C from being manipulated. zero False detection caused set C obj Non-continuous frame count accumulation; when the target detection model fails to detect action markers, set C is reset to zero. obj The cumulative frame count in the data can prevent the set C from being manipulated. obj False detection caused set C zero The accumulation of non-continuous frames further improves the accuracy of predicting the start and non-start states of repetitive actions.

[0047] Furthermore, when ojb t =0, first confirm the status of the repetitive action behavior.

[0048] After confirming that the repetitive action behavior is in the start state, Img t Simultaneously add to set C obj and set C zero middle,

[0049] After confirming that the repetitive action is not in the initial state, administer Img. t Add to set C zero In the middle, the set C is cleared. obj The cumulative number of frames in the data.

[0050] When the target detection model fails to detect action markers and the repeated action behavior is in the start state, there is a high probability of missing consecutive frames. Therefore, when this happens, Img... t Simultaneously add to set C obj and set C zero Then, by separately judging the number of elements in the set and the threshold, this kind of false detection behavior can be corrected.

[0051] A repetitive action counting framework based on deep learning and temporal determination method includes a temporal determination module, as well as a trained object detection model and a repetitive action counting model, for executing the above method.

[0052] The target detection model described in this invention preferably uses YOLOv5, which has a relatively balanced performance in terms of speed and energy consumption. The repetitive action counting model described in this invention is preferably a repetitive action counting model based on 2D convolutional neural networks and transformers, namely transRac.

[0053] This invention proposes a framework for automatically counting repetitive actions in industrial production scenarios without human intervention. Based on a deep neural network-based object detection model, a repetitive action counting model, and a temporal decision-making method, this framework enables action counting for specific processes in industrial settings, overcoming the performance limitations of existing video understanding and repetitive action counting models in real-world industrial environments. The temporal decision-making method based on model detection results has strong scalability, not only for repetitive action counting tasks but also for applications such as human behavior recognition and anomaly detection in videos.

[0054] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0055] (1) Based on the temporal determination method, this invention introduces the relatively mature task of object detection in the field of image processing into video understanding. Based on the relatively excellent detection performance of the object detection model, it determines whether action markers exist. Then, the temporal determination module is used to determine repetitive action behavior. The start and end features of repetitive action behavior can be accurately and effectively described in a supervised label form. Therefore, the start and end of repetitive action behavior in video segments can be distinguished. At the same time, this invention separates the video segments corresponding to the repetitive action behavior from the complete video segments. The repetitive action counting model performs action count prediction on the video segments where the repetitive action behavior begins. Therefore, there will be no problem that the distribution of sampling frames will be too sparse relative to the video length. The action count value of the action counting framework will not be limited by the video understanding model.

[0056] (2) By using the timing determination module and action markers, the starting time of the action in the complete video segment and the starting characteristics of the repetitive action behavior can be accurately and effectively described in a supervised label form, and the repetitive action behavior in the video segment can be effectively distinguished.

[0057] (3) This invention uses the time-series determination method to improve the performance of the model prediction results for counting repeated actions. The target detection model predicts whether the action marker appears or not, in preparation for subsequent time-series determination modules, etc.

[0058] (4) The timing determination module of the present invention judges the video segment frame by frame. When the cumulative frame count element reaches the threshold, it is the video segment corresponding to the start state of the repeated action behavior. Thus, the model detection module can effectively identify the start of the action. Based on the repeated action counting model, the action count prediction is performed on the frames in the set when K frames are accumulated, thereby effectively avoiding the limitation of the repeated action counting model on the video length.

[0059] (5) This invention uses the temporal determination method based on the prediction results of the target detection model to determine the start state and non-start state of repetitive action behavior by determining whether or not the action marker is detected. This solves the problem of the inability to effectively predict the start information of the process in video understanding to a certain extent. As a result, the model detection module can effectively identify the start bundle information of the action, thereby improving the prediction performance and efficiency.

[0060] (6) By clearing the cumulative frame count after each round of action counting prediction, the present invention effectively shortens the duration of the video segment corresponding to the repetitive action behavior. The repetitive action counting model predicts the action of a specific video segment, so it does not have the problem of the count value having an upper limit due to the distribution of the sampled frames being too sparse relative to the video length, thereby improving the accuracy performance of the repetitive action counting model.

[0061] (7) In order to ensure the smooth operation of the repetitive action counting method framework, this invention sets C obj or set C zero When the cumulative number of frames is greater than or equal to the corresponding threshold, it does not directly proceed to the next step. It is necessary to first confirm the status of repetitive action behavior, giving the framework a transition process. At the same time, confirming the status of repetitive action behavior can also reduce the occurrence of errors during the framework operation.

[0062] (8) When the target detection model of this invention detects an action marker, the set C is cleared. zero The cumulative frame count in the data can prevent the set C from being manipulated. zero False detection caused set C obj Non-continuous frame count accumulation; when the target detection model fails to detect action markers, set C is reset to zero. obj The cumulative frame count in the data can prevent the set C from being manipulated. obj False detection caused set C zero The accumulation of non-continuous frames further improves the accuracy of predicting the start and non-start states of repetitive actions.

[0063] (9) When the target detection model fails to detect action markers and the repeated action behavior state is in the start state, there is a high probability of missing consecutive frames. Therefore, when this happens, Img t Simultaneously add to set C obj and set Czero Then, by separately judging the number of elements in the set and the threshold, this kind of false detection behavior can be corrected.

[0064] (10) This invention proposes a framework for automatically counting repetitive actions in industrial production scenarios without human intervention. Based on a deep neural network-based target detection model, a repetitive action counting model, and a temporal decision-making method, this framework enables action counting for specific processes in industrial settings, overcoming the performance limitations of existing video understanding and repetitive action counting models in real-world industrial production environments. The temporal decision-making method based on model detection results has strong scalability, not only for repetitive action counting tasks but also for tasks such as human behavior recognition in videos and abnormal feature detection in videos. Attached Figure Description

[0065] Figure 1 This is a flowchart of the action counting framework based on deep learning and temporal determination method of the present invention. Detailed Implementation

[0066] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] This embodiment provides a repetitive action counting method based on deep learning and temporal judgment. It uses YOLOv5 in the object detection model to predict whether action markers appear or not, and the video understanding model transRac to predict the action count for specific repetitive actions.

[0068] Before training the object detection model, this embodiment constructs a set of object detection data based on action markers from the video to be detected. The video to be detected is video data of a process flow containing specific repetitive actions in an actual industrial production scenario. The action marker indicates that the appearance of the marker in the video to be detected means that the object of the repetitive action to be counted has appeared, and preparations can begin to count the repetitive actions.

[0069] In object detection data, the dataset is labeled by annotating the coordinates of the top-left and bottom-right vertices of the bounding rectangle of an action marker, as well as the marker's category. The object detection model trained on this labeled data can predict the presence or absence of the action marker, preparing for subsequent modules such as temporal determination methods.

[0070] Simultaneously, before the training phase of the repetitive action counting model, specific repetitive actions to be counted are labeled based on the video to be detected.

[0071] First, a complete video segment to be detected (referred to as the complete video segment) is labeled with primary tags. The primary tag represents the frame number corresponding to the start and end time periods of all specific repetitive actions to be counted within the complete video segment. Based on the complete video segment and the primary tags, a certain number of video clips of varying lengths are randomly cropped from the complete video segment. Each video clip is labeled with a secondary tag, which represents the number of specific repetitive actions contained within the video clip (including actions with zero repetitions). These video clips and tags are used as training data for the repetitive action counting model. The repetitive action counting model trained based on this repetitive action data is used for action count prediction within the framework of this embodiment.

[0072] like Figure 1 As shown, this embodiment also provides a repetitive action counting framework based on deep learning and temporal determination method. Figure 1 The upper dashed box contains the timing determination module, and the lower dashed box contains the model detection module.

[0073] The process of building this framework or using this framework to count repetitive actions is as follows:

[0074] The input video frame is the current t frame. The target detection model obtains the detection result and determines its state based on the temporal determination module. The state has two types: start and non-start.

[0075] The image corresponding to the t-th frame of a video is defined as Img. t The target detection model obtained after training is defined as f. det , Img t Input to f det The formula is as follows:

[0076] ojb t =length(f det (Img t ))

[0077] Length represents the image size (in mg). t Based on target detection model f det The predicted target quantity, ojb t >0 Here, length represents the quantity statistics.

[0078] If ojb t If Img > 0, it means the object detection model predicts that there is a motion marker in the image, and Img will be... t Add to set C obj In the set C obj This represents the set of images predicted to contain motion markers. First, set C is cleared. zero The cumulative number of frames in set C, then... objThe number of elements in the array is determined by the following formula:

[0079] length(C obj )≥K1

[0080] This indicates that action markers have appeared in K1 consecutive frames in the temporal dimension, which means that the video to be detected belongs to set C. obj The corresponding video segment is the start state of repeat action counting, and K1 represents the cumulative threshold.

[0081] If the threshold is exceeded, the repetitive action behavior status is confirmed, the start status is set to yes, and C is set after the transition is complete. obj Using all frames as input, a repeated action counting model is used to predict the action count. The action count results are accumulated into the total action count of the entire video, and then the current set C is reset to zero. obj The cumulative number of frames in the sequence determines the next prediction round. Throughout the initial state, whenever length(C) is reached... obj For each state ≥ K1, the above process will be repeated once until it is converted to a non-starting state.

[0082] On the other hand, if ojb t =0 indicates that the target detection prediction result has no action markers, and the repeated action behavior state is confirmed first:

[0083] After confirming that the repetitive action behavior is in the start state, Img t Simultaneously add to set C obj and set C zero Then, the number of elements in the set and the threshold are judged separately.

[0084] After confirming that the repetitive action is not in the initial state, administer Img. t Add to set C zero In the middle, the set C is cleared. obj The cumulative number of frames in set C, then... zero The number of elements in the array is determined by the following formula:

[0085] length(C zero )≥K2

[0086] This means that, in the temporal dimension, it can be determined that there are no repeated actions to be counted, or that the repeated actions have ended, and the video to be detected is determined to be in set C. zero If no action marker appears in the video segment corresponding to consecutive K2 frames within the frame, it is determined to be a non-start state. Enter the repetitive action behavior confirmation state, set the start state to no, and clear the current set C. zero The cumulative number of frames in the prediction will be used to advance to the next round of prediction.

[0087] As mentioned above, the condition for determining the start state is that the cumulative number of frames with action markers detected by the target detection model is greater than K1. This filters out video segments corresponding to repetitive action behaviors. Then, based on the repetitive action counting model, the number of actions in the cumulative number of frames of the video segments corresponding to repetitive action behaviors is predicted. At the same time, the condition for determining the non-start state is that the cumulative number of frames without action markers detected is greater than K2. Therefore, video segments that do not perform repetitive action behaviors can be removed.

[0088] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for counting repetitive actions based on deep learning and temporal determination, characterized in that, It includes a timing determination module, as well as a trained target detection model and a repetitive action counting model. Each repetitive action counting process includes determining whether it is a start state based on the timing determination method. And the method includes: A. The object detection model determines whether action markers exist in each video frame of the video to be detected; In section A, the target detection model determines the presence or absence of action markers in the following way: ojb t =length(f det (Img t )) Img t f represents the frame corresponding to the t-th frame of the video to be detected; det This represents the trained object detection model; lergth represents the statistical count; ojb t Indicates the image Img t Based on target detection model f det The predicted target quantity; If ojb t If the value is greater than 0, it means that the object detection model predicts that there are action markers in the image; otherwise, it means that the object detection predicts that there are no action markers in the image. B. The timing determination module determines whether to perform a repetitive action count on the video segment of the video to be detected based on the judgment results of the target detection model; the determination based on the timing determination module includes the determination of whether it is the start state and the confirmation of the repetitive action state; The timing determination module determines whether a repetitive action is in a start or non-start state in the following ways: When ojb t >0, take Img t Add to set C obj In the middle, the anti-false detection patch is a zeroing set C. zero The cumulative number of frames in When ojb t =0, first confirm the repetitive action behavior status. After confirming that the repetitive action behavior status is in the start state, put Img t Simultaneously add to set C obj and set C zero In the process, after confirming that the repetitive action is not in the initial state, Img is... t Add to set C zero In the middle, the set C is cleared. obj The cumulative number of frames in the data; Set C obj Let set C represent the cumulative set of images predicted to contain motion markers. zero Let C represent the set of images predicted to have no motion markers. obj The number of elements in set C is compared with the cumulative threshold K1 to determine the set. zero The number of elements in the data is compared with the cumulative threshold K2; When length(C) obj When K ≥ K1, the video to be detected is determined to be in set C. obj An action marker appears in a consecutive K1 frames, thus determining it to be a start state and setting the start state to yes. When length(C) zero When K ≥ K2, the video to be detected is determined to be in set C. zero If no action marker appears in the video segment corresponding to consecutive K2 frames within the frame, it is determined to be a non-start state, and the start state is set to no. C. Based on the initial state of the repetitive action, the repetitive action counting model begins to count the number of repetitive actions in the corresponding video segment; D. Accumulate the number of action counts for all video segments corresponding to repetitive actions in the video to be detected based on the repetitive action counting model. The accumulated result is the total number of repetitive actions in the video to be detected.

2. The repetitive action counting method based on deep learning and temporal determination as described in claim 1, characterized in that, In step D, after each repetitive action count is completed, the current set C is reset to zero. obj The cumulative frame count will be used to advance to the next round of prediction; Throughout the entire initial state process, whenever length(C) obj For each state ≥ K1, a round of repeated action counting in C above will be performed until the state is changed to a non-starting state.

3. A repetitive action counting framework based on deep learning and temporal decision-making, characterized in that, It includes a timing determination module, as well as a trained target detection model and a repetitive action counting model, for performing the method described in any one of claims 1-2.

Images