Clustering-based temporal action positioning method, device and storage medium for cyclic action videos

Abstract

The present invention discloses a method, device and storage medium for locating cyclic action videos based on temporal modeling. The method firstly extracts video features through self-supervised training and domain adversarial transfer learning, and secondly utilizes a clustering algorithm for efficient action location. Compared with the existing technology, the present invention can more accurately capture the essential attributes of the action, significantly improving the accuracy and generalization ability of action location. The present invention adopts a clustering-based temporal action location method, which is combined with innovative feature extraction technology to effectively identify the temporal structure of cyclic actions in videos and accurately determine the start and end frames of the action. The introduction of this method not only improves the accuracy of action location, but also enhances the overall processing efficiency due to the improvement in the efficiency and quality of feature extraction.

Description

Technical Field

[0001] The present invention relates to the fields of data processing and artificial intelligence technology, and in particular to a clustering-based temporal action positioning method, device and storage medium for a cyclic action video. Background Art

[0002] In the field of video processing technology, video temporal action localization technology aims to accurately identify and locate specific action events in video sequences. Existing technical solutions include methods based on template matching, feature matching, traditional machine learning, and deep learning. Deep learning, as an emerging technology, has attracted considerable attention due to its powerful feature learning capabilities. For example, convolutional neural networks (CNNs) combined with recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) can capture spatiotemporal features in video sequences.

[0003] While these deep learning methods have achieved remarkable results in many applications, they suffer from significant limitations when processing looping action videos, including real-time performance issues, adaptability issues, insufficient accuracy, and limited generalization capabilities. Existing deep learning models generally rely on large-scale annotated datasets for training and have complex model structures, resulting in significant computational resource consumption during implementation and making it difficult to meet the demands of real-time processing. In particular, when processing looping action videos, due to the repetitive and continuous nature of the movements, existing models often struggle to effectively identify and locate looping actions, resulting in reduced positioning accuracy.

[0004] While traditional image processing techniques, such as background subtraction, frame differencing, and optical flow, have been applied to motion target detection and tracking, they are significantly inadequate for processing the temporal information of cyclical motions and are unable to effectively utilize the cyclical nature of motion. While spatiotemporal feature fusion attempts to improve motion localization accuracy by integrating spatial and temporal features, its complex network structure and large number of parameters not only complicate model training but also hinder its real-time and adaptability when processing cyclical motion videos.

[0005] Hidden Markov Models (HMMs), as a powerful tool for modeling time series data, have been widely used in fields such as speech recognition, gene sequence analysis, and behavior recognition. HMMs can effectively capture dynamic patterns in time series data by modeling the data's implicit state transitions. Therefore, HMMs are considered an effective method for processing and analyzing time series data. However, traditional HMM applications have primarily focused on single action recognition and still have certain limitations in localizing recurrent actions in time series.

[0006] To overcome the above problems, this paper proposes a temporal action localization method for videos with cyclic actions based on HMM clustering. This method clusters video features and combines them with a cyclic node algorithm to accurately identify and localize cyclic actions in videos. Summary of the Invention

[0007] In light of this, one of the objectives of the present invention is to provide a clustering-based method for localizing temporal actions in videos containing cyclical actions. This method accurately captures the characteristics of cyclical actions in videos, improving the accuracy of action localization. By leveraging the efficient classification properties of clustering algorithms, the method increases processing speed and enables real-time processing.

[0008] One of the objectives of the present invention is achieved through the following technical solutions:

[0009] The clustering-based temporal action localization method for cyclic action videos includes:

[0010] Step S1: Collect cyclic action video data and pre-process the video to highlight the action features. The pre-processed video is classified and stored by action category; images are extracted frame by frame and stored according to the original video name; the video is annotated, with detailed segmentation and marking of the start and end time of each action to ensure data accuracy and usability;

[0011] Step S2: Perform model design and self-supervised training, and perform self-supervised learning through contrastive loss function to enhance the model's ability to learn the temporal features of video clips;

[0012] Step S3: Train the feature extractor and add smoothness constraints to achieve smoothness of video features in the time series, reducing positioning errors. A domain adversarial transfer learning mechanism is also incorporated to improve the generalization ability of the feature extractor across different domains, ensuring feature consistency and accuracy during the transfer learning process.

[0013] Step S4: Fine-tune the feature extractor using the labeled dataset in step S1;

[0014] Step S5: Select representative frame images from each video so as to extract the features of the video using the trained model;

[0015] Step S6: Identify the temporal structure of the action in the video by performing cluster analysis on the extracted features, and determine the start and end frames of the action;

[0016] Step S7: By detecting the periodic pattern in the feature sequence, the recurring category sequences are identified and determined as cyclic actions. For these periodic category sequences, the start and end boundaries of each cyclic action are determined.

[0017] Furthermore, in step S2, a neural network model based on temporal contrastive learning is first designed. The model includes an encoder and a projection head. The encoder is used to extract feature vectors of video clips, and the projection head maps the feature vectors to a low-dimensional space for contrastive learning. The specific steps are as follows:

[0018] For video V, extract segments of different time periods from the same video to generate positive sample pairs, and extract time segments from different videos to generate negative sample pairs;

[0019] The feature vectors of the positive sample pair and the negative sample pair are mapped to the low-dimensional space through the projection head and denoted as 、 and 、 , then calculate their similarity matrix, and use the cosine similarity formula to calculate the similarity between feature vectors, as follows Represents the positive sample similarity, Represents the negative sample similarity:

[0020] ;

[0021] ;

[0022] The contrast loss function InfoNCE loss is defined to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The calculation formula is:

[0023] ;

[0024] in, Indicates the samples and The loss value of the sample, is the indicator function, when When the value is 1, is the temperature parameter; the similarity matrix is optimized according to the contrastive loss function. In each round of training, the model parameters are updated using the contrastive loss function to ensure that the model can accurately learn the temporal characteristics of the video clips.

[0025] Furthermore, in step S3, according to the total number of frames of the video and clip length Calculate the interval between each frame , and calculate the corresponding offset , ensuring that the video frames are fed into the network for training in sequence, the formula is:

[0026] ;

[0027] where the subscript i represents the i-th offset, and for all i = 0, 1, 2, ..., - 1; By setting up a data loader, each video is selected from the dataset through the loader, and the frame sequence of each video is fed into the model for training in batches according to the time sequence of the video.

[0028] Furthermore, in the training process of constructing the feature extractor, domain adversarial transfer learning technology and adding smoothness constraints were adopted; this included first defining the source domain as the video data in the Something-Something V2 dataset, and the target domain as the cyclic action video data collected in step S1, and initializing a shared feature extractor to extract useful feature information from the two domains; then setting a domain classifier to accurately classify the extracted features into the source domain or the target domain.

[0029] Furthermore, the specific steps of using the domain classifier to accurately classify the extracted features into the source domain or the target domain are as follows:

[0030] First, features are extracted from the source and target domains and fed into the domain classifier;

[0031] Secondly, the domain classifier loss is calculated to evaluate its ability to distinguish between the source domain and the target domain, the domain classification loss The formula is as follows:

[0032] ;

[0033] in, is the true domain label of the i-th sample, the source domain is 1 and the target domain is 0; is the probability that the domain classifier predicts that the i-th sample is the source domain;

[0034] Again, calculate the adversarial loss , the formula is as follows:

[0035] ;

[0036] in, is the number of categories, is the true label of the i-th sample Category indicator function (1 indicates the correct category, 0 indicates other categories), The model predicts that the i-th sample belongs to class probability;

[0037] In order to ensure the smoothness of the feature extractor on the time series, a smoothness constraint is introduced to implement a smoothness loss by imposing a difference penalty on the time series. , the formula is as follows:

[0038] ;

[0039] in, is a feature extractor, and are consecutive video frames, is the total number of frames after video sampling;

[0040] The total loss function is:

[0041] ;

[0042] in, and is a weight coefficient used to balance the importance of different loss terms.

[0043] Furthermore, in step S4, the specific steps of fine-tuning the trained feature extractor on the target domain using the dataset collected and annotated in step S1 are as follows:

[0044] First, for a given video V, assume there are N different actions in the video, marked as , . . . , , each action will appear multiple times in the video. , it will be in different time periods , . . . , Appears, marked as:

[0045] ;

[0046] Due to the lens changes in the video, the same action will be presented in different images at different time periods. In order to capture the periodic consistency of the action, the action periodic constraint loss is defined. , to ensure that the feature representation of the same action in different time periods is similar, assuming that the model extracts the features of each frame through the feature extractor ф, and the action In the time period and The characteristics are and , then the action periodicity constraint loss can be expressed as:

[0047] .

[0048] Furthermore, in step S5, the total number of frames of the video is first determined, and then the index of the frames to be extracted is calculated according to the selection strategy. Finally, the trained deep learning model is used to extract features of these selected frame images. These features will be used for subsequent clustering analysis to achieve the positioning of video timing actions.

[0049] Furthermore, in step S6, an HMM clustering algorithm is used to achieve efficient recognition and positioning of video actions, which specifically includes the following steps:

[0050] First, the features extracted from step S5 are preprocessed to ensure the quality and consistency of the data;

[0051] Based on the preprocessed features, a clustering algorithm is applied for grouping. The HMM clustering algorithm divides the data into different clusters by identifying the hidden states in the time series data, and each cluster corresponds to an action mode. The algorithm optimizes the HMM parameters using the Baum-Welch algorithm, and then uses the forward-backward algorithm to calculate the hidden state probability at each time point. Finally, based on the hidden state, the time segments are assigned to the corresponding clusters, thereby achieving effective clustering of actions.

[0052] A second object of the present invention is to provide a computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the aforementioned method when executing the computer program.

[0053] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which implements the method described above when the computer program is executed by a processor.

[0054] The beneficial effects of the present invention are:

[0055] 1. This invention has made significant innovations in feature extraction methods. During the deep learning model training process, not only does it add smoothness constraints, but it also incorporates self-supervised learning and domain adversarial transfer techniques, thereby extracting smoother and domain-invariant video features. This comprehensive feature extraction method not only reduces feature fluctuations in time series but also ensures effective feature transfer between different domains, providing a more reliable and comprehensive foundation for subsequent clustering analysis. Compared with existing technologies, this invention can more accurately capture the essential attributes of actions, significantly improving the accuracy and generalization of action localization.

[0056] 2. The present invention adopts a clustering-based temporal action localization method, which is combined with innovative feature extraction technology. It can effectively identify the temporal structure of cyclic actions in videos and accurately determine the start and end frames of the action. The introduction of this method not only improves the accuracy of action localization, but also enhances the overall processing efficiency due to the improvement of the efficiency and quality of feature extraction.

[0057] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description and the preceding claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0059] Figure 1 is a flow chart of the method of the present invention;

[0060] Figure 2 This is a flow chart of feature extractor training of the present invention;

[0061] Figure 3 It is a feature distance grayscale image of a specific video in the embodiment.

[0062] Figure 4 This is a feature clustering result diagram of a specific video in the embodiment. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0064] like Figure 1 As shown, the clustering-based temporal action localization method with cyclic action video of the present invention comprises the following steps:

[0065] Step S1: Collect cyclic action video data and preprocess the video to highlight action features. The preprocessed video is categorized and stored by action category. Next, extract the images frame by frame and organize them into storage groups based on the original video names. Simultaneously, the video is annotated, with detailed segmentation and marking of the start and end times of each action to ensure data accuracy and usability.

[0066] In this example, to ensure data quality, the video was preprocessed by denoising, scaling, and cropping to highlight the action. The preprocessed videos were then stored by category. The OpenCV library was used to parse the videos, extracting images frame by frame. The frames were then organized and stored with the original video names to facilitate subsequent processing and model training.

[0067] Step S2: Model design and self-supervised training are performed, using a contrastive loss function to enhance the model's ability to learn the temporal features of the video clips. In step S2 of this embodiment, a neural network model based on temporal contrastive learning is first designed. The model includes an encoder and a projection head. The encoder is used to extract the feature vectors of the video clips, and the projection head maps the feature vectors to a low-dimensional space for contrastive learning. The specific steps are as follows:

[0068] For video V, extract segments of different time periods from the same video to generate positive sample pairs, and extract time segments from different videos to generate negative sample pairs;

[0069] The feature vectors of the positive sample pair and the negative sample pair are mapped to the low-dimensional space through the projection head and denoted as 、 and 、 , then calculate their similarity matrix, and use the cosine similarity formula to calculate the similarity between feature vectors, as follows Represents the positive sample similarity, Represents the negative sample similarity:

[0070] ;

[0071] ;

[0072] The contrast loss function InfoNCE loss is defined to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The calculation formula is:

[0073] ;

[0074] in, Indicates the samples and The loss value of the sample, is the indicator function, when When the value is 1, is the temperature parameter; the similarity matrix is optimized according to the contrastive loss function. In each round of training, the model parameters are updated using the contrastive loss function to ensure that the model can accurately learn the temporal characteristics of the video clips.

[0075] Step S3: Train the feature extractor and add smoothness constraints to achieve smoothness of video features in time series and reduce positioning errors. At the same time, a domain adversarial transfer learning mechanism is incorporated to improve the generalization ability of the feature extractor between different domains and ensure the consistency and accuracy of features during the transfer learning process.

[0076] The specific operations are:

[0077] First, based on the total number of frames in the video and clip length Calculate the interval between each frame , and calculate the corresponding offset , ensuring that the video frames are fed into the network for training in sequence, the formula is:

[0078] ;

[0079] where the subscript i represents the i-th offset, and for all i = 0, 1, 2, ..., - 1; By setting up a data loader, each video is selected from the dataset through the loader, and the frame sequence of each video is fed into the model for training in batches according to the time sequence of the video.

[0080] Furthermore, to improve network training efficiency and ensure that each video frame is trained in temporal order, a data loader was designed. This loader selects training videos from the dataset each time, but rather than feeding all frames of these videos into the model simultaneously, it feeds each video's frame sequence into the model one by one for training, following the video's temporal order. This approach avoids unnecessary performance overhead and ensures that the Euclidean distance between the features of the previous and current video frames can be calculated during training, thus providing the necessary guarantees for achieving smoothness constraints. This ensures that the model learns the temporal relationship between frames during training, which is crucial for improving the accuracy of action localization.

[0081] During the training process of building the feature extractor, we employed domain adversarial transfer learning techniques and added smoothness constraints to achieve effective feature transfer between domains and improve time series smoothness. First, we defined the features of the source and target domains and initialized a shared feature extractor to extract useful feature information from both domains. Next, we set up a domain classifier to accurately classify the extracted features as either the source or target domain. The specific steps for using the domain classifier to accurately classify the extracted features as either the source or target domain are as follows:

[0082] First, features are extracted from the source and target domains and fed into the domain classifier;

[0083] Secondly, the domain classifier loss is calculated to evaluate its ability to distinguish between the source domain and the target domain, the domain classification loss The formula is as follows:

[0084] ;

[0085] in, is the true domain label of the i-th sample, the source domain is 1 and the target domain is 0; is the probability that the domain classifier predicts that the i-th sample is the source domain;

[0086] Again, calculate the adversarial loss , the formula is as follows:

[0087] ;

[0088] in, is the number of categories, is the true label of the i-th sample Category indicator function (1 indicates the correct category, 0 indicates other categories), The model predicts that the i-th sample belongs to class probability;

[0089] In order to ensure the smoothness of the feature extractor on the time series, a smoothness constraint is introduced to implement a smoothness loss by imposing a difference penalty on the time series. , the formula is as follows:

[0090] ;

[0091] in, is a feature extractor, and are consecutive video frames, is the total number of frames after video sampling;

[0092] The total loss function is:

[0093] ;

[0094] in, and is a weight coefficient used to balance the importance of different loss terms.

[0095] Step S4: Fine-tune the trained feature extractor on the target domain to better adapt to the specific tasks of the target domain. First, for a given video V, assume that there are N different actions in the video, marked as , . . . , , each action will appear multiple times in the video. , it will be in different time periods , . . . , Appears, marked as:

[0096] ;

[0097] Due to the lens changes in the video, the same action will be presented in different images at different time periods. In order to capture the periodic consistency of the action, the action periodic constraint loss is defined. , to ensure that the feature representation of the same action in different time periods is similar, assuming that the model extracts the features of each frame through the feature extractor ф, and the action In the time period and The characteristics are and , then the action periodicity constraint loss can be expressed as:

[0098] .

[0099] Step S5: Select representative frame images from each video so as to extract the features of the video using the trained model; in this embodiment, a specific selection strategy of selecting one frame every three frames is adopted. Specifically, for each video, no fixed clip length is set, but the number of frames selected is determined based on the actual length of the video. If the video data is very long, then according to a n =3n-1, where n is a positive integer. This strategy yields more frame images, helping to more comprehensively reflect the motion changes and timing information in the video. This approach has the advantage of adapting to videos of varying lengths without requiring a predefined clip length, thus increasing the flexibility and adaptability of data processing.

[0100] During implementation, the total number of frames in the video is first determined. Then, based on the selection strategy, the indices of the frames to be extracted are calculated. Next, a trained deep learning model is used to extract features from these selected frames. These features are then used in subsequent clustering analysis to localize temporal actions in the video.

[0101] Step S6: By performing cluster analysis on the extracted features, the temporal structure of the action in the video is identified, and the start and end frames of the action are determined. In step S5 of this embodiment, the HMM clustering algorithm is used to achieve efficient recognition and positioning of the video action, which specifically includes the following steps:

[0102] First, the features extracted from step S4 are preprocessed to ensure the quality and consistency of the data; this includes standardizing, normalizing, or regularizing the features to eliminate differences between the features and improve the performance of the clustering algorithm;

[0103] Based on the preprocessed features, a clustering algorithm is applied to group the data. The HMM clustering algorithm identifies the hidden states in the time series data and divides the data into different clusters, each corresponding to an action pattern. The algorithm optimizes the HMM parameters using the Baum-Welch algorithm, then uses the forward-backward algorithm to calculate the hidden state probability at each time point. Finally, based on the hidden state, the time segments are assigned to the corresponding clusters, thus achieving effective clustering of actions.

[0104] Step S7: By detecting periodic patterns in the feature sequence, recurring category sequences are identified and determined as cyclical actions. For these periodic category sequences, the start and end boundaries of each cyclic action are determined. The start and end points of each cyclic action cycle are defined as loop nodes, which are used to identify the precise boundaries of the action. For other categories that do not exhibit a periodic pattern in time, the video content is deemed to contain no action or not be part of a cyclic action.

[0105] Specific implementation examples

[0106] Figure 3 This is a grayscale image of the distance between features extracted from a radio gymnastics video clip, along with a binarized grayscale image. The video shows an athlete repeating a certain movement twice. The video has 834 frames, with the first 417 frames and the last 417 frames essentially identical. The horizontal and vertical coordinates of the grayscale image represent the index of the video frame. White areas in the grayscale image indicate close feature distances, while black areas indicate larger distances between the corresponding two frames. The binarized grayscale image shows that the colors change regularly in both the horizontal and vertical directions.

[0107] Figure 4 The features extracted from this radio gymnastics video are clustered using HMM. Class 1 includes frames 0-196 and 422-620, class 2 includes frames 197-360 and 621-760, and class 0 includes frames 361-421 and 761-834. This indicates that the corresponding categories for this video are the sequence: class 1, class 2, class 0, class 1, class 2, class 0. Finally, the clustering results indicate that the original video's movements begin with frame 0 as the start frame of the first movement pass, end with frame 421, and begin and end with frames 422 and 834 as the second movement pass.

[0108] The present invention adopts a self-supervised training method to train the basic model. This method does not rely on a large amount of labeled data and can automatically learn useful features from video data. Subsequently, in order to further improve the pertinence and generalization ability of feature extraction, the present invention introduces domain adversarial transfer learning technology. This technology enables the model to extract more discriminative features in different video domains, so as to better focus on the action behavior itself. Finally, the present invention uses clustering algorithms and loop nodes to efficiently locate the features obtained through self-supervision and adversarial transfer learning. This method is particularly suitable for action positioning in looping videos because it can effectively identify and locate repeated actions in the video, thereby improving the continuity and stability of positioning. The present invention introduces smoothness constraints in the model training process to ensure that the extracted video features are continuous and stable in the time dimension, thereby providing a reliable basis for subsequent clustering analysis.

[0109] It should be appreciated that embodiments of the present invention can be implemented or practiced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The methods can be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner, according to the methods and figures described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed application-specific integrated circuit for this purpose.

[0110] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that can be executed by one or more processors.

[0111] Further, the methods can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, RAM, ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. In addition, the machine-readable code, or portions thereof, can be transmitted over a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A clustering-based temporal action localization method for cyclic action videos, characterized by: The method comprises: Step S1: Collect cyclic action video data and pre-process the video to highlight the action features. The pre-processed video is classified and stored by action category; images are extracted frame by frame and stored according to the original video name; the video is annotated, with detailed segmentation and marking of the start and end time of each action to ensure data accuracy and usability; Step S2: Perform model design and self-supervised training, and perform self-supervised learning through a contrastive loss function to enhance the model's ability to learn the temporal features of video clips. In step S2, a neural network model based on temporal contrastive learning is first designed. The model includes an encoder and a projection head. The encoder is used to extract feature vectors of video clips, and the projection head maps the feature vectors to a low-dimensional space for contrastive learning. The specific steps are as follows: For video V, extract segments of different time periods from the same video to generate positive sample pairs, and extract time segments from different videos to generate negative sample pairs; The feature vectors of the positive sample pair and the negative sample pair are mapped to the low-dimensional space through the projection head and denoted as 、 and 、 , then calculate their similarity matrix, and use the cosine similarity formula to calculate the similarity between feature vectors, as follows Represents the positive sample similarity, Represents the negative sample similarity: ； ； The contrast loss function InfoNCE loss is defined to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The calculation formula is: ； in, Indicates the samples and The loss value of the sample, is the indicator function, when When the value is 1, is the temperature parameter; the similarity matrix is optimized according to the contrastive loss function. In each round of training, the model parameters are updated using the contrastive loss function to ensure that the model can accurately learn the temporal characteristics of the video clips; Step S3: Train the feature extractor and add smoothness constraints to achieve smoothness of video features in time series and reduce positioning errors. At the same time, the domain adversarial transfer learning mechanism is incorporated to improve the generalization ability of the feature extractor between different domains and ensure the consistency and accuracy of features in the transfer learning process. In step S3, according to the total number of frames of the video, and clip length Calculate the interval between each frame , and calculate the corresponding offset , ensuring that the video frames are fed into the network for training in sequence, the formula is: ； where the subscript i represents the i-th offset, and for all i = 0, 1, 2, ..., -1; By setting up a data loader, each video is selected from the dataset through the loader, and the frame sequence of each video is fed into the model for training in batches according to the time sequence of the video; Step S4: Fine-tune the feature extractor using the labeled dataset in step S1; Step S5: Select representative frame images from each video so as to extract the features of the video using the trained feature extractor; Step S6: Identify the temporal structure of the action in the video by performing cluster analysis on the extracted features, and determine the start and end frames of the action; Step S7: By detecting the periodic pattern in the feature sequence, the recurring category sequences are identified and determined as cyclic actions. For these periodic category sequences, the start and end boundaries of each cyclic action are determined.

2. The clustering-based temporal action localization method for cyclic action videos according to claim 1, characterized in that: In the training process of constructing the feature extractor, domain adversarial transfer learning technology and adding smoothness constraints are adopted. This includes first defining the source domain as the video data in the Something-Something V2 dataset and the target domain as the cyclic action video data collected in step S1. A shared feature extractor is initialized to extract useful feature information from the two domains. Then, a domain classifier is set to accurately classify the extracted features into the source domain or the target domain.

3. The clustering-based temporal action localization method for cyclic action videos according to claim 2, characterized in that: The specific steps of using the domain classifier to accurately classify the extracted features into the source domain or target domain are as follows: First, features are extracted from the source and target domains and fed into the domain classifier; Secondly, the domain classification loss L is calculated to evaluate its ability to distinguish between the source domain and the target domain. domain The calculation formula is as follows: ； in, is the true domain label of the i-th sample, the source domain is 1 and the target domain is 0; is the probability that the domain classifier predicts that the i-th sample is the source domain; Again, calculate the adversarial loss , the formula is as follows: ； in, is the number of categories, is the true label of the i-th sample Category indicator function, 1 represents the correct category, 0 represents other categories; The model predicts that the i-th sample belongs to class probability; In order to ensure the smoothness of the feature extractor on the time series, a smoothness constraint is introduced to implement a smoothness loss by imposing a difference penalty on the time series. , the formula is as follows: ； in, is a feature extractor, and are consecutive video frames, is the total number of frames after video sampling; The total loss function is: ； in, and is a weight coefficient used to balance the importance of different loss terms.

4. The clustering-based temporal action localization method for cyclic action videos according to claim 1, characterized in that: In step S4, the specific steps of fine-tuning the trained feature extractor on the target domain using the dataset collected and annotated in step S1 are as follows: First, for a given video V, assume there are N different actions in the video, marked as , . . . , , each action will appear multiple times in the video. , it will be in different time periods , . . . , Appears, marked as: ； Due to the lens changes in the video, the same action will be presented in different images at different time periods. In order to capture the periodic consistency of the action, the action periodic constraint loss is defined. , to ensure that the feature representation of the same action in different time periods is similar, assuming that the model extracts the features of each frame through the feature extractor ф, and the action In the time period and The characteristics are and , then the action periodicity constraint loss can be expressed as: 。 5. The clustering-based temporal action localization method for cyclic action videos according to claim 1, characterized in that: In step S5, the total number of frames in the video is first determined, and then the index of the frames to be extracted is calculated according to the selection strategy. Finally, the trained deep learning model is used to extract features from these selected frame images. These features will be used for subsequent clustering analysis to achieve the positioning of video temporal actions.

6. The clustering-based temporal action localization method for cyclic action videos according to claim 1, characterized in that: In step S6, the HMM clustering algorithm is used to achieve efficient recognition and positioning of video actions, which specifically includes the following steps: First, the features extracted from step S5 are preprocessed to ensure the quality and consistency of the data; Based on the preprocessed features, a clustering algorithm is applied for grouping. The HMM clustering algorithm divides the data into different clusters by identifying the hidden states in the time series data, and each cluster corresponds to an action mode. The algorithm optimizes the HMM parameters using the Baum-Welch algorithm, and then uses the forward-backward algorithm to calculate the hidden state probability at each time point. Finally, based on the hidden state, the time segments are assigned to the corresponding clusters, thereby achieving effective clustering of actions.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images