Cyclic action video time sequence action positioning method and device based on time sequence modeling and storage medium

Abstract

The invention discloses a time sequence modeling-based cyclic action video time sequence action positioning method and device and a storage medium. The method comprises the following steps: firstly, carrying out video feature extraction through self-supervised training and domain adversarial transfer learning, and secondly, carrying out efficient action positioning by utilizing a clustering algorithm. Compared with the prior art, the method has the advantages that the essential attribute of the action can be more accurately captured, and the accuracy and generalization ability of action positioning are remarkably improved; the time sequence action positioning method based on clustering is adopted, the method is combined with the innovative feature extraction technology, the cyclic action time sequence structure in the video can be effectively recognized, the starting frame and the ending frame of the action can be accurately determined, and due to introduction of the method, the accuracy of action positioning is improved; moreover, the efficiency and quality of feature extraction are improved, so that the overall processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing and artificial intelligence technology, and in particular to a clustering-based temporal action positioning method, device and storage medium for a cyclic action video. Background Art

[0002] In the field of video processing technology, video temporal action localization technology aims to accurately identify and locate specific action events from video sequences. Existing technical solutions include methods based on template matching, methods based on feature matching, methods based on traditional machine learning, and methods based on deep learning, etc. Among them, deep learning methods, as an emerging technology, have attracted much attention due to their powerful feature learning capabilities, such as convolutional neural networks (CNN) combined with recurrent neural networks (RNN) or long short-term memory networks (LSTM) to capture the spatiotemporal features in video sequences.

[0003] Although these deep learning methods have achieved remarkable results in many applications, they still have some significant deficiencies when processing looping action videos, including real-time problems, adaptability problems, insufficient accuracy, and limited generalization ability. Existing deep learning models generally rely on large-scale labeled datasets for training, and the model structure is complex, resulting in huge consumption of computing resources during implementation, making it difficult to meet the needs of real-time processing. Especially when processing looping action videos, due to the repetitiveness and continuity of the action, existing models often have difficulty in effectively identifying and locating looping actions, resulting in reduced positioning accuracy.

[0004] Traditional image processing techniques, such as background subtraction, frame difference, and optical flow, have been used in motion target detection and tracking. However, when processing cyclic action videos, these methods are obviously insufficient in processing the timing information of cyclic actions and cannot effectively utilize the periodic characteristics of actions. Although spatiotemporal feature fusion technology attempts to improve the accuracy of action positioning by integrating spatial and temporal features, due to its complex network structure and a large number of parameters, it not only increases the difficulty of model training, but also makes it difficult to achieve real-time and adaptability requirements when processing cyclic action videos.

[0005] As a powerful time series data modeling tool, Hidden Markov Models (HMM) have been widely used in speech recognition, gene sequence analysis, and behavior recognition. HMM can effectively capture the dynamic patterns in time series data by modeling the implicit state transition process of the data. Therefore, HMM is considered to be one of the effective methods for processing and analyzing time series data. However, traditional HMM applications are mainly focused on single action recognition, and there are still certain limitations for the time series positioning of cyclic actions.

[0006] In order to overcome the above problems, the present invention proposes a method for locating cyclic action videos based on temporal modeling clustering. The method clusters video features and combines a cyclic node algorithm to accurately identify and locate cyclic actions in videos. Summary of the invention

[0007] In view of this, one of the purposes of the present invention is to provide a method for locating temporal actions in videos with cyclic actions based on clustering. The cyclic action features in the video can be accurately captured to improve the accuracy of action location. The efficient classification characteristics of the clustering algorithm are utilized to improve the processing speed and achieve real-time processing.

[0008] One of the purposes of the present invention is achieved through the following technical solutions: The method for locating temporal actions in cyclic action videos based on temporal modeling includes: Step S1: Collect cyclic action video data and preprocess the video to highlight the action features. The preprocessed video is classified and stored by action category; images are extracted frame by frame and organized and stored according to the original video name; the video is annotated at the same time, and the start and end time of each action is detailed and marked to ensure the accuracy and availability of the data; Step S2: Perform model design and self-supervised training, and perform self-supervised learning through contrast loss function to enhance the model's ability to learn the temporal features of video clips; Step S3: Train the feature extractor and add smoothness constraints to achieve smoothness of video features in time series and reduce positioning errors. At the same time, a domain adversarial transfer learning mechanism is incorporated to improve the generalization ability of the feature extractor between different domains and ensure the consistency and accuracy of features during the transfer learning process. Step S4: fine-tune the feature extractor using the labeled data set in step S1; Step S5: Select representative frame images from each video so as to extract the features of the video using the trained model; Step S6: Identify the action timing structure in the video by clustering the extracted features and determine the start and end frames of the action; Step S7: by detecting the periodic patterns in the feature sequence, the recurring category sequences are identified and determined as cyclic actions, and for these periodic category sequences, the start and end boundaries of each cyclic action are determined.

[0009] Further, in step S2, a neural network model based on temporal contrastive learning is first designed. The model includes an encoder and a projection head. The encoder is used to extract feature vectors of video clips. The projection head maps the feature vectors to a low-dimensional space for contrastive learning. The specific steps are as follows: For video V, extract segments of different time periods from the same video to generate positive sample pairs, and extract time segments from different videos to generate negative sample pairs; The feature vectors of the positive sample pair and the negative sample pair are mapped to the low-dimensional space through the projection head and denoted as z i 、z j and z i 、z v , then calculate their similarity matrix, and use the cosine similarity formula to calculate the similarity between feature vectors. In the following formula, s ij Represents the similarity of positive samples, s iv Represents the negative sample similarity: The contrast loss function InfoNCE loss is defined to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The calculation formula is: in, Represents the loss value of the i-th sample and the j-th sample, 1 [k≠i] is an indicator function, which takes the value of 1 when k≠i, and τ is a temperature parameter. The similarity matrix is ​​optimized according to the contrast loss function. In each round of training, the model parameters are updated using the contrast loss function to ensure that the model can accurately learn the temporal characteristics of the video clips.

[0010] Further, in step S3, according to the total number of frames N of the video f and the clip length L c Calculate the interval between each frame And calculate the corresponding offset O i , ensuring that the video frames are fed into the network in order for training, the formula is: Where the subscript i represents the i-th offset, for all i = 0, 1, 2, ..., L c -1; By setting up a data loader, each video is selected from the dataset through the loader, and the frame sequence of each video is sent to the model for training in batches according to the time sequence of the video.

[0011] Furthermore, in the training process of constructing the feature extractor, domain adversarial transfer learning technology and adding smoothness constraints were adopted; including first defining the source domain as the video data in the Something-Something V2 dataset, and the target domain as the cyclic action video data collected in step S1, and initializing a shared feature extractor to extract useful feature information from the two domains; then setting a domain classifier to accurately classify the extracted features into the source domain or the target domain.

[0012] Furthermore, the specific steps of using the domain classifier to accurately classify the extracted features into the source domain or the target domain are as follows: First, features are extracted from the source and target domains and fed into the domain classifier; Secondly, the loss of the domain classifier is calculated to evaluate its ability to distinguish between the source domain and the target domain. The domain classification loss L domain The formula is as follows: Among them, y i is the true domain label of the i-th sample, the source domain is 1 and the target domain is 0; is the probability that the i-th sample predicted by the domain classifier is the source domain; Again, calculate the adversarial loss L class , the formula is as follows: Where K is the number of categories, is the k-th category indicator function of the true label of the i-th sample (1 indicates the correct category, 0 indicates other categories), is the probability that the i-th sample belongs to the k-th class predicted by the model; In order to ensure the smoothness of the feature extractor on the time series, a smoothness constraint is introduced to implement the smoothness loss L by imposing a difference penalty on the time series. smooth , the formula is as follows: Among them, φ is the feature extractor, x i and x i+1 is the number of consecutive video frames, N is the total number of frames after video sampling; The total loss function is: L=L domain +λ class L class +λ smooth L smooth ; Among them, λ class and λ smooth is a weight coefficient used to balance the importance of different loss terms.

[0013] Furthermore, in step S4, the specific steps of fine-tuning the trained feature extractor on the target domain using the data set collected and annotated in step S1 are as follows: First, for a given video V, assume that there are N different actions in the video, marked as a1, a2, a3, ..., a n , each action will appear multiple times in the video. For action ak , it will be in different time periods t1, t2, t3, ..., t n Appears, marked as: Due to the change of the lens in the video, the same action will present different pictures in different time periods. In order to capture the periodic consistency of the action, the action periodic constraint loss Loss is defined cycle , to ensure that the feature representation of the same action in different time periods is similar, assuming that the model extracts the features of each frame through the feature extractor ф, and the action a k In time period and The characteristics are and Then the action periodicity constraint loss can be expressed as:

[0014] Furthermore, in step S5, the total number of frames of the video is first determined, and then the index of the frame to be extracted is calculated according to the selection strategy. Finally, the trained deep learning model is used to extract features of these selected frame images. These features will be used for subsequent clustering analysis to achieve the positioning of video timing actions.

[0015] Furthermore, in step S6, the HMM clustering algorithm is used to realize efficient recognition and positioning of video actions, which specifically includes the following steps: First, the features extracted from step S5 are preprocessed to ensure the quality and consistency of the data; According to the preprocessed features, the clustering algorithm is applied for grouping. The HMM clustering algorithm divides the data into different clusters by identifying the hidden states in the time series data. Each cluster corresponds to an action mode. The algorithm optimizes the HMM parameters by using the Baum-Welch algorithm, and then uses the forward-backward algorithm to calculate the hidden state probability at each time point. Finally, according to the hidden state, the time segments are assigned to the corresponding clusters, thereby achieving effective clustering of actions.

[0016] A second object of the present invention is to provide a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the method described above when executing the computer program.

[0017] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.

[0018] The beneficial effects of the present invention are: 1. Effective innovation in feature extraction methods. In the process of deep learning model training, the present invention not only adds smoothness constraints, but also incorporates self-supervised learning and domain adversarial migration technology, so as to extract smoother and domain-invariant video features. This comprehensive feature extraction method not only reduces the fluctuation of features in time series, but also ensures the effective migration of features between different domains, providing a more reliable and comprehensive foundation for subsequent clustering analysis. Compared with the prior art, the present invention can more accurately capture the essential attributes of actions and significantly improve the accuracy and generalization ability of action positioning; 2. The present invention adopts a clustering-based temporal action localization method, which is combined with innovative feature extraction technology and can effectively identify the temporal structure of cyclic actions in videos and accurately determine the start and end frames of actions. The introduction of this method not only improves the accuracy of action localization, but also enhances the overall processing efficiency due to the improvement of feature extraction efficiency and quality.

[0019] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description and the preceding claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which: Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a flow chart of feature extractor training of the present invention; Figure 3 It is a feature distance grayscale image of a specific video in the embodiment.

[0021] Figure 4 This is a feature clustering result diagram of a specific video in the embodiment. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0023] like Figure 1 As shown, the method for locating time-sequence actions in cyclic action videos based on time-sequence modeling of the present invention comprises the following steps: Step S1: Collect cyclic action video data and preprocess the video to highlight the action features; the preprocessed video is classified and stored by action category. Then, the image is extracted frame by frame and stored in an organized manner according to the original video name; at the same time, the video is annotated, and the start and end time of each action is detailed and marked to ensure the accuracy and availability of the data.

[0024] In this embodiment, to ensure data quality, the video is pre-processed by denoising, scaling and cropping to highlight the action. The pre-processed video is stored by category. The OpenCV library is used to parse the video, extract images frame by frame, and organize and store the image frames with the original video name for subsequent processing and model training.

[0025] Step S2: Perform model design and self-supervised training, and perform self-supervised learning through contrast loss function to enhance the model's learning ability of the temporal features of the video clips; In step S2 of this embodiment, a neural network model based on temporal contrastive learning is first designed, and the model includes an encoder and a projection head. The encoder is used to extract the feature vector of the video clip, and the projection head maps the feature vector to a low-dimensional space for contrastive learning. The specific steps are as follows: For video V, extract segments of different time periods from the same video to generate positive sample pairs, and extract time segments from different videos to generate negative sample pairs; The feature vectors of the positive sample pair and the negative sample pair are mapped to the low-dimensional space through the projection head and denoted as z i 、z j and z i 、z v , then calculate their similarity matrix, and use the cosine similarity formula to calculate the similarity between feature vectors. In the following formula, s ij Represents the similarity of positive samples, s iv Represents the negative sample similarity: The contrast loss function InfoNCE loss is defined to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The calculation formula is: in, Represents the loss value of the i-th sample and the j-th sample, 1 [k≠i] is an indicator function, which takes the value of 1 when k≠i, and τ is a temperature parameter. The similarity matrix is ​​optimized according to the contrast loss function. In each round of training, the model parameters are updated using the contrast loss function to ensure that the model can accurately learn the temporal characteristics of the video clips.

[0026] Step S3: Train the feature extractor and add smoothness constraints to achieve smoothness of video features in time series and reduce positioning errors. At the same time, a domain adversarial transfer learning mechanism is incorporated to improve the generalization ability of the feature extractor between different domains and ensure the consistency and accuracy of features during the transfer learning process.

[0027] The specific operations are: First, according to the total number of video frames N f and the clip length L c Calculate the interval between each frame And calculate the corresponding offset O i , ensuring that the video frames are fed into the network in order for training, the formula is: Where the subscript i represents the i-th offset, for all i = 0, 1, 2, ..., L c -1; By setting up a data loader, each video is selected from the dataset through the loader, and the frame sequence of each video is sent to the model for training in batches according to the time sequence of the video.

[0028] In addition, in order to improve the efficiency of network training and ensure that each video frame is trained in time sequence, a data loader is designed. The loader selects training videos from the data set each time, but does not send all frames of these videos to the model at the same time. Instead, it sends the frame sequence of each video to the model for training one by one according to the time sequence of the video. This method avoids extra useless performance overhead. It also ensures that the Euclidean distance between the features of the previous frame of the video and the features of the current frame can be calculated during training, thereby providing the necessary guarantee for achieving smoothness constraints. In this way, it is ensured that the model can learn the temporal relationship between frames during training, which is crucial to improving the accuracy of action localization.

[0029] In the process of training the feature extractor, domain adversarial transfer learning technology and smoothness constraints are used to achieve effective migration of features between different domains and improve their smoothness in time series. First, it is necessary to define the features of the source domain and the target domain, and initialize a shared feature extractor to extract useful feature information from the two domains. Then set a domain classifier to accurately classify the extracted features into the source domain or the target domain. The specific steps to use the domain classifier to accurately classify the extracted features into the source domain or the target domain are as follows: First, features are extracted from the source and target domains and fed into the domain classifier; Secondly, the loss of the domain classifier is calculated to evaluate its ability to distinguish between the source domain and the target domain. The domain classification loss L domain The formula is as follows: Among them, y i is the true domain label of the i-th sample, the source domain is 1 and the target domain is 0; is the probability that the i-th sample predicted by the domain classifier is the source domain; Again, calculate the adversarial loss L class , the formula is as follows: Where K is the number of categories, is the k-th category indicator function of the true label of the i-th sample (1 indicates the correct category, 0 indicates other categories), is the probability that the i-th sample belongs to the k-th class predicted by the model; In order to ensure the smoothness of the feature extractor on the time series, a smoothness constraint is introduced to implement the smoothness loss L by imposing a difference penalty on the time series. smooth , the formula is as follows: Among them, φ is the feature extractor, x i and x i+1 is the number of consecutive video frames, N is the total number of frames after video sampling; The total loss function is: L=L domain +λ class L class +λ smooth L smooth ; Among them, λ class and λ smooth is a weight coefficient used to balance the importance of different loss terms.

[0030] Step S4: Fine-tune the trained feature extractor on the target domain to better adapt to the specific tasks of the target domain. First, for a given video V, assume that there are N different actions in the video, marked as a1, a2, a3, ..., a n , each action will appear multiple times in the video. For action a k , it will be in different time periods t1, t2, t3, ..., t n Appears, marked as: Due to the change of the lens in the video, the same action will present different pictures in different time periods. In order to capture the periodic consistency of the action, the action periodic constraint loss Loss is defined cycle , to ensure that the feature representation of the same action in different time periods is similar, assuming that the model extracts the features of each frame through the feature extractor ф, and the action ak In time period and The characteristics are and Then the action periodicity constraint loss can be expressed as:

[0031] Step S5: Select representative frame images from each video so as to extract the features of the video using the trained model; in this embodiment, a specific selection strategy of selecting one frame every three frames is adopted. Specifically, for each video, a fixed clip length is not set, but the number of frames selected is determined according to the actual length of the video. If the video data is very long, then according to a n =3n-1 selection strategy, where n is a positive integer, will obtain more frame images, which helps to more comprehensively reflect the action changes and timing information in the video. The advantage of this method is that it can adapt to videos of different lengths without pre-setting the clip length, thereby improving the flexibility and adaptability of data processing.

[0032] In the implementation process, the total number of frames in the video is first determined, and then the index of the frames to be extracted is calculated based on the selection strategy. Next, the trained deep learning model is used to extract features from these selected frame images. These features will be used in subsequent clustering analysis to achieve the location of video temporal actions.

[0033] Step S6: by clustering the extracted features, identify the temporal structure of the action in the video, and determine the start and end frames of the action; in step S5 of this embodiment, the HMM clustering algorithm is used to achieve efficient recognition and positioning of the video action, which specifically includes the following steps: First, the features extracted from step S4 are preprocessed to ensure the quality and consistency of the data; this includes standardizing, normalizing or regularizing the features to eliminate differences between the features and improve the performance of the clustering algorithm; According to the preprocessed features, clustering algorithms are applied for grouping. The HMM clustering algorithm divides the data into different clusters by identifying the hidden states in the time series data. Each cluster corresponds to an action mode. The algorithm optimizes the HMM parameters using the Baum-Welch algorithm, and then uses the forward-backward algorithm to calculate the hidden state probability at each time point. Finally, according to the hidden state, the time segment is assigned to the corresponding cluster, thereby achieving effective clustering of actions. Step S7: By detecting the periodic pattern in the feature sequence, the recurring category sequence is identified and determined as a cyclic action, and for these periodic category sequences, the start and end boundaries of each cyclic action are determined. The start and end points of each cyclic action cycle are defined as cycle nodes to identify the exact boundaries of the action. For other categories that do not present a cycle in time sequence, it is considered that the part of the video content does not contain an action or is not part of a cyclic action.

[0034] Specific implementation examples Figure 3 This is a grayscale image of the distance between features extracted from a radio gymnastics video clip and a grayscale image after binarization. The video content is an athlete repeating a certain action twice. The video has a total of 834 frames, and the content of the first 417 frames and the last 417 frames are basically the same; the horizontal and vertical coordinates of the grayscale image both represent the index of the video frame. The white part in the grayscale image indicates that the feature distance is close, and the black part indicates that the distance between the corresponding two frames is large. From the binarized grayscale image, it can be seen that the color changes regularly in both the horizontal and vertical directions. Figure 4 The results of HMM clustering are obtained by extracting the features of the radio gymnastics video, where class1 includes: 0-196 frames, 422-620 frames, class2 includes: 197-360 frames, 621-760 frames, and class0 includes: 361-421, 761-834 frames. It can be concluded that the category corresponding to the video should be: class1, class2, class0, class1, class2, class0 sequence. Finally, the clustering result shows that the action of the original video starts from frame 0 as the start frame of the first action, frame 421 as the end frame of the first action, and frames 422 and 834 correspond to the start frame and end frame of the second action.

[0035] The present invention adopts a self-supervised training method to train the basic model. This method does not need to rely on a large amount of labeled data and can automatically learn useful features from video data. Subsequently, in order to further improve the pertinence and generalization ability of feature extraction, the present invention introduces domain adversarial transfer learning technology. This technology enables the model to extract more discriminative features in different video domains, so as to better focus on the action behavior itself. Finally, the present invention uses clustering algorithms and loop nodes to efficiently locate the features obtained through self-supervision and adversarial transfer learning. This method is particularly suitable for action positioning in loop videos because it can effectively identify and locate repeated actions in videos, improving the continuity and stability of positioning. The present invention introduces smoothness constraints in the model training process to ensure that the extracted video features are continuous and stable in the time dimension, thereby providing a reliable basis for subsequent clustering analysis.

[0036] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0037] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0038] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for locating sequential actions in cyclic action videos based on sequential modeling, characterized by: The method comprises: Step S1: Collect cyclic action video data and preprocess the video to highlight the action features. The preprocessed video is classified and stored by action category; images are extracted frame by frame and organized and stored according to the original video name; the video is annotated at the same time, and the start and end time of each action is detailed and marked to ensure the accuracy and availability of the data; Step S2: Perform model design and self-supervised training, and perform self-supervised learning through contrast loss function to enhance the model's ability to learn the temporal features of video clips; Step S3: Train the feature extractor and add smoothness constraints to achieve smoothness of video features in time series and reduce positioning errors. At the same time, a domain adversarial transfer learning mechanism is incorporated to improve the generalization ability of the feature extractor between different domains and ensure the consistency and accuracy of features during the transfer learning process. Step S4: fine-tune the feature extractor using the labeled data set in step S1; Step S5: Select representative frame images from each video so as to extract the features of the video using the trained feature extractor; Step S6: identifying the action timing structure in the video by clustering the extracted features, and determining the start and end frames of the action; Step S7: by detecting the periodic patterns in the feature sequence, the recurring category sequences are identified and determined as cyclic actions, and for these periodic category sequences, the start and end boundaries of each cyclic action are determined.

2. The method for locating sequential actions in cyclic action videos based on sequential modeling according to claim 1, characterized in that: In step S2, a neural network model based on temporal contrastive learning is first designed. The model includes an encoder and a projection head. The encoder is used to extract feature vectors of video clips. The projection head maps the feature vectors to a low-dimensional space for contrastive learning. The specific steps are as follows: For video V, extract segments of different time periods from the same video to generate positive sample pairs, and extract time segments from different videos to generate negative sample pairs; The feature vectors of the positive sample pair and the negative sample pair are mapped to the low-dimensional space through the projection head and denoted as z i 、z j and z i 、z v , then calculate their similarity matrix, and use the cosine similarity formula to calculate the similarity between feature vectors. In the following formula, s ij Represents the similarity of positive samples, s iv Represents the negative sample similarity: The contrast loss function InfoNCE loss is defined to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The calculation formula is: in, Represents the loss value of the i-th sample and the j-th sample, 1 [k≠i] is an indicator function, which takes the value of 1 when k≠i, and τ is a temperature parameter. The similarity matrix is ​​optimized according to the contrast loss function. In each round of training, the model parameters are updated using the contrast loss function to ensure that the model can accurately learn the temporal characteristics of the video clips.

3. The method for locating sequential actions in cyclic action videos based on sequential modeling according to claim 1 or 2, characterized in that: In step S3, according to the total number of frames N of the video f and the clip length L c Calculate the interval between each frame And calculate the corresponding offset O i , ensuring that the video frames are fed into the network in order for training, the formula is: Where the subscript i represents the i-th offset, for all i = 0, 1, 2, ..., L c -1; By setting up a data loader, each video is selected from the dataset through the loader, and the frame sequence of each video is sent to the model for training in batches according to the time sequence of the video.

4. The method for locating sequential actions in cyclic action videos based on sequential modeling according to claim 3, characterized in that: In the training process of constructing the feature extractor, domain adversarial transfer learning technology and adding smoothness constraints are used; including first defining the source domain as the video data in the Something-Something V2 dataset, and the target domain as the cyclic action video data collected in step S1, and initializing a shared feature extractor to extract useful feature information from the two domains; then setting a domain classifier to accurately classify the extracted features into the source domain or the target domain.

5. The method for locating sequential actions in cyclic action videos based on sequential modeling according to claim 4, characterized in that: The specific steps of using the domain classifier to accurately classify the extracted features into the source domain or the target domain are as follows: First, features are extracted from the source and target domains and fed into the domain classifier; Secondly, the loss of the domain classifier is calculated to evaluate its ability to distinguish between the source domain and the target domain. The domain classification loss L domain The calculation formula is as follows: Among them, y i is the true domain label of the i-th sample, the source domain is 1 and the target domain is 0; is the probability that the i-th sample predicted by the domain classifier is the source domain; Again, calculate the adversarial loss L class , the formula is as follows: Where K is the number of categories, is the k-th category indicator function of the true label of the i-th sample, 1 represents the correct category and 0 represents other categories; is the probability that the i-th sample belongs to the k-th class predicted by the model; In order to ensure the smoothness of the feature extractor on the time series, a smoothness constraint is introduced to implement the smoothness loss L by imposing a difference penalty on the time series. smooth , the calculation formula is as follows: Among them, φ is the feature extractor, x i and x i+1 is the number of consecutive video frames, N is the total number of frames after video sampling; The total loss function is: L=L domain +λ class L class +λ smooth L smooth ; Among them, λ class and λ smooth is a weight coefficient used to balance the importance of different loss terms.

6. The method for locating sequential actions in cyclic action videos based on sequential modeling according to claim 1, characterized in that: In step S4, the specific steps of fine-tuning the trained feature extractor on the target domain using the data set collected and annotated in step S1 are as follows: First, for a given video V, assume that there are N different actions in the video, marked as a1, a2, a3, ..., a n , each action will appear multiple times in the video. For action a k , it will be in different time periods t1, t2, t3, ..., t n Appears, marked as: Due to the change of the lens in the video, the same action will present different pictures in different time periods. In order to capture the periodic consistency of the action, the action periodic constraint loss Loss is defined cycle , to ensure that the feature representation of the same action in different time periods is similar, assuming that the model extracts the features of each frame through the feature extractor ф, and the action a k In time period and The characteristics are and Then the action periodicity constraint loss can be expressed as:

7. The method for locating sequential actions in cyclic action videos based on sequential modeling according to claim 1, characterized in that: In step S5, the total number of frames of the video is first determined, and then the index of the frame to be extracted is calculated according to the selection strategy. Finally, the trained deep learning model is used to extract features of these selected frame images. These features will be used for subsequent clustering analysis to achieve the positioning of video timing actions.

8. The method for locating sequential actions in cyclic action videos based on sequential modeling according to claim 1, characterized in that: In step S6, the HMM clustering algorithm is used to achieve efficient recognition and positioning of video actions, which specifically includes the following steps: First, the features extracted from step S5 are preprocessed to ensure the quality and consistency of the data; According to the preprocessed features, the clustering algorithm is applied for grouping. The HMM clustering algorithm divides the data into different clusters by identifying the hidden states in the time series data. Each cluster corresponds to an action mode. The algorithm optimizes the HMM parameters by using the Baum-Welch algorithm, and then uses the forward-backward algorithm to calculate the hidden state probability at each time point. Finally, according to the hidden state, the time segments are assigned to the corresponding clusters, thereby achieving effective clustering of actions.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images