A method, device and medium for unconstrained video action monitoring and positioning

Abstract

The application provides a non-constrained video action monitoring and positioning method, device and medium, and relates to the field of video action positioning. The method comprises the following steps: obtaining a video set, performing video-level contrast learning; performing instance-level contrast learning, bringing video instances with the same pseudo label closer, and bringing video instances with different pseudo labels farther apart, to obtain a preliminary action positioning result; performing feature-level contrast learning, combining the preliminary action positioning result, and obtaining a class activation sequence; generating a video-level pseudo label of each video through action class mapping; positioning the action segment of each video through action proposal positioning, combining the class activation sequence and the video-level pseudo label; and inferring the positioned action segment to realize final action positioning. The technical scheme of the application does not require annotation information of a data set, thereby saving a large amount of manpower and material resources, and being not affected by the annotation information, and having better scalability.

Description

Technical Field

[0001] This application relates to the field of video motion localization, and in particular to a method, device and medium for motion monitoring and localization of unconstrained video. Background Technology

[0002] In recent years, with the explosive growth of the number of videos on the Internet and the improvement of computing power of computer hardware, video motion localization has gradually become the focus of attention of researchers and has been widely used in fields such as event detection, video summarization, intelligent security and Internet video retrieval.

[0003] Existing video action localization methods and techniques can be mainly divided into fully supervised methods and weakly supervised methods. Fully supervised methods include S-CNN, SSN, R-C3D, GTAN, and DPP, while weakly supervised methods include W-TALC, 3C-Net, HAM-Net, and MAAN. These existing video action localization methods have achieved good results on supervised datasets, but they still face the following problems: 1) High training cost. Both fully supervised and weakly supervised methods rely on labeled training information, and the labeling of this information requires a significant amount of human resources, resulting in high training costs; 2) Limited scalability. Fully supervised and weakly supervised methods typically rely on labeled information for training, but because the number of actions covered by the labeled information is limited, many actions not present in real-world scenarios in unconstrained videos without labeled information are difficult to identify, thus limiting the application of these methods in real-world scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a method, device, and medium for motion monitoring and localization of unconstrained video, in order to solve the problem that existing video motion localization methods require a large amount of annotation information and the annotation information cannot cover all actions.

[0005] The above-mentioned objective of this application is achieved through the following technical solution:

[0006] S1: The acquired video set is used for video-level comparative learning;

[0007] S2: Perform instance-level comparative learning, bringing video instances with the same pseudo-label closer together, while moving video instances with different pseudo-labels further apart, to obtain preliminary action localization results;

[0008] S3: Perform feature-level comparative learning and combine it with the preliminary action localization results to obtain the class activation sequence;

[0009] S4: Generate video-level pseudo-labels for each video through action category mapping; locate action segments in each video by combining action proposal localization with class activation sequences and video-level pseudo-labels;

[0010] S5: Infer the localized action segments to achieve final action localization.

[0011] Optionally, the specific steps of the video-level contrastive learning include:

[0012] S11: Extract enhanced horizontal and vertical feature pairs from the acquired video set;

[0013] The set of videos in the training batch is represented as Where i is the video set The index, where N represents the number of videos in the training batch;

[0014] V for each video i Divide into T non-overlapping segments, denoted as Where t is the segment index in the video;

[0015] V for each video i Divide it into two equal parts, front and back. and

[0016]

[0017] Using a pre-trained two-stream dilated 3D convolutional neural network as the backbone network, extracting... and Spatiotemporal characteristics;

[0018] If the backbone network is represented as f(·), then the spatiotemporal features are represented as and Dimensions Where d represents the feature size of each segment;

[0019] Using two independent perceptrons g h (·) and g v (·), extract spatiotemporal features and The horizontal and vertical feature pairs include:

[0020] Use g h (·) Extract the enhanced level feature pairs, denoted as: and Use g v (·) Extract the enhanced vertical feature pairs, denoted as: and g h (·) and gv (·) is a two-layer nonlinear multilayer perceptron, and C is the number of predefined video action categories;

[0021] S12: The video-level contrast learning includes: horizontal contrast learning and vertical contrast learning; horizontal contrast learning and vertical contrast learning are performed in parallel;

[0022] Comparative learning of execution levels, specifically including:

[0023] Based on the horizontal feature pair enhancement, two horizontal feature matrices are obtained from the video set of the training batch, denoted as follows: and Where N is the number of videos in the batch;

[0024] Horizontal feature vectors from the same video are considered positive contrast samples, while horizontal feature vectors from different videos are considered negative contrast samples.

[0025] Positive contrast samples come from the same video, denoted as and The remaining 2N-2 horizontal feature vectors in the two horizontal feature matrices are negative contrast samples;

[0026] The horizontal contrastive learning loss function is shown below:

[0027]

[0028] Where τ h It is the horizontal temperature hyperparameter, and d(·) represents the cosine similarity;

[0029] Performing vertical contrastive learning specifically includes:

[0030] Vertical feature pairs and Combined into two vertical feature matrices, represented as and In the two vertical feature matrices, the number of columns c corresponds to the predefined number of clusters C;

[0031] Sampling samples with the same column c as vertically contrasting samples are vertical feature vectors. and Where c represents a cluster; the remaining 2C-2 vertical feature vectors in the training batch are used as vertical negative contrast samples;

[0032] Both vertical positive contrast samples and vertical negative contrast samples learn cluster-level representations with different action categories through vertical contrast learning;

[0033] The vertical contrast learning loss function is shown below:

[0034]

[0035] Where τ v It is the vertical temperature hyperparameter, and d(·) represents the cosine similarity;

[0036] The overall objective function L for video-level contrastive learning VCL Including horizontal contrastive learning loss L H and vertical contrast learning loss L V As shown below:

[0037] L VCL =L H +L V .

[0038] Optionally, step S2 includes:

[0039] Given the spatiotemporal features of the i-th video As input;

[0040] Using temporal convolution operations g with ReLU activation function embed (·), X i Embedded into a task-specific feature space, the formula is as follows:

[0041] F i =g embed (X i )

[0042] in These are task-specific embedding features, where d and T represent the feature size of each segment and the number of non-overlapping small segments in the video, respectively.

[0043] In a training batch of N videos, video V will be compared with video V. i Video features with the same video-level pseudo-labels are considered positive contrast samples and will be compared with video V. i Other features with different video-level pseudo-labels are considered as negative contrast samples;

[0044] The set of positive contrast samples is defined as F. A The set of negative contrast samples is defined as F. B ;

[0045] Instance-level contrastive learning loss function L ICL Represented as:

[0046]

[0047] Where τ in It is a hyperparameter for instance-level contrastive learning; exp() represents the natural exponential function.

[0048] Optionally, step S3 includes:

[0049] Active features of action can be extracted from class activation sequences using feature separation. Negative characteristics of action Positive background characteristics Negative characteristics of the background

[0050] Using action-background separation loss L abs Improve the positive characteristics of movement and background positive characteristics The distinguishability is as follows:

[0051]

[0052] Where, ‖·‖ is a norm norm function, and q is a predefined maximum characteristic magnitude;

[0053] According to negative contrast loss L con The distance between a feature and its positive and negative samples is calculated as follows:

[0054]

[0055] Where τ is a hyperparameter in contrastive learning; and They are and Transpose of; and These are the t-th elements of the background positive features and the action positive features, respectively; k p yes and

[0056] The final feature-level contrastive learning loss is as follows:

[0057] L FCL =L abs +L con .

[0058] Optionally, step S4 includes:

[0059] S41: Action category mapping, as detailed below:

[0060] Through video-level comparative learning, a well-trained perceptron g is obtained. v (·);

[0061] For video V i ,Will As a sensor g v Input (·) to obtain Z i,vView as video V i Scores on C clusters;

[0062] Select the cluster with the highest score, and then select video V. i They are assigned to the highest-scoring clusters, resulting in C clusters.

[0063] Map the C clusters to action categories, specifically including:

[0064] Map each cluster to the most frequently occurring action category and assign a category label to each cluster;

[0065] Then map a cluster to multiple actions;

[0066] Suppose that in cluster z, C a It is the most frequently occurring action category, and N appears. a For those appearing more than once in cluster z; Any action category C b Map cluster z to action C b ;

[0067] Video V i Video-level pseudo-tags It means that y i,c ∈{0,1}, where 0 represents a negative example, i.e., no action category exists, and 1 represents a positive example, i.e., the action category exists;

[0068] S42: Utilizing the mapped action category y i,c Based on the class activation sequence, robust and effective action proposal localization is performed, as follows:

[0069] Aggregating class activation sequences, including: using class activation sequences For each action category y i,c The top-k scores of the class activation sequences are aggregated and the average is calculated to obtain V for each video. i Each action category a i；c Category score;

[0070] The aggregation process of class activation sequences is shown below:

[0071]

[0072] in, This represents the action score for each video, and the classification score for each action category. i；c This represents the predicted action score for the action category of the i-th video;

[0073] Using the multi-label cross-entropy loss function L cls To optimize classifier g clsTo predict multiple action categories, classifier g cls The loss function is defined as follows:

[0074]

[0075] in, Let y represent the predicted classification probability of the c-th category in the i-th video. i,c These are video-level pseudo-tags generated through clustering.

[0076] Optionally, step S5 includes:

[0077] Given a video A, compute the class activation sequence of video A and predict the action probability p. i；c ;

[0078] Set threshold θ class For p i；c Perform filtering and retain those that satisfy p i；c >θ class All action categories c;

[0079] For the retained action categories, use a threshold θ act Thresholding is applied to the class activation sequence to obtain candidate action segments;

[0080] A non-maximum suppression algorithm is applied to each candidate action segment to eliminate duplicate candidate action segments and obtain the final localization result.

[0081] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform a non-constrained video motion monitoring and positioning method.

[0082] A computer-readable storage medium storing instructions that, when executed, perform a motion monitoring and localization method for unconstrained video.

[0083] The beneficial effects of the technical solution provided in this application are:

[0084] 1. To address the issue of poor clustering results in video data, a video-level contrastive learning component is introduced. This component can cluster action categories in unlabeled videos, thereby obtaining better pseudo-labels and providing more accurate guidance for subsequent localization.

[0085] 2. To address the issue of inaccurate action background boundary localization, feature-level contrastive learning was employed, and a feature separation module and a separation-contrast learning method were introduced to refine the boundary features, thereby improving the recognition accuracy of the boundary features and achieving better localization results.

[0086] 3. An instance-level contrastive learning approach is adopted to further optimize action localization. This approach encourages video features with the same pseudo-label to be grouped together, while features with different pseudo-labels are grouped apart. This allows for better differentiation of different action features and enables coarse action localization. By learning the similarities and differences between video features, the accuracy and discriminative power of action localization are improved. Attached Figure Description

[0087] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0088] Figure 1 This is a flowchart illustrating the algorithm model of the motion detection and localization method for unconstrained video in this application embodiment;

[0089] Figure 2 This is a flowchart of the video-level comparative learning of the motion detection and localization method for unconstrained videos in the embodiments of this application;

[0090] Figure 3 This is a schematic diagram of the electronic device structure of the motion monitoring and positioning method for unconstrained video in the embodiments of this application. Detailed Implementation

[0091] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0092] The embodiments of this application provide a method for motion monitoring and localization of unconstrained video.

[0093] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the algorithm model of an unconstrained video motion detection and localization method according to an embodiment of this application, including:

[0094] S1: The acquired video set is used for video-level comparative learning;

[0095] S2: Perform instance-level comparative learning, bringing video instances with the same pseudo-label closer together, while moving video instances with different pseudo-labels further apart, to obtain preliminary action localization results;

[0096] S3: Perform feature-level comparative learning and combine it with the preliminary action localization results to obtain the class activation sequence;

[0097] S4: Generate video-level pseudo-labels for each video through action category mapping; locate action segments in each video by combining action proposal localization with class activation sequences and video-level pseudo-labels;

[0098] S5: Infer the localized action segments to achieve final action localization.

[0099] The specific steps of the video-level contrastive learning include:

[0100] S11: Extract enhanced horizontal and vertical feature pairs from the acquired video set;

[0101] The set of videos in the training batch is represented as Where i is the video set The index, where N represents the number of videos in the training batch;

[0102] V for each video i Divide into T non-overlapping segments, denoted as Where t is the segment index in the video;

[0103] V for each video i Divide it into two equal parts, front and back. and

[0104]

[0105] Using a pre-trained two-stream dilated 3D convolutional neural network as the backbone network, extracting... and Spatiotemporal characteristics;

[0106] If the backbone network is represented as f(·), then the spatiotemporal features are represented as and Dimensions Where d represents the feature size of each segment;

[0107] Using two independent perceptrons g h (·) and g v (·), extract spatiotemporal features and The horizontal and vertical feature pairs include:

[0108] Use g h (·) Extract the enhanced level feature pairs, denoted as: and Use g v (·) Extract the enhanced vertical feature pairs, denoted as: and g h (·) and g v(·) is a two-layer nonlinear multilayer perceptron, and C is the number of predefined video action categories;

[0109] S12: The video-level contrast learning includes: horizontal contrast learning and vertical contrast learning; horizontal contrast learning and vertical contrast learning are performed in parallel;

[0110] Comparative learning of execution levels, specifically including:

[0111] Based on the horizontal feature pair enhancement, two horizontal feature matrices are obtained from the video set of the training batch, denoted as follows: and Where N is the number of videos in the batch;

[0112] Horizontal feature vectors from the same video are considered positive contrast samples, while horizontal feature vectors from different videos are considered negative contrast samples.

[0113] Positive contrast samples come from the same video, denoted as and The remaining 2N-2 horizontal feature vectors in the two horizontal feature matrices are negative contrast samples;

[0114] The horizontal contrastive learning loss function is shown below:

[0115]

[0116] Where τ h It is the horizontal temperature hyperparameter, and d(·) represents the cosine similarity;

[0117] Performing vertical contrastive learning specifically includes:

[0118] Vertical feature pairs and Combined into two vertical feature matrices, represented as and In the two vertical feature matrices, the number of columns c corresponds to the predefined number of clusters C;

[0119] Sampling samples with the same column c as vertically contrasting samples are vertical feature vectors. and Where c represents a cluster; the remaining 2C-2 vertical feature vectors in the training batch are used as vertical negative contrast samples;

[0120] Both vertical positive contrast samples and vertical negative contrast samples learn cluster-level representations with different action categories through vertical contrast learning;

[0121] The vertical contrast learning loss function is shown below:

[0122]

[0123] Where τ v It is the vertical temperature hyperparameter, and d(·) represents the cosine similarity;

[0124] The overall objective function L for video-level contrastive learning VCL Including horizontal contrastive learning loss L H and vertical contrast learning loss L V As shown below:

[0125] L VCL =L H +L V .

[0126] Specifically, horizontal contrastive learning aims to maximize the similarity of positive contrasting samples while minimizing the similarity of negative contrasting samples. The horizontal contrastive learning loss function encourages high similarity for positive contrasting samples and low similarity for negative contrasting samples, thereby obtaining rich video-level information. In vertical contrastive learning, the goal is to cluster videos into different action clusters to learn common clustering-level features. Similar to clustering tasks, the number of clusters in the unlabeled dataset also needs to be predefined to facilitate the clustering process. Vertical contrastive learning achieves this goal by maximizing the cluster similarity of positive contrasting samples and minimizing the cluster similarity of negative contrasting samples.

[0127] Specifically, each column feature vector in the two vertical feature matrices represents action cluster information used to generate pseudo-labels for action categories for each video. The vertical contrastive learning loss function encourages positive contrasting samples to have high similarity and negative contrasting samples to have low similarity, thereby obtaining rich cluster information. Ultimately, video clustering can also be obtained through vertical contrastive learning.

[0128] Specifically, to generate video pseudo-labels to guide action localization, this method introduces video-level contrastive learning. The purpose of video-level contrastive learning is to learn video-level features and cluster-level features. Video-level features contain action-specific information for each video, while cluster-level features contain common action information. This method designs two contrastive learning mechanisms: horizontal contrastive learning and vertical contrastive learning. Horizontal contrastive learning is used to learn video-level features, while vertical contrastive learning is used to learn common cluster-level features that can be used to generate action pseudo-labels. Figure 2 The video-level comparative learning process is shown.

[0129] Specifically, g h (·) and g v (·) is a two-layer nonlinear multilayer perceptron (MLP).

[0130] Step S2 includes:

[0131] Given the spatiotemporal features of the i-th video As input;

[0132] Using temporal convolution operations g with ReLU activation function embed (·), X i Embedded into a task-specific feature space, the formula is as follows:

[0133] F i =g embed (X i )

[0134] in These are task-specific embedding features, where d and T represent the feature size of each segment and the number of non-overlapping small segments in the video, respectively.

[0135] In a training batch of N videos, video V will be compared with video V. i Video features with the same video-level pseudo-labels are considered positive contrast samples and will be compared with video V. i Other features with different video-level pseudo-labels are considered as negative contrast samples;

[0136] The set of positive contrast samples is defined as F. A The set of negative contrast samples is defined as F. B ;

[0137] Instance-level contrastive learning loss function L ICL Represented as:

[0138]

[0139] Where τ in It is a hyperparameter for instance-level contrastive learning; exp() represents the natural exponential function.

[0140] Specifically, after acquiring video-level features and their corresponding pseudo-labels through video-level contrastive learning, this invention employs instance-level contrastive learning to further optimize action localization. This method encourages video features with the same pseudo-label to be grouped together, while distancing features with different pseudo-labels, thereby better distinguishing features of different actions and achieving coarse action localization. This method improves the accuracy and discriminative power of action localization by learning the similarities and differences between video features. Through instance-level contrastive learning, the commonalities and variations of actions can be better captured, resulting in more precise action localization. By optimizing the instance-level contrastive learning loss, it encourages similar action features to have high similarity scores while reducing the similarity scores between action-related and action-independent features. While this process helps achieve coarse action localization, it is still difficult to obtain fine-grained action localization, such as specific action categories within video segments.

[0141] Step S3 includes:

[0142] Active features of action can be extracted from class activation sequences using feature separation. Negative characteristics of action Positive background characteristics Negative characteristics of the background

[0143] Using action-background separation loss L abs Improve the positive characteristics of movement and background positive characteristics The distinguishability is as follows:

[0144]

[0145] Where, ‖·‖ is a norm norm function, and q is a predefined maximum characteristic magnitude;

[0146] According to negative contrast loss L con The distance between a feature and its positive and negative samples is calculated as follows:

[0147]

[0148] Where τ is a hyperparameter in contrastive learning; and They are and Transpose of; and These are the t-th elements of the background positive features and the action positive features, respectively; k p yes and

[0149] The final feature-level contrastive learning loss is as follows:

[0150] L FCL =L abs +L con .

[0151] Specifically, through this learning process, positive action features and positive background features can be effectively separated in the feature space, thus better understanding the subsequent negative features for comparative learning. Next, to further eliminate potentially difficult-to-locate information in action and background features, the feature separation module introduces comparative learning for negative action and negative background features.

[0152] Specifically, to achieve more refined action localization, this method further introduces feature-level contrastive learning, focusing on learning fine-grained action-background boundary features within each video. However, localizing action and background boundary features presents certain challenges because they may contain similar information, introducing interference or confusion into the learning process. To address this issue, this invention defines four categories of features: positive action features, negative action features, positive background features, and negative background features. Regions located at the center of action or background segments are called positive action or positive background regions, and their features are referred to as positive action features or positive background features. These features are relatively obvious and easily distinguishable. Regions located at the boundaries of action or background segments are called negative action or negative background regions, and their features are referred to as negative action features or negative background features. Because these regions transition between action and background, they are prone to confusion and have poor distinguishability.

[0153] Specifically, by learning through fine-grained feature-level comparisons, the localization of boundary regions will be more accurate.

[0154] Step S4 includes:

[0155] S41: Action category mapping, as detailed below:

[0156] Through video-level comparative learning, a well-trained perceptron g is obtained. v (·);

[0157] For video V i ,Will As a sensor g v Input (·) to obtain Z i,v View as video V i Scores on C clusters;

[0158] Select the cluster with the highest score, and then select video V. i They are assigned to the highest-scoring clusters, resulting in C clusters.

[0159] Map the C clusters to action categories, specifically including:

[0160] Map each cluster to the most frequently occurring action category and assign a category label to each cluster;

[0161] Then map a cluster to multiple actions;

[0162] Suppose that in cluster z, C a It is the most frequently occurring action category, and N appears. a For those appearing more than once in cluster z; Any action category C b Map cluster z to action C b ;

[0163] Video V i Video-level pseudo-tags It means that y i,c ∈{0,1}, where 0 represents a negative example, i.e., no action category exists, and 1 represents a positive example, i.e., the action category exists;

[0164] S42: Utilizing the mapped action category y i,c Based on the class activation sequence, robust and effective action proposal localization is performed, as follows:

[0165] Aggregating class activation sequences, including: using class activation sequences For each action category y i,c The top-k scores of the class activation sequences are aggregated and the average is calculated to obtain V for each video. i Each action category a i；c Category score;

[0166] The aggregation process of class activation sequences is shown below:

[0167]

[0168] in, This represents the action score for each video, and the classification score for each action category. i；c This represents the predicted action score for the action category of the i-th video;

[0169] Using the multi-label cross-entropy loss function L cls To optimize classifier g cls To predict multiple action categories, classifier g cls The loss function is defined as follows:

[0170]

[0171] in, Let y represent the predicted classification probability of the c-th category in the i-th video. i,c These are video-level pseudo-tags generated through clustering.

[0172] Specifically, a cluster is mapped to multiple action categories to cover the different actions that may exist in the video. Through this mapping process, the correct category label can be assigned to each video, allowing for fair comparison and evaluation with other supervision methods.

[0173] Specifically, since video motion localization involves motion category classification, we need to further classify motion segments into the correct motion categories to facilitate the final fine-grained localization.

[0174] Step S5 includes:

[0175] Given a video A, compute the class activation sequence of video A and predict the action probability p. i；c ;

[0176] Set threshold θ class For p i；c Perform filtering and retain those that satisfy p i；c >θ class All action categories c;

[0177] For the retained action categories, use a threshold θ act Thresholding is applied to the class activation sequence to obtain candidate action segments;

[0178] A non-maximum suppression algorithm is applied to each candidate action segment to eliminate duplicate candidate action segments and obtain the final localization result.

[0179] This application also discloses an electronic device. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0180] The communication bus 502 is used to enable communication between these components.

[0181] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0182] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0183] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0184] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for motion detection and localization in unconstrained video, characterized in that, The method includes the following steps: S1: The acquired video set is used for video-level comparative learning; The specific steps of the video-level contrastive learning include: S11: Extract enhanced horizontal and vertical feature pairs from the acquired video set; The set of videos in the training batch is represented as ,in It is a collection of videos index, This indicates the number of videos in the training batch; Each video Divided into A non-overlapping small segment, represented as ,in It is an index of video clips; Each video Divide it into two equal parts, front and back. and ; Using a pre-trained two-stream dilated 3D convolutional neural network as the backbone network, extracting... and Spatiotemporal characteristics; Represent the backbone network as Then the spatiotemporal characteristics are expressed as and , dimension ,in Indicates the feature size of each segment; Using two independent perceptrons and Extracting spatiotemporal features and The horizontal and vertical feature pairs include: use Extract enhanced horizontal feature pairs, denoted as and ;use Extract the enhanced vertical feature pairs, denoted as: and ; and It is a two-layer nonlinear multilayer perceptron. It is the number of predefined video action categories; S12: The video-level contrast learning includes: horizontal contrast learning and vertical contrast learning; horizontal contrast learning and vertical contrast learning are performed in parallel; Comparative learning of execution levels, specifically including: Based on the horizontal feature pair enhancement, two horizontal feature matrices are obtained from the video set of the training batch, denoted as follows: and ,in This refers to the number of videos in the batch; Horizontal feature vectors from the same video are considered positive contrast samples, while horizontal feature vectors from different videos are considered negative contrast samples. Positive contrast samples come from the same video, denoted as and The rest in the two horizontal feature matrices Each horizontal feature vector is a negative contrast sample; The level contrast learning loss function is shown below: in It is a horizontal temperature overparameter. Indicates cosine similarity; Performing vertical contrastive learning specifically includes: Vertical feature pairs and Combined into two vertical feature matrices, represented as and In the two vertical feature matrices, the number of columns Corresponding to the predefined number of clusters ; Samples with the same column Vertical feature vectors as vertical positive contrast samples and ,in Indicates a cluster; the remaining clusters in the training batch. Each vertical feature vector is used as a vertical negative contrast sample; Both vertical positive contrast samples and vertical negative contrast samples learn cluster-level representations with different action categories through vertical contrast learning; The vertical contrast learning loss function is shown below: in It is a vertical temperature hyperparameter. Indicates cosine similarity; The overall objective function of video-level contrastive learning Including horizontal contrastive learning loss and vertical contrast learning loss As shown below: S2: Perform instance-level comparative learning, bringing video instances with the same pseudo-label closer together, while moving video instances with different pseudo-labels further apart, to obtain preliminary action localization results; S3: Perform feature-level comparative learning and combine it with the preliminary action localization results to obtain the class activation sequence; S4: Generate video-level pseudo-labels for each video through action category mapping; locate action segments in each video by combining action proposal localization with class activation sequences and video-level pseudo-labels; S5: Infer the localized action segments to achieve final action localization.

2. The method for motion detection and localization of unconstrained video as described in claim 1, characterized in that, Step S2 includes: Given the first Spatiotemporal characteristics of a video As input; Temporal convolution operations with ReLU activation function ,Will Embedded into a task-specific feature space, the formula is as follows: in These are task-specific embedding features. and These represent the feature size of each segment and the number of non-overlapping small segments in the video, respectively. In a training batch In each video, there will be videos Video features with the same video-level pseudo-labels are considered positive contrast samples and will be compared with the video. Other features with pseudo-labels of different video levels are considered as negative contrast samples; Define the set of positive contrast samples as The set of negative contrast samples is defined as ; Instance-level contrastive learning loss function Represented as: in It is a hyperparameter for instance-level contrastive learning; This represents the natural exponential function.

3. The method for motion detection and localization of unconstrained video as described in claim 2, characterized in that, Step S3 includes: Active features of action can be extracted from class activation sequences using feature separation. Negative characteristics of action Positive characteristics of the background Negative characteristics of the background ; Using action-background separation loss Improve the positive characteristics of movement and background positive characteristics The distinguishability is as follows: in, It is a norm norm function. It is the predefined maximum characteristic amplitude; Based on negative comparison loss The distance between a feature and its positive and negative samples is calculated as follows: in, These are hyperparameters in contrastive learning; and They are and transpose; and The positive features of the background and the positive features of the action are respectively the first One element; It is a feature set and The sum of its elements; The final feature-level contrastive learning loss is as follows: 。 4. The motion detection and localization method for unconstrained video as described in claim 3, characterized in that, Step S4 includes: S41: Action category mapping, as detailed below: A well-trained perceptron is obtained through video-level comparative learning. ; For video ,Will As a sensor Input, obtain , Watch as video exist Scores on each cluster; Select the cluster with the highest score and upload the video. Assign to the highest scorer and receive A cluster; Will Each cluster is mapped to an action category, specifically including: Map each cluster to the most frequently occurring action category and assign a category label to each cluster; Then map a cluster to multiple actions; Assuming in cluster middle, It is the most frequently occurring action category, and it has appeared Next; for in cluster The number of times it appears is greater than Any action category , will cluster Mapping to Action ; video Video-level pseudo-tags It means that among them , This represents a negative example, i.e., an action category that does not exist. This indicates a positive example, meaning that the action category exists. S42: Utilizing Mapped Action Categories Based on the class activation sequence, robust and effective action proposal localization is performed, as follows: Aggregating class activation sequences, including: using class activation sequences For each action category top-class activation sequences The scores are aggregated and averaged to obtain the score for each video. Each action category Category score; The aggregation process of class activation sequences is shown below: in, This represents the motion score for each video and the classification score for each motion category. Indicates the first The predicted motion score for each video's motion category; Using the multi-label cross-entropy loss function To optimize the classifier A classifier that makes predictions for multiple action categories. The loss function is defined as follows: in, Indicates the first The first video The predicted classification probability of each category. These are video-level pseudo-tags generated through clustering.

5. The motion detection and localization method for unconstrained video as described in claim 4, characterized in that, Step S5 includes: Given a video A, compute the class activation sequence of video A and predict the action probability. ; Set threshold right Filter and retain those that meet the requirements. All action categories ; For the retained action categories, use a threshold. Thresholding is applied to the class activation sequence to obtain candidate action segments; A non-maximum suppression algorithm is applied to each candidate action segment to eliminate duplicate candidate action segments and obtain the final localization result.

6. An electronic device, characterized in that, The device includes a processor (501), a memory (505), a user interface (503), and a network interface (504). The memory (505) is used to store instructions. The user interface (503) and the network interface (504) are used to communicate with other devices. The processor (501) is used to execute the instructions stored in the memory (505) to cause the electronic device to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the steps of the method as described in any one of claims 1-5.

Images