Semantic query learning-based point supervision video action positioning method

Abstract

The invention relates to a semantic query learning-based point supervision video action positioning method, which comprises the following steps of: S1, giving an unedited video which comprises a plurality of action instances; s2, extracting spatio-temporal features by using a feature extractor; s3, semantic query learning; s4, carrying out integrity learning; and S5, positioning the position interval of the action according to the predicted action instances, and filtering redundant predicted action instances through a non-maximum suppression algorithm to obtain a final point supervision video action positioning result. Compared with the prior art, the method has the advantage of high accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and video understanding, and in particular to a point-supervised video action localization method based on semantic query learning. Background Art

[0002] Temporal Action Localization (TAL) is a fundamental task in video understanding, which aims to detect the temporal boundaries of actions in uncut videos. However, traditional methods rely on a large amount of instance-level annotated data, which is often difficult to obtain in real applications.

[0003] Point-Supervised Temporal Action Localization (PS-TAL) is an emerging paradigm that significantly reduces the annotation cost by only annotating the key points of the action. However, how to learn the completeness and discriminability of actions with only point supervision remains a challenge. Existing methods have two main defects:

[0004] 1) Most existing PS-TAL methods cannot learn the completeness of actions without instance-level annotations, which results in the predicted action regions being often fragmented. Since these methods usually rely only on point annotations, it is difficult to capture the global context and completeness of actions, so the predicted boundaries are often not accurate enough.

[0005] 2) The semantic information of action clips is crucial for detecting complete actions. Action clips with similar representations should be considered as part of the same action category. However, existing methods often fail to fully utilize this semantic information to enhance the distinguishability of clip-level features, limiting the ability to detect complete actions. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a more accurate point-supervised video action localization method based on semantic query learning.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A point-supervised video action localization method based on semantic query learning, the method comprising:

[0009] Step S1: Given an uncut video V, which includes multiple action instances, each action instance is provided with a start and end timestamp, and there is a random timing point and an action category label corresponding to the timing point;

[0010] Step S2, extracting spatiotemporal features X from the uncut video V using a feature extractor;

[0011] Step S3, semantic query learning: using a cross-attention mechanism based on a dynamic semantic query vector to propagate semantic information between segment-level representations of the spatiotemporal feature X to generate a semantically enhanced spatiotemporal feature X′, classifying and convolving the semantically enhanced spatiotemporal feature X′, respectively, and fusing the generated first activation sequence and the first attention score sequence to obtain a first weighted category activation sequence P; wherein each query in the dynamic semantic query vector corresponds to an action of a specific category and is dynamically updated according to the video context;

[0012] Step S4, integrity learning: input the semantic enhanced spatiotemporal feature X′ into the second activation sequence and the second attention score sequence generated by the fully connected layer, fuse them to obtain the second weighted category activation sequence P′, and use the integrity learning mechanism based on the integrity assessment head and the boundary refinement head to obtain the predicted action instance;

[0013] Step S5: locate the position interval of the action in the uncut video V according to the predicted action instance, and filter the redundant predicted action instances through the non-maximum suppression algorithm to obtain the final point-supervised video action positioning result.

[0014] Preferably, the step S1 specifically comprises:

[0015] Given an uncut video V, which contains M action instances and Respectively represent the mth action instance A m The start and end timestamps of

[0016] For each action instance, there is a random timing point p i And the action category label at this timing point Where C is the number of action categories. If the time point p i The corresponding action category is c, then The corresponding values ​​in the remaining dimensions are 0.

[0017] Preferably, the feature extractor in step S2 is an I3D feature extractor.

[0018] Preferably, step S3 includes the following sub-steps:

[0019] Step S31: construct a set of query vectors G∈R that express the semantic information of a specific action category C×D , where C is the number of action categories and D is the dimension of spatiotemporal features;

[0020] Step S32: Dynamically update the query vector G∈R through the cross-attention mechanism C×D, generate a dynamic semantic query vector G′ that can perceive the video content, and based on the dynamic semantic query vector G′, enhance the discriminability of the spatiotemporal feature X through the cross attention mechanism to obtain the enhanced spatiotemporal feature E∈R C×D :

[0021] Step S33, inputting the enhanced spatiotemporal feature E into a fully connected layer, a linear rectification layer and a fully connected layer in sequence to generate a first feature O, combining the first feature O with the enhanced spatiotemporal feature E and inputting them into a regularization layer to obtain a semantically enhanced spatiotemporal feature X′;

[0022] Step S34: Input the semantically enhanced spatiotemporal feature X′ into a segment-level classifier to generate a category-aware first activation sequence S∈R T×C , input to a convolutional layer to generate a class-independent first attention score sequence A∈R T×1 , where each value in A represents the confidence of the action to which the corresponding video frame belongs;

[0023] Step S35: Use element-wise product to fuse the first activation sequence S and the first attention score sequence A to obtain the first weighted category activation sequence P∈R C×T .

[0024] Preferably, the step S32 is specifically:

[0025] Dynamically update the query vector G∈R via the criss-cross attention mechanism C×D Generate a dynamic semantic query vector G′ that can perceive the video content, expressed as:

[0026]

[0027] Where: Q = GW q , K = XW k , V = XW v , W q ∈R D×D , W k ∈R D×D and W v ∈R D×D is the learnable parameter matrix; d k is the dimension of the feature; U∈R C×T is the attention mask matrix, which is used to aggregate the context information of the semantic part, and its expression is:

[0028]

[0029] Where: B∈R C×TIt is a binary matrix whose element value is determined by the probability confidence of each frame belonging to the action. If the confidence is greater than the set value, the corresponding value is set to 1, otherwise it is set to 0;

[0030] Based on the dynamic semantic query vector G′, the spatial-temporal feature X is discriminated and enhanced through the cross-attention mechanism to obtain the enhanced spatial-temporal feature E∈R C×D , the expression is:

[0031]

[0032] Where: Q′=XW′ q , K′=G′W′ k , V′=G′W′ v , W′ q ∈R D×D , W′ k ∈R D×D and W′ v ∈R D×D is the learnable parameter matrix; U T is the transpose of the attention mask matrix U.

[0033] Preferably, the step S3 further includes:

[0034] Combining the prior information provided by the point annotation and the category-independent first attention score sequence A, pseudo-action instances are selected As positive samples and pseudo background instances For a segment near a given point annotation, when the category-independent attention score value is greater than a preset threshold, the segment is considered a pseudo action instance; for the area between two consecutive point annotations, the segment with the lowest attention score or an attention score below a given threshold is considered a pseudo background instance.

[0035] Preferably, step S4 includes the following sub-steps:

[0036] Step S41: for each action category, select an action category in the first weighted category activation sequence P whose activation value is greater than a first preset threshold value θ p After removing the area without the annotation point, the remaining area is used as the candidate positive sample; for the first attention score sequence A that is irrelevant to the category, the activation value is less than the second preset threshold θ a The area is taken as a candidate negative sample;

[0037] Step S42: input the semantically enhanced spatiotemporal features X′ into the fully connected layer to generate a category-aware second activation sequence S′ and a category-independent second attention score sequence A′, and fuse them by element-wise product to obtain a second weighted category activation sequence P′;

[0038] Step S43: for each action category, select the action category with an activation value greater than the first preset threshold value θ in the second weighted category activation sequence P′. p , and the remaining areas are taken as candidate positive samples; for the second attention score sequence A′ that is irrelevant to the category, select the area with an activation value less than the second preset threshold θ a The area is taken as a candidate negative sample;

[0039] Step S44: Use the integrity assessment head to predict the action integrity score using boundary-sensitive regional features The boundary refinement head is used to perform boundary regression processing to obtain the start timestamp prediction value and the end timestamp prediction value Get predicted action instance in Indicates the confidence that the candidate region contains action c.

[0040] Preferably, the step S44 specifically includes:

[0041] Given a candidate region, its boundaries are expanded through hyperparameters to generate three regions: left, middle, and right. The left features are extracted from the three regions using region alignment and maximum pooling operations. Central Features and the right feature The subscript i corresponds to the number of the candidate region;

[0042] The completeness assessment head is used to predict the action completeness score using boundary-sensitive regional features. For the candidate region K i , its left feature With central features The difference and center features And the right side features With central features After the difference cascade operation, it is sent to a regularization layer, a fully connected layer, a linear rectification layer and an activation layer in turn to predict the completeness score of the action Among them, the instance-level loss function is used for supervised training;

[0043] The boundary refinement head is used to perform boundary regression processing to obtain the start timestamp prediction value and the end timestamp prediction value For the positive sample candidate region K i , according to its left feature and central features Predict the offset value Δs of the action start boundary, based on its central feature and the right feature Predict the offset value Δe of the action end boundary, and calculate the start timestamp prediction value of the video frame based on the offset value Δs of the action start boundary and the offset value Δe of the action end boundary and the end timestamp prediction value

[0044] Preferably, in step S44, supervised training is performed on the integrity assessment head using an instance-level loss function, and the loss function expression is:

[0045]

[0046] Where: N p and N n Represent the number of positive and negative samples respectively; SmoothL1 is the smoothed one-norm loss function, the true value u i It is calculated by calculating the intersection-over-union ratio of the candidate region and its most matching pseudo-annotated instance;

[0047] The refinement loss function is used to supervise the training of the boundary refinement head. The loss function expression is:

[0048]

[0049] Where: and They represent the completeness score of the original action instance and the updated action completeness score respectively.

[0050] Preferably, the confidence level of the candidate region in step S44 that action c is included is It is obtained by considering the confidence of the candidate region interval and the adjacent candidate regions at the same time. The specific calculation expression is:

[0051]

[0052] Where: is the confidence that the candidate region contains action c, is the confidence that the adjacent candidate region includes action c; P′[t,c] is the second category activation sequence corresponding to action c; Indicates the dilation contrast region.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1) This paper is the first to use the inherent semantic consistency of the same action to perform discriminative representation learning in the field of point-supervised temporal action localization (PS-TAL). It introduces a semantic query learning framework and designs a set of learnable query vectors for semantic propagation. Each query represents a specific action category and is dynamically updated according to the video context. In the absence of segment-level annotation, the semantic consistency between segments of the same action is captured and utilized to optimize the representation of segment-level features, thereby improving the accuracy of temporal action localization. By dynamically updating queries, it can adapt to changes in video content, more flexibly capture the features of different action instances, and achieve more accurate action localization.

[0055] 2) The present invention constructs an integrity learning mechanism including an integrity assessment head and a boundary refinement head. The boundary refinement head is used to fine-tune the action boundary, and the integrity assessment head is used to evaluate and predict the integrity of the action instance. With the combined effect of the two, the complete action instance can be located more accurately.

[0056] 3) The present invention uses an I3D feature extractor to achieve seamless spatiotemporal feature extraction, which facilitates more efficient subsequent semantic query learning and action integrity learning.

[0057] 4) By designing the attention mask matrix, the present invention can aggregate semantically rich partial context information more efficiently and accurately.

[0058] 5) The present invention supervises boundary regression by refining the loss function, thereby improving the positioning capability and ensuring the accuracy and reliability of the action positioning result prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0061] Example

[0062] like Figure 1 As shown, this embodiment provides a method for point-supervised video action localization based on semantic query learning, and the method includes the following steps:

[0063] Step S1: Given an uncut video V of any length, containing M action instances For the mth action instance A m , and Respectively represent the start and end timestamps of the action instance. For each action instance, there is a random timing point p i , and the action category label of the point Where C is the number of action categories in the dataset. If the time point p i The corresponding action category is c, then The corresponding values ​​in the remaining dimensions are 0.

[0064] Step S2: Use the feature extractor to extract the spatiotemporal features X∈R of the video V T×D , where T represents the number of frames in the video and D represents the dimension of the feature.

[0065] In this embodiment, the feature extractor uses an I3D feature extractor, which can achieve seamless spatiotemporal feature extraction, facilitating more efficient subsequent semantic query learning and action integrity learning.

[0066] Step S3, semantic query learning: adopt the cross-attention mechanism based on the dynamic semantic query vector to propagate the semantic information between the segment-level representations of the spatiotemporal feature X to generate the semantically enhanced spatiotemporal feature X′, classify and convolve the semantically enhanced spatiotemporal feature X′, and fuse the generated first activation sequence and the first attention score sequence to obtain the first weighted category activation sequence P; wherein each query in the dynamic semantic query vector corresponds to an action of a specific category and is dynamically updated according to the video context. Specifically, it includes the following sub-steps:

[0067] Step S31: Due to the sparsity of labels in point supervision scenarios, it is very difficult to learn the semantic consistency of action instances. Therefore, the present invention assumes that the same action should have similar semantic clues. Specifically, a set of semantic query vectors G∈R is set and learned. C×D , where C is the number of action categories in the dataset, D is the dimension of spatiotemporal features, and each vector expresses the semantic information of a specific action category;

[0068] Step S32: Dynamically update the semantic query vector G∈R according to the video content through a cross-attention mechanism C ×D , generate a dynamic semantic query vector G′ that can perceive the video content. Specifically:

[0069] Q=GW q

[0070] K=XW k

[0071] V=XW v

[0072]

[0073] Where: W q ∈R D×D , W k ∈R D×D and W v ∈R D×D is the learnable parameter matrix; d k is the dimension size of the feature. Different from the traditional attention mechanism that uses a fully connected graph to associate all context information, an attention mask matrix U∈R is designed here C×T , which is used to aggregate semantically rich partial context information and is calculated as follows:

[0074]

[0075] Where: B∈R C×T It is a binary matrix whose element values ​​are determined by the probability confidence of each frame belonging to the action. If the confidence is greater than the set value, the corresponding value is set to 1, otherwise it is set to 0.

[0076] Using the dynamic semantic query vector G′, the temporal and spatial features X of the video are discriminated through the cross attention mechanism to obtain the enhanced temporal and spatial features E∈R C×D , the specific calculation process is as follows:

[0077] Q′=XW′ q

[0078] K′=G′W′ k

[0079] V′=G′W′ v

[0080]

[0081] Where: W′ q ∈R D×D , W′ k ∈R D×D and W′ v ∈R D×D is the learnable parameter matrix; U T is the transpose of the attention mask matrix U.

[0082] Step S33, the enhanced spatiotemporal feature E is further input into a series of feedforward units for processing, the series of feedforward units include a fully connected layer, a linear rectification layer and a fully connected layer arranged in sequence to generate a first feature O, the first feature O is combined with the enhanced spatiotemporal feature E and input into a regularization layer to obtain a semantically enhanced spatiotemporal feature X′, the specific mathematical expression is:

[0083] E′=LN(E)

[0084] O = FC(ReLU(FC(E)))

[0085] X′=LN(O+E)

[0086] Where: LN represents the layer normalization operation; FC represents the fully connected layer; ReLU is the linear rectification function.

[0087] Step S34: The semantically enhanced spatiotemporal features X′ are input into a segment-level classifier to generate a category-aware first category activation sequence S∈R T×C , and use a convolutional layer to generate a category-independent attention score sequence A∈R T×1 , each value in A represents the confidence that the corresponding video frame belongs to an action;

[0088] Step S35: Use element-wise product to fuse S and A to obtain the first weighted class activation sequence P∈R C×T :

[0089] P=S T ·A

[0090] In the point supervision paradigm, each interval containing an action must have a timestamp that has been manually annotated, and two adjacent point annotations must be located in different action intervals. Combining the prior information provided by the point annotations and the category-independent attention score sequence A, reliable pseudo-action instances can be selected. As a positive sample (f i represents the i-th positive sample, N act is the number of action instances), and pseudo-background instances (f j represents the jth negative sample, N bkg is the number of background instances). Specifically, for a segment near a given point annotation, when the category-independent attention score is greater than a preset threshold, the segment is considered a pseudo action instance; at the same time, for the area between two consecutive point annotations, the segment with the lowest attention score or an attention score below a given threshold is considered a pseudo background instance. In order to learn this process, we perform supervised training with the following loss function:

[0091]

[0092] Where: FL represents focal loss (Focal Loss);

[0093] In addition, in order to ensure that the generated dynamic query vector G′ is semantically relevant, a fully connected layer is used to map it to the category space and supervise it through the focal loss:

[0094] Z=G′W θ

[0095]

[0096] Where: W θ ∈R D×C is the learnable mapping matrix in the fully connected layer.

[0097] Step S4, completeness learning: Due to the sparsity of PS-TAL annotations, it is very difficult to locate complete action clips. The pseudo-labels and semantically enhanced spatiotemporal features generated in step S3 are used to learn the completeness of the action. Specifically, an integrity assessment head and a boundary refinement head are designed, and a novel refinement loss is combined to improve the accuracy of the action boundary. It includes the following sub-steps:

[0098] Step S41: for each action category, select an action category in the first weighted category activation sequence P whose activation value is greater than a first preset threshold value θ p After removing the area without the annotation point, the remaining area is used as the candidate positive sample; for the first attention score sequence A that is irrelevant to the category, the activation value is less than the second preset threshold θ a The area is taken as a candidate negative sample;

[0099] Step S42: Input the semantically enhanced spatiotemporal feature X′ into the fully connected layer to generate a second category-aware activation sequence S′ and a second category-independent attention score sequence A′, and fuse them by element-wise product to obtain a second weighted category activation sequence P′∈R C×T :

[0100] P′=S′·A′

[0101] Step S43: for each action category, select the action category with an activation value greater than the first preset threshold value θ in the second weighted category activation sequence P′. p , and the remaining areas are taken as candidate positive samples; for the second attention score sequence A′ that is irrelevant to the category, select the area with an activation value less than the second preset threshold θ a The area is taken as a candidate negative sample;

[0102] Step S44: For the positive and negative candidate samples generated above, their integrity needs to be determined.

[0103] 1) Given the i-th candidate region The boundary is expanded by a hyperparameter α to generate three regions: left, middle, and right. The left features are extracted from the three regions using RoIAlign and maximum pooling operations. Central Features and the right feature

[0104] 2) Use the integrity assessment head to predict the action integrity score using boundary-sensitive regional features For the candidate region K i , its left feature With central features The difference and center features And the right side features With central features After the difference cascade operation, it is sent to a regularization layer, a fully connected layer, a linear rectification layer and an activation layer in turn to predict the completeness score of the action

[0105]

[0106] Where: Cat represents the cascade operation; Sigmoid is the activation function; Represents the predicted action completeness score; supervised training is performed using an instance-level loss function, and the loss function expression is:

[0107]

[0108] Where: N p and N n Respectively represent the number of positive and negative samples; SmoothL1 is the smoothed one-norm loss function; the true value u i It is calculated by calculating the intersection and union of the candidate region and its most matching pseudo-annotated instance;

[0109] 3) Use the boundary refinement head to perform boundary regression processing to obtain the start timestamp prediction value and the end timestamp prediction value For the positive sample candidate region K i , according to its left feature and central features Predict the offset value Δs of the action start boundary, based on its central feature and the right feature Predict the offset value Δe of the action end boundary, and calculate the start timestamp prediction value of the video frame based on the offset value Δs of the action start boundary and the offset value Δe of the action end boundary and the end timestamp prediction value

[0110] According to its left feature and central features The offset value Δs of the predicted action start boundary is expressed as:

[0111]

[0112] Where: Conv represents the convolutional layer;

[0113] Similarly, according to its central feature and the center feature on the right The offset value Δe of the predicted action end boundary is expressed as:

[0114]

[0115] Then, the action area is updated using the starting offset value Δs and the ending offset value Δe, and the expression is:

[0116]

[0117] Where: represents the length of the original candidate region, i.e.

[0118] Construct a refined loss function to supervise the boundary regression, the expression is:

[0119]

[0120] Where: and They represent the completeness score of the original action instance and the updated action completeness score respectively, and the action completeness is improved in this way.

[0121] 4) Finally get the predicted action instance in It represents the confidence that the candidate region contains action c, which is obtained by considering the confidence of the candidate region interval and the adjacent candidate regions at the same time. The specific calculation expression is:

[0122] The confidence level that the candidate region in step S44 contains action c It is obtained by considering the confidence of the candidate region interval and the adjacent candidate regions at the same time. The specific calculation expression is:

[0123]

[0124]

[0125] Where: is the confidence that the candidate region contains action c, is the confidence that the adjacent candidate region includes action c; P′[t,c] is the second category activation sequence corresponding to action c; Indicates the dilation contrast region.

[0126] Step S5: locate the position interval of the action in the uncut video V according to the predicted action instance, and filter the redundant predicted action instances through the non-maximum suppression algorithm to obtain the final point-supervised video action positioning result.

[0127] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A point-supervised video action localization method based on semantic query learning, characterized in that: The method includes: Step S1: Given an uncut video V, which includes multiple action instances, each action instance is provided with a start and end timestamp, and there is a random timing point and an action category label corresponding to the timing point; Step S2, extracting spatiotemporal features X from the uncut video V using a feature extractor; Step S3, semantic query learning: using a cross-attention mechanism based on a dynamic semantic query vector to propagate semantic information between segment-level representations of the spatiotemporal feature X to generate a semantically enhanced spatiotemporal feature X ′ , to enhance the spatiotemporal features X ′ Classification and convolution processing are performed respectively, and the generated first activation sequence and the first attention score sequence are fused to obtain a first weighted category activation sequence P; wherein each query in the dynamic semantic query vector corresponds to an action of a specific category and is dynamically updated according to the video context; Step S4: Completeness learning: semantically enhanced spatiotemporal features X ′ The second activation sequence and the second attention score sequence generated by the fully connected layer are input and fused to obtain the second weighted category activation sequence P ′ , the predicted action instance is obtained by using the integrity learning mechanism based on the integrity assessment head and the boundary refinement head; Step S5: locate the position interval of the action in the uncut video V according to the predicted action instance, and filter the redundant predicted action instances through the non-maximum suppression algorithm to obtain the final point-supervised video action positioning result.

2. According to the method of claim 1, the method is characterized in that: The step S1 is specifically as follows: Given an uncut video V, which contains M action instances and Respectively represent the mth action instance A m The start and end timestamps of For each action instance, there is a random timing point p i And the action category label at this timing point Where C is the number of action categories. If the time point p i The corresponding action category is c, then The corresponding values ​​in the remaining dimensions are 0.

3. The method for point-supervised video action localization based on semantic query learning according to claim 1, characterized in that: The feature extractor in step S2 is an I3D feature extractor.

4. The method for point-supervised video action localization based on semantic query learning according to claim 1, characterized in that: The step S3 comprises the following sub-steps: Step S31: construct a set of query vectors G∈R that express the semantic information of a specific action category C×D , where C is the number of action categories and D is the dimension of spatiotemporal features; Step S32: Dynamically update the query vector G∈R through the cross-attention mechanism C×D , generate a dynamic semantic query vector G′ that can perceive the video content, and based on the dynamic semantic query vector G′, enhance the discriminability of the spatiotemporal feature X through the cross attention mechanism to obtain the enhanced spatiotemporal feature E∈R C×D : Step S33, inputting the enhanced spatiotemporal feature E into a fully connected layer, a linear rectification layer and a fully connected layer in sequence to generate a first feature O, combining the first feature O with the enhanced spatiotemporal feature E and inputting them into a regularization layer to obtain a semantically enhanced spatiotemporal feature X′; Step S34: Input the semantically enhanced spatiotemporal feature X′ into a segment-level classifier to generate a category-aware first activation sequence S∈R T×C , input to a convolutional layer to generate a class-independent first attention score sequence A∈R T×1 , where each value in A represents the confidence of the action to which the corresponding video frame belongs; Step S35: Use element-wise product to fuse the first activation sequence S and the first attention score sequence A to obtain the first weighted category activation sequence P∈R C×T .

5. The method for point-supervised video action localization based on semantic query learning according to claim 4, characterized in that: The step S32 is specifically as follows: Dynamically update the query vector G∈R via the criss-cross attention mechanism C×D Generate a dynamic semantic query vector G′ that can perceive the video content, expressed as: Where: Q = GW q , K = XW k , V = XW v , W q ∈R D×D , W k ∈R D×D and W v ∈R D×D is the learnable parameter matrix; d k is the dimension of the feature; U∈R C×T is the attention mask matrix, which is used to aggregate the context information of the semantic part, and its expression is: Where: B∈R C×T It is a binary matrix whose element value is determined by the probability confidence of each frame belonging to the action. If the confidence is greater than the set value, the corresponding value is set to 1, otherwise it is set to 0; Based on the dynamic semantic query vector G′, the spatial-temporal feature X is discriminated and enhanced through the cross-attention mechanism to obtain the enhanced spatial-temporal feature E∈R C×D , the expression is: Where: Q′=XW′ q , K′=G′W′ k , V′=G′W′ v , W′ q ∈R D×D , W′ k ∈R D×D and W′ v ∈R D×D is the learnable parameter matrix; U T is the transpose of the attention mask matrix U.

6. The method for point-supervised video action localization based on semantic query learning according to claim 4, characterized in that: The step S3 further includes: Combining the prior information provided by the point annotation and the category-independent first attention score sequence A, pseudo-action instances are selected As positive samples and pseudo background instances For a segment near a given point annotation, when the category-independent attention score value is greater than a preset threshold, the segment is considered a pseudo action instance; for the area between two consecutive point annotations, the segment with the lowest attention score or an attention score below a given threshold is considered a pseudo background instance.

7. The method for point-supervised video action localization based on semantic query learning according to claim 4, characterized in that: The step S4 comprises the following sub-steps: Step S41: for each action category, select an action category in the first weighted category activation sequence P whose activation value is greater than a first preset threshold value θ p After removing the area without the annotation point, the remaining area is used as the candidate positive sample; for the first attention score sequence A that is irrelevant to the category, the activation value is less than the second preset threshold θ a The area is taken as a candidate negative sample; Step S42: input the semantically enhanced spatiotemporal features X′ into the fully connected layer to generate a category-aware second activation sequence S′ and a category-independent second attention score sequence A′, and fuse them by element-wise product to obtain a second weighted category activation sequence P′; Step S43: for each action category, select the action category with an activation value greater than the first preset threshold θ in the second weighted category activation sequence P′. p , and the remaining areas are taken as candidate positive samples; for the second attention score sequence A′ that is irrelevant to the category, select the area with an activation value less than the second preset threshold θ a The area is taken as a candidate negative sample; Step S44: Use the integrity assessment head to predict the action integrity score using boundary-sensitive regional features The boundary refinement head is used to perform boundary regression processing to obtain the start timestamp prediction value and the end timestamp prediction value Get predicted action instance in Indicates the confidence that the candidate region contains action c.

8. The method for point-supervised video action localization based on semantic query learning according to claim 7, characterized in that: The step S44 specifically includes: Given a candidate region, its boundary is expanded by hyperparameters to generate three regions: left, middle, and right. The left features are extracted from the three regions respectively by region alignment and maximum pooling operations. Central Features and the right feature The subscript i corresponds to the number of the candidate region; The completeness assessment head is used to predict the action completeness score using boundary-sensitive regional features. For the candidate region K i , its left feature With central features The difference and center features And the right side features With central features After the difference cascade operation, it is sent to a regularization layer, a fully connected layer, a linear rectification layer and an activation layer in turn to predict the completeness score of the action Among them, the instance-level loss function is used for supervised training; The boundary refinement head is used to perform boundary regression processing to obtain the start timestamp prediction value and the end timestamp prediction value For the positive sample candidate region K i , according to its left feature and central features Predict the offset value Δs of the action start boundary, based on its central feature and the right feature Predict the offset value Δe of the action end boundary, and calculate the start timestamp prediction value of the video frame based on the offset value Δs of the action start boundary and the offset value Δe of the action end boundary and the end timestamp prediction value 9. The method for point-supervised video action localization based on semantic query learning according to claim 7, characterized in that: In step S44, supervised training is performed on the integrity assessment head using an instance-level loss function, and the loss function expression is: Where: N p and N n Represent the number of positive and negative samples respectively; SmoothL1 is the smoothed one-norm loss function, the true value u i It is calculated by calculating the intersection-over-union ratio of the candidate region and its most matching pseudo-annotated instance; The refinement loss function is used to supervise the training of the boundary refinement head. The loss function expression is: Where: and They represent the completeness score of the original action instance and the updated action completeness score respectively.

10. The method for point-supervised video action localization based on semantic query learning according to claim 7, characterized in that: The confidence level that the candidate region in step S44 contains action c It is obtained by considering the confidence of the candidate region interval and the adjacent candidate regions at the same time. The specific calculation expression is: Where: is the confidence that the candidate region contains action c, is the confidence that the adjacent candidate region includes action c; P′[t, c] is the second category activation sequence corresponding to action c; Indicates the dilation contrast region.

Images