Timing behavior detection method, apparatus, device, medium, and program product

Abstract

Embodiments of the present application disclose a timing behavior detection method, device, equipment, medium and program product. The method of the present application determines a plurality of key frames and a plurality of key objects from a target video frame sequence based on an action query amount, then updates the action query amount by obtaining a first feature correlation relationship between the key objects, a second feature correlation relationship between the key frames, and a third feature correlation relationship between the key frames and the key objects. Based on the updated action query amount, a plurality of target key frames are re-determined from the target video frame sequence, and the starting time, ending time and behavior category of the target behavior in the target video frame sequence are determined according to the target key frames and their timing positions in the target video frame sequence. In the present scheme, the spatial object semantics is introduced into the behavior action detection, and the semantic information in the spatial and time dimensions is combined, thereby improving the detection accuracy of the timing behavior in the first perspective scene.

Description

Technical Field

[0001] This application relates to the field of information processing technology, specifically to a method, apparatus, electronic device, computer storage medium, and computer program product for detecting temporal behavior. Background Technology

[0002] With the development of the internet and the advancement of technology, video-based research has received increasing attention. Videos contain temporal information, which can generally be understood as the continuity between frames; therefore, video processing typically requires a large amount of computation. Temporal behavior detection, on the other hand, involves locating and identifying behavioral segments within an undressed video input.

[0003] In existing technologies, temporal behavior detection schemes focus on complex temporal modeling. However, in scenarios involving first-person perspective temporal detection, the actor performing the action cannot be observed in the first-person video, and the image changes between frames are more drastic, easily causing significant noise and affecting detection results. Furthermore, since first-person perspective behavior is more reflected in the objects and environment interacting with the actor, such spatial semantic information is difficult to extract through modeling with only temporal dimension information.

[0004] It is known that the detection accuracy of existing technologies in performing temporal behavior detection is relatively poor. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting temporal behavior, which can improve the detection accuracy of temporal behavior.

[0006] A temporal behavior detection method, comprising:

[0007] Based on the action query volume, multiple key frames and multiple key objects are identified from the target video frame sequence;

[0008] Obtain the first feature correlation between the key objects, the second feature correlation between the key frames, and the third feature correlation between the key frames and the key objects;

[0009] The action query volume is updated based on the first feature correlation, the second feature correlation, and the third feature correlation.

[0010] Based on the updated action query volume, multiple target keyframes are re-determined from the target video frame sequence;

[0011] Based on the target keyframe and its temporal position in the target video frame sequence, determine the start time, end time, and behavior category of the target behavior in the target video frame sequence.

[0012] Accordingly, embodiments of this application also provide a timing behavior detection device, including:

[0013] The first determining unit is used to determine multiple key frames and multiple key objects from the target video frame sequence based on the action query volume.

[0014] The acquisition unit is used to acquire a first feature correlation between the key objects, a second feature correlation between the key frames, and a third feature correlation between the key frames and the key objects;

[0015] The update unit is used to update the action query volume based on the first feature correlation, the second feature correlation, and the third feature correlation.

[0016] The second determining unit is used to redetermine multiple target keyframes from the target video frame sequence based on the updated action query volume.

[0017] The third determining unit is used to determine the start time, end time, and behavior category of the target behavior in the target video frame sequence based on the target keyframe and its temporal position in the target video frame sequence.

[0018] Optionally, in some implementations, the action query quantity includes: a query vector and multiple query points; the first determining unit is used for:

[0019] Based on the query vector, key objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0020] Based on the key object, multiple query points are distributed at corresponding temporal positions in the target video frame sequence, and the video frame at the temporal position of the query point is determined as the key frame.

[0021] Optionally, in some implementations, the updating unit is used for:

[0022] The query vector is updated based on the first feature correlation, the second feature correlation, and the third feature correlation.

[0023] Based on the updated query vector, determine the position offset of the query point;

[0024] The position distribution of the query point in the target video frame sequence is updated based on the position offset.

[0025] Optionally, in some embodiments, the first feature correlation includes a first feature matrix, the second feature correlation includes a second feature matrix, and the third feature correlation includes a third feature matrix;

[0026] When updating the query vector based on the first feature correlation, the second feature correlation, and the third feature correlation, the updating unit is further configured to:

[0027] Based on the trained weighting parameters, the first feature matrix, the second feature matrix, and the third feature matrix are weighted respectively.

[0028] The weighted first feature matrix, the second feature matrix, and the third feature matrix are fused to obtain a fused feature matrix.

[0029] The query vector is updated based on the fused feature matrix.

[0030] Optionally, in some embodiments, the second determining unit is used to:

[0031] Based on the updated location distribution of the query points, multiple target keyframes are re-determined from the target video frame sequence.

[0032] Optionally, in some implementations, when identifying key objects related to the category of the behavior to be detected from the target video frame sequence based on the query vector, the first determining unit is further configured to:

[0033] Extract image features from each video frame in the target video frame sequence;

[0034] Based on the query vector and the image features, candidate objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0035] The key object is determined from the candidate objects based on the sample object label set.

[0036] Optionally, in some embodiments, the acquisition unit is used for:

[0037] Feature extraction is performed on the keyframes to obtain keyframe features;

[0038] Feature extraction is performed on the key object to obtain the key object features;

[0039] Based on the keyframe features, the key object features, and the query vector, the first feature correlation, the second feature correlation, and the third feature correlation are determined.

[0040] Optionally, in some embodiments, the third determining unit is used for:

[0041] Convert the target keyframe into a pseudo-timing box;

[0042] Based on the pseudo-sequence frame, determine the start and end times of the target behavior;

[0043] Based on the target keyframe, determine the confidence level of the target behavior across multiple categories of behaviors to be detected;

[0044] The category of the behavior to be detected with the highest confidence level is determined as the behavior category of the target behavior.

[0045] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to implement the steps in the timing behavior detection method provided in the embodiments of this application.

[0046] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the steps in the timing behavior detection method provided in embodiments of this application.

[0047] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the timing behavior detection method provided in embodiments of this application.

[0048] In this embodiment, based on action query data, multiple keyframes and multiple key objects are determined from a target video frame sequence. Then, the action query data is updated by acquiring the first feature correlation between key objects, the second feature correlation between keyframes, and the third feature correlation between keyframes and key objects. Based on the updated action query data, multiple target keyframes are re-determined from the target video frame sequence. Based on the target keyframes and their temporal positions in the target video frame sequence, the start time, end time, and behavior category of the target behavior in the target video frame sequence are determined. This solution introduces spatial object semantics into behavior action detection, combining semantic information in both spatial and temporal dimensions to improve the detection accuracy of temporal behaviors in first-person perspective scenes. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of a scenario for the temporal behavior detection method provided in the embodiments of this application;

[0051] Figure 2 This is a flowchart illustrating the temporal behavior detection method provided in the embodiments of this application;

[0052] Figure 3 This is a schematic diagram of the system architecture of the temporal behavior detection model provided in the embodiments of this application;

[0053] Figure 4 This is a schematic diagram of the architecture of the hybrid interaction module provided in the embodiments of this application;

[0054] Figure 5 This is a schematic diagram of the structure of the timing behavior detection device provided in the embodiments of this application;

[0055] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting timing behavior. The timing behavior detection apparatus can be integrated into an electronic device, which may be a server or a terminal, etc.

[0058] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, network acceleration services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0059] For example, see Figure 1Taking the integration of a temporal behavior detection device into an electronic device as an example, the electronic device determines multiple key frames and multiple key objects from the target video frame sequence based on the action query quantity; obtains the first feature correlation between key objects, the second feature correlation between key frames, and the third feature correlation between key frames and key objects; updates the action query quantity based on the first feature correlation, the second feature correlation, and the third feature correlation; re-determines multiple target key frames from the target video frame sequence based on the updated action query quantity; and determines the start time, end time, and behavior category of the target behavior in the target video frame sequence based on the target key frames and their temporal positions in the target video frame sequence.

[0060] The temporal behavior detection method provided in this application relates to machine learning in artificial intelligence. This application introduces spatial object semantics into behavior detection. Based on a query-based detection paradigm, it adaptively samples key objects and keyframes of actions using learnable action queries. It utilizes important semantic information from both spatial and temporal dimensions to provide finer-grained semantic details for first-person perspective temporal behavior detection.

[0061] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0062] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, large-scale temporal behavior detection, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and the theory of complex information in algorithms. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0064] It is understood that in the specific implementation of this application, related data such as attribute data, attribute sets and attribute subsets are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0065] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0066] This embodiment describes the angular information of the timing behavior detection device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), or other smart devices capable of timing behavior detection.

[0067] This application provides a temporal behavior detection method, comprising: determining multiple keyframes and multiple key objects from a target video frame sequence based on action query data; obtaining a first feature correlation between key objects, a second feature correlation between keyframes, and a third feature correlation between keyframes and key objects; updating the action query data based on the first feature correlation, the second feature correlation, and the third feature correlation; re-determining multiple target keyframes from the target video frame sequence based on the updated action query data; and determining the start time, end time, and behavior category of a target behavior in the target video frame sequence based on the target keyframes and their temporal positions in the target video frame sequence.

[0068] like Figure 2 As shown, the specific process of this temporal behavior detection method is as follows:

[0069] 101. Based on the action query volume, identify multiple key frames and multiple key objects from the target video frame sequence.

[0070] In this embodiment, the aim is to locate and identify the actions initiated by the videographer in first-person perspective videos (i.e., videos where the action initiator is not in the frame) that have not undergone temporal cropping. The target video frame sequence is the sequence of all image frames contained in the video to be detected. Keyframes are the most important video frames for each type of action recognition, and key objects are the objects contained in the keyframes that are most important for each type of action recognition, such as physical objects like books, electronic devices, and sporting goods.

[0071] In this system, an action query can independently predict an action instance, while the same action occurring at different time periods constitutes different behavior instances. In practice, keyframes and key objects can be adaptively sampled based on the features of each query.

[0072] In this embodiment, the action query quantity includes a query vector and multiple query points. The query vector is responsible for updating the category of the action and the location of the query points. In this scheme, the extraction of temporal features must first be completed. Specifically, the video input (i.e., the target video frame sequence) is divided into multiple video segments according to a specified step size, and a multi-dimensional feature vector is output for each video segment through a temporal feature extraction network. In order to extract object features, key objects are also extracted in each sampled frame according to the same specified step size. That is, in some embodiments, when determining multiple key frames and multiple key objects from the target video frame sequence based on the action query quantity, the specific process may include the following:

[0073] Based on query vectors, key objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0074] Based on the key objects, multiple query points are distributed at the corresponding temporal positions in the target video frame sequence, and the video frames at the temporal positions of the query points are determined as key frames.

[0075] In this system, the target behavior category is a closed set of behavior categories formed during training, and the query vector represents the learned features of a certain behavior category. Therefore, based on the query vector, key objects related to the target behavior category within this closed set can be identified from the target video frame sequence. Then, according to the temporal position of the video frames containing the key objects, multiple query points are distributed at corresponding temporal positions; the specific distribution depends on the number of query points. Finally, the temporal positions where the query points are distributed are determined, and the video frames at those positions are identified as keyframes.

[0076] In some implementations, identifying key objects related to the category of the behavior to be detected from a sequence of target video frames based on a query vector may specifically include the following process:

[0077] Extract image features from each video frame in the target video frame sequence;

[0078] Based on the query vector and image features, candidate objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0079] Based on the sample object label set, key objects are identified from the candidate objects.

[0080] In this scheme, a pre-defined set of sample object labels is required to limit the scope of object recognition. In practical applications, this set of sample object labels can be a vocabulary list, and textual features extracted from video frames using a visual-language model can be used as object labels. By constructing a vocabulary list, it is easier to identify more object labels related to the category of the behavior to be detected from the video frame sequence, further improving detection performance.

[0081] 102. Obtain the first feature correlation between key objects, the second feature correlation between key frames, and the third feature correlation between key frames and the key objects.

[0082] Specifically, the process of obtaining the first feature correlation between key objects, the second feature correlation between key frames, and the third feature correlation between key frames and key objects can include the following steps:

[0083] Feature extraction is performed on keyframes to obtain keyframe features;

[0084] Feature extraction is performed on key objects to obtain their features;

[0085] Based on keyframe features, key object features, and query vectors, determine the first feature correlation, the second feature correlation, and the third feature correlation.

[0086] In practical implementation, the keyframe features and key object features can be feature matrices. To simultaneously optimize the keyframe features and key object features in a high-dimensional space, the temporal keyframe features can be extended in the spatial dimension, and the keyframe features and key object features are concatenated along the channel dimension to form a joint feature matrix. When determining the first feature correlation, the second feature correlation, and the third feature correlation based on the keyframe features, key object features, and query vector, these correlations can be obtained through calculations between the joint feature matrix and the query vector.

[0087] In practical applications, a hybrid interactive module can be used to model the joint feature matrix, adaptively modeling inter-frame temporal sequence and object correlation.

[0088] In some embodiments, for key objects, the relevance of each object token can be calculated by performing cross-attention between all key objects and a query vector, wherein the query vector is used as a key input, and the joint feature matrix is used as a query input and a value input. Since adjacent frames may contain a large number of repeated object tokens, the relevance weight of each object after accumulation may tend to be biased towards objects that are repeated more times. Therefore, in the present solution, a count matrix can be calculated for all object tokens, its reciprocal matrix can be calculated as a normalization mask, and the mask is multiplied with the cross-attention map to obtain a corresponding feature matrix, which is used as the first correlation relationship.

[0089] In some embodiments, for key frames, dynamic mixing can be used to adaptively fuse features corresponding to the key frames based on query vectors. The query vector is projected through a linear layer to generate N p ×N p weight matrix, which is used as a parameter of dynamic mixing to model the temporal dependency of the feature matrix, and obtain the temporally mixed feature matrix as the third correlation relationship. It should be noted that the weight matrix for temporal mixing is shared by key objects in the same frame, and the weight matrix is different for different action queries.

[0090] In some embodiments, for the relationship between key frames and key objects, a dynamic mixing model with a bottleneck structure can be adopted, and the weight matrices are two matrices with sizes of (D×D h ),(D h ×D) obtained after transformation of the query vector, where D h is a hidden feature dimension (D h <D), which realizes mixing of temporal feature channels and object feature channels. Specifically, the above joint feature matrix is multiplied by the two matrices obtained after transformation respectively, and the corresponding matrix is output as the third correlation relationship.

[0091] 103. Update the action query based on the first feature correlation relationship, the second feature correlation relationship and the third feature correlation relationship.

[0092] Specifically, the query may include a query vector and a plurality of query points. When updating the action query based on the first feature correlation relationship, the second feature correlation relationship and the third feature correlation relationship, the query vector may be updated according to the first feature correlation relationship, the second feature correlation relationship and the third feature correlation relationship; then, the position offset of the query points is determined according to the updated query vector; finally, the position distribution of the query points in the target video frame sequence is updated according to the position offset.

[0093] In some implementations, the first feature correlation includes a first feature matrix, the second feature correlation includes a second feature matrix, and the third feature correlation includes a third feature matrix. When updating the query vector based on the first, second, and third feature correlations, the process may specifically include the following:

[0094] Based on the trained weighting parameters, the first feature matrix, the second feature matrix, and the third feature matrix are weighted respectively.

[0095] The weighted first feature matrix, second feature matrix and third feature matrix are fused to obtain the fused feature matrix;

[0096] Update the query vector based on the fused feature matrix.

[0097] The trained weighted parameters can be predicted from the query vector through linear projection.

[0098] Specifically, when updating the query vector based on the fused feature matrix, the fused feature matrix is ​​averaged across the key object dimensions, the resulting features are projected into a vector with the same dimension as the query vector through a linear layer, and the query vector is updated through the residual structure.

[0099] 104. Based on the updated action query volume, redetermine multiple target keyframes from the target video frame sequence.

[0100] Specifically, multiple target keyframes can be re-determined from the target video frame sequence based on the query points after the updated location distribution. That is, the temporal position of the query point after the updated location distribution is determined, and the video frame at that temporal position is identified as the target video frame.

[0101] For example, a query vector might correspond to 5 query points, initially distributed at temporal positions [3, 4, 5, 5, 6] in the video frame sequence. As the number of iterations increases, the query vector calculates the offset of each query point to return it to the temporal position of the key frame corresponding to the action. For instance, after one iteration, the temporal position of the query point might change from [3, 4, 5, 5, 6] to [1, 3, 4, 6, 8], which is the temporal position of the target key frame.

[0102] 105. Based on the target keyframes and their temporal positions in the target video frame sequence, determine the start time, end time, and behavior category of the target behavior in the target video frame sequence.

[0103] Specifically, based on the target keyframes and their temporal positions in the target video frame sequence, determining the start time, end time, and behavior category of the target behavior in the target video frame sequence can include the following process:

[0104] Convert the target keyframe into a pseudo-timing box;

[0105] Based on the pseudo-sequence box, determine the start and end times of the target behavior;

[0106] Based on the target keyframes, determine the confidence level of the target behavior across multiple categories of detected behaviors;

[0107] The category of the behavior to be detected with the highest confidence level is determined as the category of the target behavior.

[0108] In this scheme, the location of the action boundary is obtained by transforming the query point using a "point-pseudo-boundary" transformation function (e.g., PartialMin-max, a local maximum-minimum transformation function). Here, "point-pseudo-boundary" refers to a method that takes a set of temporal points distributed across a video frame sequence and transforms them into action start and end points. Specifically, the temporal position with the smallest temporal value in this set of points is taken as the action start point, and the temporal position with the largest temporal value is taken as the action end point. The start and end points together form the "temporal bounding box" of the action; since it is not the true temporal bounding box of the action, it is also called a "pseudo-boundary."

[0109] Behavior categories can be predicted from the query vector using a category header structure consisting of linear layers. In practice, confidence scores are predicted for each category in the closed set of behavior categories to be detected, and the category with the highest confidence score is taken as the final detected behavior category. In practice, behavior categories can be given in the form of noun-verb pairs, such as "playing football" or "playing basketball".

[0110] As can be seen from the above, the temporal behavior detection method provided in this application, based on the action query quantity, determines multiple keyframes and multiple key objects from the target video frame sequence. Then, it updates the action query quantity by obtaining the first feature correlation between key objects, the second feature correlation between keyframes, and the third feature correlation between keyframes and key objects. Based on the updated action query quantity, multiple target keyframes are re-determined from the target video frame sequence, and the start time, end time, and behavior category of the target behavior in the target video frame sequence are determined according to the target keyframes and their temporal positions in the target video frame sequence. In this scheme, spatial object semantics are introduced into behavior action detection. Based on the query quantity detection paradigm, learnable action queries are used to adaptively sample action key objects and keyframes. Combining important semantic information in the spatial and temporal dimensions, more fine-grained semantic details are provided for first-person perspective temporal action detection perception, improving the detection accuracy of temporal behavior in first-person perspective scenes.

[0111] Based on the method described in the above embodiments, the following examples will provide further detailed explanations. For instance... Figure 3 As shown, in another embodiment of this application, a temporal behavior detection model is also provided, which is described in detail below:

[0112] In this scheme, for a given input video V, the set of actions appearing within the video is marked as... in This indicates the start and end of the truth value of the nth action. For the action category, N g This represents the number of action annotations. This scheme employs a query-based sparse detection framework, using N... q Predicting first-person human behavior in video clips based on individual action query volume: in These are the start and end time positions of the predicted behavior, respectively, c n The predicted behavior category.

[0113] The sparse detection paradigm is a detection framework that learns the target to be detected directly from the input using a series of sparsely distributed queries. During training, action queries are matched one-to-one with equally sparse detection labels to select positive samples for targeted training, thus eliminating the need for time-consuming post-processing steps like non-maximum suppression during the detection process.

[0114] In this approach, the model's input consists of a series of temporal features. The K most relevant object tokens for each frame. Where D v D l Let i = {1, 2, ..., N} represent the number of channels (i.e., dimensions) for temporal features and object features, respectively. T}, N T This represents the number of video frames.

[0115] In this scheme, temporal features are first encoded using stacked temporal convolutions and local self-attention modules, and then object features are linearly transformed using a multilayer perceptron. Then, based on a sparse detection framework, N... q Behavior detection is performed using a set of learnable action queries, where each query includes N. p Query points and a query vector Where D is the dimension of the vector features. Then, each action query is sampled from the input sequence N. p Keyframes and N p ×K key objects form a joint feature matrix Next, the joint feature matrix is ​​modeled using a hybrid interaction module, adaptively modeling inter-frame temporal relationships and object correlations. The joint feature matrix updates the query vector q through the residual structure after the hybrid interaction module. The updated query vector predicts the query point offset using a location head (loc.head) composed of linear layers to update its position, and is then transformed using a local maxima-minimum transformation function to obtain the position of the behavior boundary (i.e., the start and end temporal positions of the behavior action). The behavior category is predicted using a category head (cls.head) structure composed of linear layers.

[0116] For example, consider a first-person perspective video uploaded by a user, shot with a GoPro, containing a segment of someone kicking a soccer ball. The start and end times of this segment are 1 second and 8 seconds, respectively. By directly inputting the video frame sequence into the model, the model can detect the behavior category as "kicking a soccer ball" end-to-end, with start and end times of "1 second" and "8 seconds," respectively. In the specific implementation of the model, it internally receives Nq action query vectors as input, and each action query vector is accompanied by Np query points. In a possible implementation, Nq can be set to 50, Np can be set to 12, and the dimension of each query vector can be set to 512.

[0117] In this embodiment, for temporal features, a Slowfas backbone network can be used for feature extraction. The video input V can be divided into segments of 32 frames each, with a stride of 16 frames. The Slowfas backbone network outputs a D for each video segment. v The model uses 16-dimensional feature vectors. To extract object features, it also uses a 16-frame step size, extracting the K most likely objects in each sampled frame (selecting the K most relevant objects from the detected objects in the video frames). This extraction process employs a Contrastive Language-Image Pre-Training (CLIP) model to extract textual features from the video frames as object labels. By constructing a custom vocabulary list for the dataset (i.e., the dataset used to train the model), the model can identify more object labels relevant to the dataset from the video, further improving detection performance.

[0118] In this embodiment, the location and category of a behavior are predicted using sparsely distributed action queries. Each query adaptively learns keyframes and key objects from both temporal and spatial perspectives to obtain discriminative features. For the temporal dimension, one-dimensional query points are used for keyframe selection. Keyframe features are extracted using deformable convolution, and the resulting feature matrix is ​​denoted as . In the spatial dimension, each query adaptively samples a set of key objects. The set of key objects is determined based on the assumption that "all important objects are contained within the keyframe," and the key object features are denoted as follows:

[0119] In practice, to simultaneously optimize keyframe features and key object features in a high-dimensional space, the temporal keyframe features F can be extended in the spatial dimension. frame and F token and F frame A joint feature matrix is ​​formed by concatenating along the channel dimension.

[0120] This scheme proposes a hybrid interaction module that adaptively optimizes the joint feature matrix F by utilizing semantic cues between keyframes and key objects. q . refer to Figure 4 The hybrid interaction module adopts a three-branch structure to model the correlation between key objects and behavior prediction, the temporal dependency between key frames, and the correlation between feature channels.

[0121] For key objects, the relevance of each object label is calculated by performing cross-attention between all key objects and the query vector q. Here, the query vector serves as the key input (K), the feature matrix as the query input (Q), and the value input (V). Since adjacent frames may contain a large number of repeated object labels, the accumulated relevance weights for each object may favor objects that appear more frequently. Therefore, this scheme calculates a count matrix for all object labels, uses its reciprocal matrix as a normalized mask, and multiplies the mask with the cross-attention map. The feature matrix after modeling the relevance of key object labels is represented as F. q,k .

[0122] For keyframes, a dynamic blending approach will be used to adaptively fuse the features corresponding to the keyframes. Specifically, the query vector... Projecting through a linear layer generates N. p ×N p The weight matrix is ​​used as a parameter for dynamic mixing to model the temporal dependency of the feature matrix, resulting in the temporally mixed feature matrix represented as F. q,t The temporally mixed weight matrix is ​​shared by K key objects in the same frame.

[0123] For channel mixing, this solution adopts a dynamic mixing model with a bottleneck structure, and the weight matrix is ​​of size (D×D) obtained after transforming the query vector. h ),(D h Two matrices (×D) are used to mix the temporal feature channels and the object feature channels. Where D... h For latent feature dimension (D) h<D), the feature matrix after channel mixing is represented as F q,d .

[0124] After parallel feature modeling from three branches, learnable weighting is adopted to fuse the feature matrices obtained from the three branches to obtain the fused feature matrix F q,out , as follows:

[0125] F q,out =α·F q,k +β·F q,t +(1-α-β)·F q,d

[0126] Wherein, the learnable weighting parameters α and β are predicted from the query vector through linear projection.

[0127] Finally, the model performs averaging on the fused feature matrix F q,out in the key object feature dimension, projects the resulting feature into a D-dimensional vector through a linear layer, and updates the query vector through a residual structure. The query vector is input to the next layer for iterative optimization, and the update algorithm is as follows:

[0128] q'=q+Linear(Mean(F q,out ))

[0129] When performing model training in this embodiment, one-to-one matching between predictions and ground truths is implemented for training positive sample allocation, and the set of ground truth actions is added with empty labels to be expanded into a calibration set with a size comparable to that of the action prediction set. The positive sample matching cost can be expressed as:

[0130]

[0131] Wherein, σ(·) represents a valid matching between the prediction result and the ground truth result; p n represents the query point position; T(·) adopts a default local maximum-minimum transformation function to convert key frames into pseudo-temporal boxes. By minimizing the matching cost, the algorithm adopts a matching algorithm for detection tasks (such as the Hungarian algorithm) to find the optimal matching σ * (·) to allocate labels, and calculates the loss function based on this matching.

[0132] For example, given a video input, suppose the model predicts 5 time frames (i.e., 5 action instances), while there might only be 3 actual time frames (i.e., 3 action instances). However, model training requires each predicted time frame to have a corresponding ground truth box (i.e., one of the 3 ground truth boxes) to evaluate the quality of the predicted time frames. Therefore, the Hungarian matching algorithm can be used to match the 5 predicted time frames with the 3 ground truth time frames. Each predicted time frame can correspond to a ground truth time frame, allowing the model to be trained.

[0133] In this scheme, training loss Due to positioning loss and classification loss The composition, as shown in the formula:

[0134]

[0135] Among them, the positioning loss consists of the mean absolute error. The loss function is composed of the Intersection over Union (tIoU) loss function, and the formula is as follows:

[0136]

[0137] The classification loss function is calculated using the cross-entropy loss function, as shown in the following formula:

[0138]

[0139] During the prediction process, the behavior prediction results are decoded using a linear head structure. Specifically, the query vector is transformed by a combination of stacked linear layers, layer normalization, and activation functions to obtain the behavior category label, and the query point is transformed into the start and end positions of the behavior using a local maximum-minimum transformation function.

[0140] As shown above, this scheme introduces a first-person perspective temporal behavior detection model based on key object representations. It utilizes learnable query points to regress and classify the key object features extracted from each frame, thus better completing the first-person perspective temporal behavior detection task. Each query in this scheme independently predicts an action instance. Keyframes and key object labels are adaptively sampled based on the features of each query. Furthermore, based on the joint features of keyframes and key objects, a hybrid interaction module is proposed. This module adaptively models the temporal dependencies of keyframes for each query and calculates the correlation of key objects in the spatial dimension. It utilizes important semantic information from both the spatial and temporal dimensions of the video, enabling more fine-grained semantic details to be perceived in first-person perspective temporal behavior detection.

[0141] Furthermore, this solution was validated through comparative experiments on two first-person perspective temporal behavior detection benchmark datasets, Epic-Kitchens-100 and Ego4D-MQ. The Epic-Kitchens-100 dataset contains 100 hours of market video content, 20 million frames, and 90,000 behavior segments, with an average of 128 behavior annotations per video, presented in noun-verb pairs. The Ego4D-MQ dataset contains 326.4 hours of market video content, 2,488 video segments, and 22,200 behavior annotations, with labels spanning 110 fine-grained behavior classes. Compared to Epic-Kitchens-100, Ego4D-MQ features more diverse video shooting scenarios, capturing more everyday activities in both indoor and outdoor environments.

[0142] This approach uses mean average precision (mAP) as the evaluation metric. Under five different tIoU thresholds ({0.1, 0.2, 0.3, 0.4, 0.5}) and their mean values, the mAP results for BMN, G-TAD, PointTAD, VSGN, and this approach are reported. The mAP results on the Epic-Kitchens-100 dataset are shown in Table 1 below.

[0143] Table 1

[0144]

[0145] The results on the Ego4D-MQ dataset are shown in Table 2 below:

[0146] Table 2

[0147]

[0148] As shown above, on both datasets, this scheme outperforms other schemes by a significant margin across all tIoU thresholds.

[0149] Therefore, this solution introduces spatial object semantics into action detection and, based on an advanced query-based detection paradigm, adaptively samples key action objects from the input sequence using learnable action queries, significantly solving the problem of poor detection accuracy in first-person perspective scenarios in existing action detection solutions.

[0150] To better implement the above methods, this application also provides a timing behavior detection device, which can be integrated into an electronic device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0151] For example, such as Figure 5 As shown, the timing behavior detection device may include: a first determining unit 301, an acquiring unit 302, an updating unit 303, a second determining unit 304, and a third determining unit 305, as follows:

[0152] The first determining unit 301 is used to determine multiple key frames and multiple key objects from the target video frame sequence based on the action query quantity;

[0153] The acquisition unit 302 is used to acquire the first feature correlation between key objects, the second feature correlation between key frames, and the third feature correlation between key frames and key objects;

[0154] Update unit 303 is used to update the action query volume based on the first feature correlation, the second feature correlation and the third feature correlation;

[0155] The second determining unit 304 is used to redetermine multiple target keyframes from the target video frame sequence based on the updated action query quantity.

[0156] The third determining unit 305 is used to determine the start time, end time, and behavior category of the target behavior in the target video frame sequence based on the target key frame and its temporal position in the target video frame sequence.

[0157] Optionally, in some implementations, the action query quantity includes: a query vector and multiple query points; the first determining unit 301 is used for:

[0158] Based on query vectors, key objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0159] Based on the key objects, multiple query points are distributed at the corresponding temporal positions in the target video frame sequence, and the video frames at the temporal positions of the query points are determined as key frames.

[0160] Optionally, in some implementations, the update unit 303 is used for:

[0161] Update the query vector based on the correlation of the first feature, the correlation of the second feature, and the correlation of the third feature;

[0162] Determine the position offset of the query point based on the updated query vector;

[0163] Update the position distribution of the query point in the target video frame sequence based on the position offset.

[0164] Optionally, in some implementations, the first feature correlation includes a first feature matrix, the second feature correlation includes a second feature matrix, and the third feature correlation includes a third feature matrix;

[0165] When updating the query vector based on the first feature correlation, the second feature correlation, and the third feature correlation, the update unit 303 is further used for:

[0166] Based on the trained weighting parameters, the first feature matrix, the second feature matrix, and the third feature matrix are weighted respectively.

[0167] The weighted first feature matrix, second feature matrix and third feature matrix are fused to obtain the fused feature matrix;

[0168] Update the query vector based on the fused feature matrix.

[0169] Optionally, in some embodiments, the second determining unit 304 is used to:

[0170] Based on the updated location distribution of the query points, multiple target keyframes are re-determined from the target video frame sequence.

[0171] Optionally, in some implementations, when identifying key objects related to the category of the behavior to be detected from the target video frame sequence based on the query vector, the first determining unit 301 is further configured to:

[0172] Extract image features from each video frame in the target video frame sequence;

[0173] Based on the query vector and image features, candidate objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0174] Based on the sample object label set, key objects are identified from the candidate objects.

[0175] Optionally, in some embodiments, the acquisition unit 302 is used for:

[0176] Feature extraction is performed on keyframes to obtain keyframe features;

[0177] Feature extraction is performed on key objects to obtain their features;

[0178] Based on keyframe features, key object features, and query vectors, determine the first feature correlation, the second feature correlation, and the third feature correlation.

[0179] Optionally, in some embodiments, the third determining unit 305 is used to:

[0180] Convert the target keyframe into a pseudo-timing box;

[0181] Based on the pseudo-sequence box, determine the start and end times of the target behavior;

[0182] Based on the target keyframes, determine the confidence level of the target behavior across multiple categories of detected behaviors;

[0183] The category of the behavior to be detected with the highest confidence level is determined as the category of the target behavior.

[0184] As can be seen from the above, the temporal behavior detection device provided in this application, based on the action query quantity, determines multiple keyframes and multiple key objects from the target video frame sequence. Then, it updates the action query quantity by acquiring the first feature correlation between key objects, the second feature correlation between keyframes, and the third feature correlation between keyframes and key objects. Based on the updated action query quantity, multiple target keyframes are re-determined from the target video frame sequence, and the start time, end time, and behavior category of the target behavior in the target video frame sequence are determined according to the target keyframes and their temporal positions in the target video frame sequence. In this solution, spatial object semantics are introduced into behavior action detection, and by combining semantic information in the spatial and temporal dimensions, the detection accuracy of temporal behavior in first-person perspective scenes is improved.

[0185] This application also provides an electronic device, such as... Figure 6 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0186] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0187] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0188] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and timing behavior detection by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0189] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0190] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0191] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0192] Based on the action query volume, multiple key frames and multiple key objects are identified from the target video frame sequence;

[0193] Obtain the first feature correlation between key objects, the second feature correlation between key frames, and the third feature correlation between key frames and key objects;

[0194] Update the action query volume based on the correlation of the first feature, the correlation of the second feature, and the correlation of the third feature;

[0195] Based on the updated action query volume, multiple target keyframes are re-determined from the target video frame sequence;

[0196] Based on the target keyframes and their temporal positions in the target video frame sequence, determine the start time, end time, and behavior category of the target behavior in the target video frame sequence.

[0197] In some implementations, the action query quantity includes: a query vector and multiple query points; based on the action query quantity. When determining multiple keyframes and multiple key objects from a target video frame sequence, the processor 401 is used to:

[0198] Based on query vectors, key objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0199] Based on the key objects, multiple query points are distributed at the corresponding temporal positions in the target video frame sequence, and the video frames at the temporal positions of the query points are determined as key frames.

[0200] In some implementations, when updating the action query volume based on the first feature correlation, the second feature correlation, and the third feature correlation, the processor 401 is used to:

[0201] Update the query vector based on the correlation of the first feature, the correlation of the second feature, and the correlation of the third feature;

[0202] Determine the position offset of the query point based on the updated query vector;

[0203] Update the position distribution of the query point in the target video frame sequence based on the position offset.

[0204] In some implementations, the first feature correlation includes a first feature matrix, the second feature correlation includes a second feature matrix, and the third feature correlation includes a third feature matrix;

[0205] When updating the query vector based on the first feature correlation, the second feature correlation, and the third feature correlation, processor 401 is used to:

[0206] Based on the trained weighting parameters, the first feature matrix, the second feature matrix, and the third feature matrix are weighted respectively.

[0207] The weighted first feature matrix, second feature matrix and third feature matrix are fused to obtain the fused feature matrix;

[0208] Update the query vector based on the fused feature matrix.

[0209] In some implementations, the processor 401 specifically performs the following: Based on the updated action query, multiple target keyframes are re-determined from the target video frame sequence.

[0210] Based on the updated location distribution of the query points, multiple target keyframes are re-determined from the target video frame sequence.

[0211] In some implementations, when identifying key objects related to the category of the behavior to be detected from a sequence of target video frames based on a query vector, the processor 401 is used to:

[0212] Extract image features from each video frame in the target video frame sequence;

[0213] Based on the query vector and image features, candidate objects related to the category of the behavior to be detected are identified from the target video frame sequence;

[0214] Based on the sample object label set, key objects are identified from the candidate objects.

[0215] In some implementations, when acquiring the first feature correlation between key objects, the second feature correlation between key frames, and the third feature correlation between key frames and key objects, the processor 401 is specifically used to:

[0216] Feature extraction is performed on keyframes to obtain keyframe features;

[0217] Feature extraction is performed on key objects to obtain their features;

[0218] Based on keyframe features, key object features, and query vectors, determine the first feature correlation, the second feature correlation, and the third feature correlation.

[0219] In some implementations, when determining the start time, end time, and behavior category of a target behavior in a target video frame sequence based on the target keyframe and its temporal position in the target video frame sequence, the processor 401 is used to:

[0220] Convert the target keyframe into a pseudo-timing box;

[0221] Based on the pseudo-sequence box, determine the start and end times of the target behavior;

[0222] Based on the target keyframes, determine the confidence level of the target behavior across multiple categories of detected behaviors;

[0223] The category of the behavior to be detected with the highest confidence level is determined as the category of the target behavior.

[0224] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0225] As described above, in this embodiment, the electronic device determines multiple keyframes and multiple key objects from the target video frame sequence based on the action query quantity. Then, it updates the action query quantity by acquiring the first feature correlation between key objects, the second feature correlation between keyframes, and the third feature correlation between keyframes and key objects. Based on the updated action query quantity, multiple target keyframes are re-determined from the target video frame sequence. Furthermore, based on the target keyframes and their temporal positions within the target video frame sequence, the start time, end time, and behavior category of the target behavior in the target video frame sequence are determined. This solution introduces spatial object semantics into behavior detection, combining semantic information in both spatial and temporal dimensions to improve the detection accuracy of temporal behaviors in first-person perspective scenarios.

[0226] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0227] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the timing behavior detection methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0228] Based on the action query volume, multiple key frames and multiple key objects are identified from the target video frame sequence;

[0229] Obtain the first feature correlation between key objects, the second feature correlation between key frames, and the third feature correlation between key frames and key objects;

[0230] Update the action query volume based on the correlation of the first feature, the correlation of the second feature, and the correlation of the third feature;

[0231] Based on the updated action query volume, multiple target keyframes are re-determined from the target video frame sequence;

[0232] Based on the target keyframes and their temporal positions in the target video frame sequence, determine the start time, end time, and behavior category of the target behavior in the target video frame sequence.

[0233] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0234] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0235] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the timing behavior detection methods provided in the embodiments of this application, the beneficial effects that any of the timing behavior detection methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0236] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the aforementioned timing behavior detection.

[0237] The foregoing has provided a detailed description of a timing behavior detection method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting temporal behavior, characterized in that, include: Based on the action query volume, multiple key frames and multiple key objects are identified from the target video frame sequence; The process involves obtaining a first feature correlation between key objects, a second feature correlation between key frames, and a third feature correlation between key frames and key objects. Key frame features and key object features are feature matrices, which are concatenated along the channel dimension to form a joint feature matrix. The first feature correlation is calculated by performing cross-attention between all key objects and the query vector to determine the relevance of each object label. A counting matrix is ​​calculated for all object labels, and its reciprocal is used as a normalized mask. The mask is multiplied by the cross-attention map to obtain the corresponding feature matrix, which serves as the first feature correlation. The second feature correlation is generated by projecting the query vector through a linear layer to produce an N_p × N_p weight matrix, which is used as a parameter for dynamic mixing to model the temporal dependency of the feature matrix, resulting in a temporally mixed feature matrix, which serves as the second feature correlation. The third feature correlation is achieved by using a dynamic mixing model where the weight matrix is ​​of size (D × D_h) obtained after transforming the query vector. Given two matrices (D_h×D) and (D_h×D), multiply the joint feature matrix with the two transformed matrices respectively, and output the corresponding matrix as the third feature correlation; The action query volume is updated based on the first feature correlation, the second feature correlation, and the third feature correlation. Based on the updated action query volume, multiple target keyframes are re-determined from the target video frame sequence; Based on the target keyframe and its temporal position in the target video frame sequence, determine the start time, end time, and behavior category of the target behavior in the target video frame sequence.

2. The temporal behavior detection method according to claim 1, characterized in that, The action query quantity includes: a query vector and multiple query points; the determination of multiple keyframes and multiple key objects from the target video frame sequence based on the action query quantity includes: Based on the query vector, key objects related to the category of the behavior to be detected are identified from the target video frame sequence; Based on the key object, multiple query points are distributed at corresponding temporal positions in the target video frame sequence, and the video frame at the temporal position of the query point is determined as the key frame.

3. The temporal behavior detection method according to claim 2, characterized in that, The step of updating the action query volume based on the first feature correlation, the second feature correlation, and the third feature correlation includes: The query vector is updated based on the first feature correlation, the second feature correlation, and the third feature correlation. Based on the updated query vector, determine the position offset of the query point; The position distribution of the query point in the target video frame sequence is updated based on the position offset.

4. The temporal behavior detection method according to claim 3, characterized in that, The first feature correlation includes a first feature matrix, the second feature correlation includes a second feature matrix, and the third feature correlation includes a third feature matrix; The step of updating the query vector based on the first feature correlation, the second feature correlation, and the third feature correlation includes: Based on the trained weighting parameters, the first feature matrix, the second feature matrix, and the third feature matrix are weighted respectively. The weighted first feature matrix, the second feature matrix, and the third feature matrix are fused to obtain a fused feature matrix. The query vector is updated based on the fused feature matrix.

5. The temporal behavior detection method according to claim 3, characterized in that, The process of re-determining multiple target keyframes from the target video frame sequence based on the updated action query data includes: Based on the updated location distribution of the query points, multiple target keyframes are re-determined from the target video frame sequence.

6. The temporal behavior detection method according to claim 2, characterized in that, The step of identifying key objects related to the category of the behavior to be detected from the target video frame sequence based on the query vector includes: Extract image features from each video frame in the target video frame sequence; Based on the query vector and the image features, candidate objects related to the category of the behavior to be detected are identified from the target video frame sequence; The key object is determined from the candidate objects based on the sample object label set.

7. The temporal behavior detection method according to claim 1, characterized in that, The step of obtaining the first feature correlation between the key objects, the second feature correlation between the key frames, and the third feature correlation between the key frames and the key objects includes: Feature extraction is performed on the keyframes to obtain keyframe features; Feature extraction is performed on the key object to obtain the key object features; Based on the keyframe features, the key object features, and the query vector, the first feature correlation, the second feature correlation, and the third feature correlation are determined.

8. The temporal behavior detection method according to any one of claims 1-7, characterized in that, The step of determining the start time, end time, and behavior category of the target behavior in the target video frame sequence based on the target keyframe and its temporal position in the target video frame sequence includes: Convert the target keyframe into a pseudo-timing box; Based on the pseudo-timing frame, determine the start and end times of the target behavior; Based on the target keyframe, determine the confidence level of the target behavior across multiple categories of behaviors to be detected; The category of the behavior to be detected with the highest confidence level is determined as the behavior category of the target behavior.

9. A temporal behavior detection device, characterized in that, include: The first determining unit is used to determine multiple key frames and multiple key objects from the target video frame sequence based on the action query volume. An acquisition unit is used to acquire the first feature correlation between the key objects, the second feature correlation between the key frames, and the third feature correlation between the key frames and the key objects; wherein, the key frame features and key object features are feature matrices, and the key frame features and key object features are concatenated along the channel dimension to form a joint feature matrix; the first feature correlation is: the correlation of each object label is calculated by performing cross attention between all key objects and the query vector; the counting matrix is ​​calculated for all object labels, its reciprocal matrix is ​​calculated as a normalization mask, and the mask is multiplied by the cross attention map to obtain the corresponding feature matrix, which is the first feature correlation; the second feature correlation is: the query vector is projected through a linear layer to generate an N_p×N_p weight matrix, which is used as a parameter for dynamic mixing to model the temporal dependency of the feature matrix, resulting in a temporally mixed feature matrix, which is the second feature correlation; the third feature correlation is: a dynamic mixing model is adopted, and the weight matrix is ​​of size (D×D_h) obtained after transforming the query vector. Given two matrices (D_h×D) and (D_h×D), multiply the joint feature matrix with the two transformed matrices respectively, and output the corresponding matrix as the third feature correlation; The update unit is used to update the action query volume based on the first feature correlation, the second feature correlation, and the third feature correlation. The second determining unit is used to redetermine multiple target keyframes from the target video frame sequence based on the updated action query volume; The third determining unit is used to determine the start time, end time, and behavior category of the target behavior in the target video frame sequence based on the target keyframe and its temporal position in the target video frame sequence.

10. The timing behavior detection device according to claim 9, characterized in that, The action query quantity includes: a query vector and multiple query points; the first determining unit is used for: Based on the query vector, key objects related to the category of the behavior to be detected are identified from the target video frame sequence; Based on the key object, multiple query points are distributed at corresponding temporal positions in the target video frame sequence, and the video frame at the temporal position of the query point is determined as the key frame.

11. The timing behavior detection device according to claim 9, characterized in that, The update unit is used for: The query vector is updated based on the first feature correlation, the second feature correlation, and the third feature correlation. Based on the updated query vector, determine the position offset of the query point; The position distribution of the query point in the target video frame sequence is updated based on the position offset.

12. The timing behavior detection device according to claim 9, characterized in that, The third determining unit is used for: Convert the target keyframe into a pseudo-timing box; Based on the pseudo-timing frame, determine the start and end times of the target behavior; Based on the target keyframe, determine the confidence level of the target behavior across multiple categories of behaviors to be detected; The category of the behavior to be detected with the highest confidence level is determined as the behavior category of the target behavior.

13. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program in the memory to perform the steps in the timing behavior detection method according to any one of claims 1-8.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the timing behavior detection method according to any one of claims 1-8.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps in the timing behavior detection method according to any one of claims 1-8.

Images