Time action positioning method and device based on long memory transformer

Abstract

Embodiments of the present disclosure provide a long memory transformer-based time action positioning method and device. Applied to the field of time action positioning; the method comprises: clipping a target video to obtain a short video set; randomly selecting one or more short videos in the short video set and inputting them into a pre-trained short-term Transformer encoder to obtain the features of the randomly selected one or more short videos; using a long memory module to sample from long-term memory to obtain the features of the short videos in the short video set that have not been selected; inputting the features of the randomly selected one or more short videos and the features of the short videos that have not been selected into a time fusion module to obtain fused video features; and inputting the fused video features into a time boundary positioning module to obtain four-tuples of each action instance. In this way, the speed of time action positioning is improved, the memory consumption of processing resources and the training time are reduced, and faster prediction speed and better action classification accuracy are achieved.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and more particularly to the field of temporal motion localization based on long-memory transformers. Background Technology

[0002] Action recognition is a fundamental short-term modeling task in video understanding. With the success of deep learning, numerous methods have achieved impressive performance benchmarks on standard action recognition using 2D and 3D CNNs. Recently, VisionTransformer-based methods have proven to significantly outperform previous CNN-based approaches. Due to large-scale pre-training on action recognition datasets, pre-trained models in this field are widely used as short-term feature extractors for temporal action localization. A limitation of modern video converter models is their slow training speed and high memory requirements due to the quadratic memory complexity of self-attention. Therefore, it is difficult to apply them to long-term modeling tasks such as temporal action localization.

[0003] Temporal Action Localization (TAL) is used to output a quadruple of action instances (action start timestamp, action end timestamp, action class, and probability of the instance) from video features. Due to limited processing resources and memory, most existing methods use pre-extracted action recognition features as input to the TAL model. However, since these features are extracted using models pre-trained on different datasets, using these features for TAL often leads to poor performance. To address these issues, recent methods AFSD and DaoTAD have proposed end-to-end trainable frameworks. However, to accommodate limited processing resources and memory, these models operate at very low spatial video resolutions (e.g., 96×96 and 112×112, respectively), which results in a significant drop in TAL accuracy. To our knowledge, no existing method can simultaneously achieve high spatial resolution and long-term end-to-end training. This proposal addresses this problem by suggesting a simple, end-to-end trainable, transformer-based TAL approach. Summary of the Invention

[0004] This disclosure provides a method and apparatus for time-based motion localization based on a long-memory transformer.

[0005] According to a first aspect of this disclosure, a method for temporal action localization based on a long-memory transformer is provided, comprising:

[0006] Edit the target video to obtain a collection of short videos;

[0007] One or more short videos are randomly selected from the short video set and input into a pre-trained short-term Transformer encoder to obtain the features of the randomly selected one or more short videos.

[0008] Features of unselected short videos in the short video set are obtained by sampling from long-term memory using a long memory module;

[0009] The features of one or more randomly selected short videos and the features of the unselected short videos are input into the time fusion module to obtain the fused video features.

[0010] The fused video features are input into the temporal boundary localization module to obtain a quadruple for each action instance; wherein, the quadruple for each action instance is the action start timestamp, the action end timestamp, the action category, and the probability of the action instance.

[0011] In some implementations of the first aspect, the long-term memory includes a pre-trained transformer encoder extracting features from each short video in the short video set to obtain features of all frames of each short video.

[0012] In some implementations of the first aspect, the step of inputting the features of the randomly selected one or more short videos and the features of the unselected short videos into the time fusion module includes:

[0013] The features of one or more randomly selected short videos and the features of the unselected short videos are concatenated through feature channels to obtain the concatenated video features.

[0014] The spliced ​​video features are input into the time fusion module.

[0015] In some implementations of the first aspect, the step of inputting the fused video features into the temporal boundary localization module to obtain the quadruplets of each action instance includes:

[0016] The fused video features are input into the temporal boundary localization module to obtain the start time offset value, end time offset value, and probability of each action category corresponding to each action instance for each time frame; wherein, the temporal boundary localization module is a linear action recognition classifier with shared parameters.

[0017] The start time offset value and end time offset value of each time anchor frame are added to its start timestamp and end timestamp respectively to obtain the action start timestamp and action end timestamp of the corresponding action instance;

[0018] The probabilities of each action category corresponding to each action instance are sorted, and the action category with the highest probability is selected as the action category of the corresponding action instance. The probability of the action category with the highest probability is then used as the probability of the corresponding action instance.

[0019] In some implementations of the first aspect, the size of each time anchor frame is generated using an action time anchor frame generator based on a preset time scale ratio.

[0020] In some implementations of the first aspect, the short-term Transformer encoder is trained using the following method:

[0021] Obtain a sample set, wherein the sample set includes a collection of short videos corresponding to each video;

[0022] One or more short videos are randomly selected from the short video set and input into a short-term Transformer encoder to obtain the features of the randomly selected one or more short videos;

[0023] Features of unselected short videos in the short video set are obtained by sampling from long-term memory using a long memory module;

[0024] The features of one or more randomly selected short videos and the features of the unselected short videos are input into the time fusion module to obtain the fused video features.

[0025] The fused video features are input into the temporal boundary localization module to obtain quadruplets for each action instance;

[0026] Based on the action start timestamp, action end timestamp, and action category label in the quadruple of each action instance, calculate the loss value and update the parameters in the short-term Transformer encoder.

[0027] The parameters in the short-term Transformer encoder are continuously updated iteratively until the preset number of iterations is reached.

[0028] In some implementations of the first aspect, the method further includes:

[0029] During each iteration of the short-term Transformer encoder, the long memory module caches features of one or more randomly selected short videos obtained by the short-term Transformer encoder and stores the cached features of one or more randomly selected short videos in the long memory to update the long memory.

[0030] According to a second aspect of this disclosure, a time-action positioning device based on a long-memory transformer is provided, the device comprising:

[0031] The video editing module is used to edit the target video to obtain a collection of short videos;

[0032] The feature extraction module is used to randomly select one or more short videos from the short video set, input them into a pre-trained short-term Transformer encoder, and obtain the features of the randomly selected one or more short videos.

[0033] The feature sampling module is used to sample features of unselected short videos from the short video set using the long memory module;

[0034] The feature fusion module is used to input the features of one or more randomly selected short videos and the features of the unselected short videos into the time fusion module to obtain the fused video features;

[0035] The time-action localization module is used to input the fused video features into the time boundary localization module to obtain a quadruple for each action instance; wherein, the quadruple for each action instance is the action start timestamp, the action end timestamp, the action category, and the probability of the action instance.

[0036] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0037] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0038] This disclosure first edits the target video to obtain a set of short videos; then, it randomly selects one or more short videos from the set and inputs them into a pre-trained short-term Transformer encoder to obtain features from the randomly selected short videos; it then uses a long memory module to sample features from long-term memory for the unselected short videos in the set; next, it inputs the features of the randomly selected short videos and the unselected short videos into a temporal fusion module to obtain fused video features; finally, it inputs the fused video features into a temporal boundary localization module to obtain quadruplets for each action instance. This approach improves the speed of action localization to some extent; it solves the problem of excessive memory consumption in current temporal action localization methods, significantly reducing processing resource memory consumption and training time; and it addresses the problem of current temporal action localization methods relying on external classifiers by using a shared linear action recognition classifier, thus eliminating the dependence on external classifiers and improving generalization ability and inference speed.

[0039] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0040] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0041] Figure 1 A flowchart of a time-motion localization method based on a long-memory transformer according to an embodiment of the present disclosure is shown;

[0042] Figure 2 A block diagram of a time-motion positioning device based on a long-memory transformer according to an embodiment of the present disclosure is shown;

[0043] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0045] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0046] In this disclosure, a set of short videos is obtained by editing the target video; then, one or more short videos are randomly selected from the set and input into a pre-trained short-term Transformer encoder to obtain features of the randomly selected short videos; features of the unselected short videos in the set are sampled from long-term memory using a long memory module; next, the features of the randomly selected short videos and the unselected short videos are input into a temporal fusion module to obtain fused video features; finally, the fused video features are input into a temporal boundary localization module to obtain quadruplets for each action instance. This approach improves the speed of action localization to a certain extent; solves the problem of excessive memory consumption in current temporal action localization methods, significantly reducing processing resource memory consumption and training time; and solves the problem of current temporal action localization methods relying on external classifiers by using a shared linear action recognition classifier, eliminating the dependence on external classifiers and improving generalization ability and inference speed.

[0047] Figure 1 A flowchart of a time-motion localization method based on a long-memory transformer according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the time-motion positioning method 100 includes:

[0048] S101: Edit the target video to obtain a collection of short videos;

[0049] S102: Randomly select one or more short videos from the short video set and input them into a pre-trained short-term Transformer encoder to obtain the features of the randomly selected one or more short videos;

[0050] S103: Use the long memory module to sample the features of the short videos that were not selected in the short video set from the long memory;

[0051] S104: Input the features of one or more randomly selected short videos and the features of the unselected short videos into the time fusion module to obtain the fused video features;

[0052] S105: Input the fused video features into the temporal boundary localization module to obtain a quadruple for each action instance; wherein, the quadruple for each action instance is the action start timestamp, the action end timestamp, the action category, and the probability of the action instance.

[0053] In S101, the target video is edited to obtain a collection of short videos.

[0054] In some implementations, considering that the undressed long video contains multiple short videos with consecutive actions, and in order to avoid computing the features of each short video, a fixed number of short videos are randomly selected and input into a pre-trained short-term Transformer encoder. Therefore, the target video needs to be edited before feature extraction to obtain a set of short videos consisting of multiple short videos.

[0055] In S102, one or more short videos are randomly selected from the short video set and input into a pre-trained short-term Transformer encoder to obtain the features of the randomly selected one or more short videos.

[0056] In some implementations, to avoid computing features for each short video, a fixed number of short videos are randomly selected and input into a pre-trained short-term Transformer encoder to obtain features for the randomly selected short videos.

[0057] In some implementations, the short-term Transformer encoder is trained using the following method: First, a sample set is acquired, which includes multiple sets of short videos obtained by editing multiple videos. Then, one or more short videos from the short video set are randomly selected and input into the short-term Transformer encoder to obtain features of the randomly selected short videos. Features of the unselected short videos in the short video set are sampled from long-term memory using a long memory module. Next, the features of the randomly selected short videos and the features of the unselected short videos are input into a temporal fusion module to obtain fused video features. The fused video features are input into a temporal boundary localization module to obtain quadruplets for each action instance. Finally, based on the action start timestamp, action end timestamp, and action category label in the quadruplets of each action instance, the loss value of the short-term Transformer encoder is calculated, and the parameters in the short-term Transformer encoder are updated using a traditional gradient backpropagation mechanism. The parameters in the short-term Transformer encoder are then iteratively updated until a preset number of iterations is reached.

[0058] Optionally, a short-term transformer encoder can use VideoSwin.

[0059] In some implementations, before training the short-term Transformer encoder, a Transformer encoder pre-trained on other datasets is used to extract features from each short video sample in the short video set, and then the features of all frames of each short video in the short video set are cached in the device memory, i.e., long-term memory.

[0060] In addition, during each iteration of the Short-Term Transformer encoder, the Long Memory module caches features of one or more randomly selected short videos obtained by the Short-Term Transformer encoder and stores the cached features of one or more randomly selected short videos into the Long Memory to update the Long Memory.

[0061] According to embodiments of this disclosure, the temporal action localization method of this disclosure can be trained end-to-end on long, high spatial resolution videos without reducing the spatial video resolution, freezing the backbone, or resorting to weak short-term feature extraction of the backbone.

[0062] In S103, the long memory module is used to sample the features of the short videos that were not selected in the short video set from the long memory.

[0063] In some implementations, the long memory module enables the method of this disclosure to train on long and high-resolution videos. Specifically, the long memory module adopts the same model structure as the short-term Transformer encoder. On the one hand, it can cache the video features of one or more randomly selected short videos obtained by feature extraction from the short-term Transformer encoder, and store the cached video features of one or more randomly selected short videos in long memory to update the long memory. On the other hand, it can sample the video features of short videos that were not selected in the short video set from the long memory.

[0064] In some implementations, long-term memory stores features of all frames of each short video obtained by feature extraction of each short video in the short video set by a pre-trained transformer encoder.

[0065] According to embodiments of this disclosure, by utilizing long memory modules and long-term memory, end-to-end trainability, low memory consumption, and high computation speed can be achieved, eliminating the need to process hundreds of redundant video frames during each training session, thereby significantly reducing processing resource memory consumption and training time, and saving memory and computation costs.

[0066] In S104, the features of one or more randomly selected short videos and the features of the unselected short videos are input into the time fusion module to obtain the fused video features.

[0067] In some implementations, inputting the features of the randomly selected one or more short videos and the features of the unselected short videos into the time fusion module includes: splicing the features of the randomly selected one or more short videos and the features of the unselected short videos through a feature channel to obtain spliced ​​video features; and inputting the spliced ​​video features into the time fusion module.

[0068] As we know, channel concatenation refers to concatenating two tensors along the feature channel direction. Concatenating video features obtained through a short-term Transformer encoder and video features sampled from long-term memory merges the feature channels of the two video features, thereby increasing the number of features and enriching the feature representation. Simultaneously, concatenating along the feature channel direction preserves the spatial dimension of subsequent input video features, which is crucial for the convolutional neural network used later, as convolutional neural networks require filtering operations in the spatial dimension.

[0069] Specifically, you can use the `concatenate` function or the `torch.cat` function to concatenate two tensors along a specified axis. In this case, you need to use the channel dimension of the tensor as the concatenation axis.

[0070] According to embodiments of this disclosure, concatenating two video features can increase the number of features, enrich the representation of features, and keep the spatial dimension of the feature tensor unchanged, thereby improving the expressive power, accuracy, and robustness of the model used subsequently.

[0071] Furthermore, the spliced ​​video features are input into the time fusion module to obtain the fused video features.

[0072] In some implementations, the temporal fusion module is a convolutional neural network. Video features typically consist of multiple temporally ordered image frames, so 3D convolutional neural networks (3D-CNNs) or temporal convolutional neural networks (TCNs) can be used to process video features. Feature fusion using convolutional neural networks includes two implementation methods: bypass connections and multi-branch networks.

[0073] Bypass connections introduce cross-layer connections into neural networks, directly adding or concatenating the output of one layer with the input of a subsequent layer, thereby preserving more feature information. The most commonly used type of bypass connection is the residual connection, which directly adds the input to the output, allowing the network to more easily learn residual information, thus improving network performance.

[0074] Multi-branch networks incorporate multiple branches, each specializing in processing a specific type of feature. The outputs of all branches are then fused together. For example, in object detection tasks, images can be fed into different convolutional branches to extract features at different scales. These features can then be concatenated or weighted to obtain more accurate detection results. Multi-branch networks can more effectively utilize diverse feature information, improving model performance.

[0075] In terms of implementation, in order to ensure the effect of feature fusion, some hyperparameters need to be adjusted, such as the concatenation dimension and weight settings.

[0076] According to embodiments of this disclosure, by fusing video features, the performance of the model used subsequently and the accuracy of classification can be improved.

[0077] In S105, the fused video features are input into the temporal boundary localization module to obtain a quadruple for each action instance; wherein, the quadruple for each action instance is the action start timestamp, the action end timestamp, the action category, and the probability of the action instance.

[0078] In some implementations, inputting the fused video features into the temporal boundary localization module to obtain the quadruple of each action instance includes: inputting the fused video features into the temporal boundary localization module to obtain the start time offset value, end time offset value, and probability of each action category corresponding to each action instance for each time frame; wherein, the temporal boundary localization module is a linear action recognition classifier with shared parameters; adding the start time offset value and end time offset value of each time frame to its start timestamp and end timestamp respectively to obtain the action start timestamp and action end timestamp of the corresponding action instance; sorting the probabilities of each action category corresponding to each action instance, selecting the action category with the highest probability as the action category of the corresponding action instance, and using the probability corresponding to the action category with the highest probability as the probability of the corresponding action instance.

[0079] In some implementations, the dimensions of each time anchor frame are generated using an action time anchor frame generator based on a preset time scale ratio.

[0080] Specifically, the action time anchor frame generator generates multiple time anchor frames (preset action proposals) of different fixed sizes (usually 3 to 6, depending on the actual situation) based on the preset time scale ratio (i.e., the time window length divided by the total time length) and centered on each time point.

[0081] Understandably, the preset time anchor frame is generally composed of small-scale frame, medium-scale frame, and large-scale frame (small, medium, and large scales usually have one or more set frames). In this way, the preset time anchor frame can cover every part of the video as much as possible.

[0082] In some implementations, multiple pre-generated time anchor boxes can serve as prior information inputs to the time boundary localization module, providing prior information and thus reducing the training difficulty of the time boundary localization module. When the fused video features are input into the time boundary localization module, corresponding action instances can be matched based on the pre-generated multiple time anchor boxes.

[0083] Next, after the fused video features are input into the temporal boundary localization module, the temporal boundary localization module will predict the start time offset value, end time offset value, and probability of each action category corresponding to each action instance for each time anchor box.

[0084] Specifically, the time boundary localization module is a linear action recognition classifier with shared parameters. The output dimension of this classifier is: T*N*(2+C), where T is the total time length, N is the number of preset time anchor boxes at each time point, C is the number of action categories, and 2 is the dimension occupied by the start time offset value and the end time offset value.

[0085] Furthermore, the start and end time offset values ​​of each time anchor frame are added to their respective start and end timestamps to obtain the action start and end timestamps of the corresponding action instance. Since multiple time anchor frames of different sizes can match the corresponding action instances, the start and end time offset values ​​of the time anchor frames are added to their respective start and end timestamps to obtain the action start and end timestamps of the corresponding action instance.

[0086] The probabilities of each action category corresponding to each action instance are sorted, and the action category with the highest probability is selected as the action category of the corresponding action instance. The probability of the action category with the highest probability is then used as the probability of the corresponding action instance. Here, the action category is a pre-defined set of categories. The temporal boundary localization module predicts the probability for each action category for each action instance. For example, the category set includes three categories: Category 1, Category 2, and Category 3. For a given action instance, the predicted probability of it being Category 1 is 30%, Category 2 is 50%, and Category 3 is 20%. These three probabilities are sorted, and the category with the highest probability is selected as Category 2, which is then used as the action category for that action instance.

[0087] According to embodiments of this disclosure, combining a linear action recognition classifier with shared parameters and an action time anchor box generator achieves faster prediction speed, better action classification accuracy, and eliminates the need for an external action recognition classifier.

[0088] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0089] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0090] Figure 2A block diagram of a time-motion positioning device based on a long-memory transformer according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, the time-motion positioning device 200 includes:

[0091] The video editing module 201 is used to edit the target video to obtain a collection of short videos;

[0092] The feature extraction module 202 is used to randomly select one or more short videos from the short video set, input them into a pre-trained short-term Transformer encoder, and obtain the features of the randomly selected one or more short videos.

[0093] The feature sampling module 203 is used to sample the features of the short videos that were not selected in the short video set from the long memory using the long memory module;

[0094] The feature fusion module 204 is used to input the features of one or more randomly selected short videos and the features of the unselected short videos into the time fusion module to obtain fused video features;

[0095] The time-action localization module 205 is used to input the fused video features into the time boundary localization module to obtain a quadruple for each action instance; wherein, the quadruple for each action instance is the action start timestamp, the action end timestamp, the action category, and the probability of the action instance.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0098] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0099] Electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in ROM 302 or a computer program loaded into RAM 303 from storage unit 308. RAM 303 can also store various programs and data required for the operation of electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O interface 305 is also connected to bus 304.

[0100] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0102] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0107] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0108] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A temporal action localization method based on long-memory transformer, characterized in that, The method includes: Edit the target video to obtain a collection of short videos; One or more short videos are randomly selected from the short video set and input into a pre-trained short-term Transformer encoder to obtain the features of the randomly selected one or more short videos. Features of unselected short videos in the short video set are obtained by sampling from long-term memory using a long memory module; The features of one or more randomly selected short videos and the features of the unselected short videos are input into the time fusion module to obtain the fused video features. The fused video features are input into the temporal boundary localization module to obtain a quadruple for each action instance; wherein, the quadruple for each action instance is the action start timestamp, the action end timestamp, the action category, and the probability of the action instance.

2. The method according to claim 1, characterized in that, The long-term memory includes features extracted from each short video in the short video set by a pre-trained transformer encoder, resulting in features of all frames of each short video.

3. The method according to claim 1, characterized in that, The step of inputting the features of one or more randomly selected short videos and the features of the unselected short videos into the time fusion module includes: The features of one or more randomly selected short videos and the features of the unselected short videos are concatenated through feature channels to obtain the concatenated video features. The spliced ​​video features are input into the time fusion module.

4. The method according to claim 1, characterized in that, The step of inputting the fused video features into the temporal boundary localization module to obtain the quadruplets for each action instance includes: The fused video features are input into the temporal boundary localization module to obtain the start time offset value, end time offset value, and probability of each action category corresponding to each action instance for each time frame; wherein, the temporal boundary localization module is a linear action recognition classifier with shared parameters. The start time offset value and end time offset value of each time anchor frame are added to its start timestamp and end timestamp respectively to obtain the action start timestamp and action end timestamp of the corresponding action instance; The probabilities of each action category corresponding to each action instance are sorted, and the action category with the highest probability is selected as the action category of the corresponding action instance. The probability of the action category with the highest probability is then used as the probability of the corresponding action instance.

5. The method according to claim 4, characterized in that, The dimensions of each time anchor frame are generated using an action time anchor frame generator based on a preset time scale ratio.

6. The method according to claim 1, characterized in that, The short-term Transformer encoder was trained using the following method: Obtain a sample set, wherein the sample set includes a collection of short videos corresponding to each video; One or more short videos are randomly selected from the short video set and input into a short-term Transformer encoder to obtain the features of the randomly selected one or more short videos; Features of unselected short videos in the short video set are obtained by sampling from long-term memory using a long memory module; The features of one or more randomly selected short videos and the features of the unselected short videos are input into the time fusion module to obtain the fused video features. The fused video features are input into the temporal boundary localization module to obtain quadruplets for each action instance; Based on the action start timestamp, action end timestamp, and action category label in the quadruple of each action instance, calculate the loss value and update the parameters in the short-term Transformer encoder. The parameters in the short-term Transformer encoder are continuously updated iteratively until the preset number of iterations is reached.

7. The method according to claim 6, characterized in that, The method further includes: During each iteration of the short-term Transformer encoder, the long memory module caches features of one or more randomly selected short videos obtained by the short-term Transformer encoder and stores the cached features of one or more randomly selected short videos in the long memory to update the long memory.

8. A time-motion positioning device based on a long-memory transformer, characterized in that, The device includes: The video editing module is used to edit the target video to obtain a collection of short videos; The feature extraction module is used to randomly select one or more short videos from the short video set, input them into a pre-trained short-term Transformer encoder, and obtain the features of the randomly selected one or more short videos. The feature sampling module is used to sample features of the short videos that were not selected in the short video set from long memory using the long memory module; The feature fusion module is used to input the features of one or more randomly selected short videos and the features of the unselected short videos into the time fusion module to obtain the fused video features; The time-action localization module is used to input the fused video features into the time boundary localization module to obtain a quadruple for each action instance; wherein, the quadruple for each action instance is the action start timestamp, the action end timestamp, the action category, and the probability of the action instance.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory is characterized in that it stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images