Motion recognition method and device of video, electronic equipment and readable medium

Abstract

The present disclosure provides a motion recognition method and device for a video, an electronic device and a readable medium, wherein the motion recognition method for the video comprises: in response to a received motion recognition task, encoding adjacent video window features collected by two specified models through a transformer model embedded with an NWA module; and determining the weight of the video window features on the motion recognition task according to the encoding result. Through the embodiments of the present disclosure, the reliability and accuracy of motion recognition for the video can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of action recognition, in particular to a video action recognition method and device, an electronic device and a readable medium. BACKGROUND

[0002] Currently, temporal action localization (TAL), also known as temporal action detection, is an important field of video understanding.

[0003] In the related art, a video is usually untrimmed, the length of the video is long, and the start and end times of actions are not equal, some actions only occur in a very short time in the video, in addition, the video may contain multiple actions or no action at all, and the amount of information contained is huge

[0004] However, since TAL is considered to be composed of two sub-tasks, one is to predict the start and end time interval of the action, and the other is to predict the category of the action, the actions in a long video may have very different lengths, so it is difficult to capture them with a fixed window or block.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the present disclosure is to provide a video action recognition method and device, an electronic device and a readable medium, which at least partially overcome the problem of poor accuracy of video action recognition due to the limitations and defects of the related art.

[0007] According to a first aspect of an embodiment of the present disclosure, a video action recognition method is provided, comprising: in response to a received action recognition task, encoding adjacent video window features collected by two specified models through a transformer model embedded with an NWA module; and determining the weight of the video window features on the action recognition task according to the encoding result.

[0008] In an exemplary embodiment of the present disclosure, before responding to the received action recognition task, the method further comprises:

[0009] extracting specified frames from the original video at a preset interval, and performing image enhancement on the specified frames;

[0010] extracting video window features of the specified frames subjected to image enhancement and having position encoding through a slowfast model, and recording the first features;

[0011] extracting, by a timesformer model, a video window feature of a specified frame subjected to image enhancement and with position encoding, and denoted as a second feature,

[0012] wherein the specified model comprises the slowfast model and / or the timesformer model, and the video window feature comprises the first feature and / or the second feature.

[0013] In an exemplary embodiment of the present disclosure, further comprising:

[0014] determining a network predicted action label and an actual action label;

[0015] determining an action start time label and an action end time label;

[0016] constructing a loss function of the transformer model according to a preset loss weight, the network predicted action label, the actual action label, the action start time label and the action end time label.

[0017] In an exemplary embodiment of the present disclosure, further comprising:

[0018] assuming that an optimal solution of the loss function is smooth around the optimal solution and the optimal solution can be parameterized as t;

[0019] deriving the t within a specified range to determine the optimal solution according to a derivation result.

[0020] In an exemplary embodiment of the present disclosure, an expression of the loss function comprises:

[0021] L(X)=α i (L cls (Y C ,Y GC )+L Tbox (Y Tbox ,Y GTbox ))+β i L p ),

[0022] wherein the X represents an input feature, the Y C represents a network predicted action label, the Y GC represents an actual action label, the Y Tbox represents a network predicted start time label and an end time label of a recognized action, the Y GTbox represents an actual start time label and an end time label of an action, the α i and the β i represent weight parameters of the loss function, and L CLSa network classification loss function, L Tbox a network regression loss function, L p a gradient loss output by the network NWA module.

[0023] In an example embodiment of the present disclosure, in response to the received action recognition task, encoding, by a transformer model embedded with the NWA module, adjacent video window features collected by two specified models comprises:

[0024] In response to the received action recognition task, splitting the features extracted from the same window into a plurality of sub-window features;

[0025] Encoding, by the transformer model embedded with the NWA module, the sub-window features.

[0026] In an example embodiment of the present disclosure, the transformer model embedded with the NWA module comprises a multi-head-in-head transformer, and encoding, by the transformer model embedded with the NWA module, the sub-window features comprises:

[0027] Encoding, by the multi-head-in-head transformer module, the sub-window features.

[0028] According to a second aspect of the embodiments of the present disclosure, there is provided an action recognition apparatus for a video, comprising:

[0029] An encoding module configured to, in response to a received action recognition task, encode, by a transformer model embedded with an NWA module, adjacent video window features collected by two specified models;

[0030] A determining module configured to determine, according to an encoding result, a weight of the video window features for the action recognition task.

[0031] According to a third aspect of the present disclosure, there is provided an electronic device, comprising a memory and a processor coupled to the memory, the processor being configured to execute a method as described in any of the above aspects based on instructions stored in the memory.

[0032] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a program, which, when executed by a processor, implements an action recognition method for a video as described in any of the above aspects.

[0033] The embodiment of the present disclosure effectively improves the encoding ability of the transformer model for action features, ensures that sufficient features are allocated to short action segment candidates, in addition, quantifies the importance of the re-encoded action features to the action recognition task, enhances the relevance and effectiveness between adjacent feature windows, and improves the reliability and accuracy of action recognition of the video.

[0034] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0036] Figure 1 A schematic diagram of an exemplary system architecture of a video action recognition scheme to which embodiments of the present application can be applied is shown;

[0037] Figure 2 is a flowchart of a video action recognition method in an exemplary embodiment of the present disclosure;

[0038] Figure 3 is a flowchart of another video action recognition method in an exemplary embodiment of the present disclosure;

[0039] Figure 4 is a flowchart of another video action recognition method in an exemplary embodiment of the present disclosure;

[0040] Figure 5 is a flowchart of another video action recognition method in an exemplary embodiment of the present disclosure;

[0041] Figure 6 is a flowchart of another video action recognition method in an exemplary embodiment of the present disclosure;

[0042] Figure 7 is a flowchart of another video action recognition method in an exemplary embodiment of the present disclosure;

[0043] Figure 8is a schematic diagram of an NWA network structure of a video action recognition scheme in an example embodiment of the present disclosure;

[0044] Figure 9 is a flow processing schematic diagram of an NWA network structure of a video action recognition scheme in an example embodiment of the present disclosure;

[0045] Figure 10 is a schematic diagram of an embedded NWA Block model structure of a video action recognition scheme in an example embodiment of the present disclosure;

[0046] Figure 11 is a block diagram of a video action recognition apparatus in an example embodiment of the present disclosure;

[0047] Figure 12 is a block diagram of an electronic device in an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations. In the following description, numerous specific details are provided to give a thorough understanding of implementations of the disclosure. One skilled in the relevant art will recognize, however, that the implementations of the disclosure can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures have not been described in detail to avoid obscuring aspects of the disclosure.

[0049] Furthermore, the accompanying drawings are only schematic and are non-limiting detailed descriptions of implementations, as will be described below. Like references numerals can be used to denote like parts throughout the description. Each example or embodiment is intended to cover any and all modifications and variations under the scope of the disclosure. Therefore, features disclosed in one example or embodiment need not necessarily be combined with features of a different example or embodiment to benefit from the disclosure, even if that combination can be advantageous in some cases. It is intended to encompass all alternatives, modifications and variations falling within the scope of the present disclosure. The present disclosure also covers all the single features individually.

[0050] Figure 1 is a schematic diagram of an example system architecture of a video action recognition scheme to which embodiments of the present disclosure can be applied.

[0051] As Figure 1As shown, the system architecture 100 can include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0052] It should be understood that the number of terminal devices, networks, and servers in the system architecture 100 is only illustrative. Any number of terminal devices, networks, and servers can be provided according to implementation needs. For example, the server 105 can be a server cluster composed of multiple servers, etc. Figure 1

[0053] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, and desktop computers, etc.

[0054] In some embodiments, the action recognition method of the video provided by the embodiments of the present disclosure is generally executed by the server 105, and accordingly, the action recognition apparatus of the video is generally provided in the terminal device 103 (or the terminal device 101 or 102). In other embodiments, some terminals can have similar functions as the server device to perform the present disclosure.

[0055] The example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0056] Figure 2 is a flowchart of the action recognition method of the video in the example embodiments of the present disclosure.

[0057] Referring to Figure 2 , the action recognition method of the video can include:

[0058] Step S202, in response to the received action recognition task, encoding the adjacent video window features collected by the two specified models through a transformer model embedded with a neighborhood-window-attention (NWA) module.

[0059] Step S204, determining the weight of the video window features on the action recognition task according to the encoding result.

[0060] ​The embodiment of the present disclosure effectively improves the encoding capability of the transformer model for action features, ensures that sufficient features are allocated to short action segment candidates, in addition, quantifies the importance of the re-encoded action features to the action recognition task, enhances the relevance and effectiveness between adjacent feature windows, and improves the reliability and accuracy of action recognition of the video.

[0061] Next, each step of the action recognition method for the video is described in detail.

[0062] In an exemplary embodiment of the present disclosure, as shown in Figure 3 Before responding to the received action recognition task, it further includes:

[0063] Step S302: Extracting specified frames from the original video according to a preset interval, and performing image enhancement on the specified frames.

[0064] Step S304: Extracting video window features of the specified frames with image enhancement and position encoding by the slowfast model, and recording the first features.

[0065] In the above embodiment, the SlowFast algorithm is composed of two convolution branches:

[0066] (1) Slow branch: learning spatial semantic information with fewer frames and more channel numbers.

[0067] (2) Fast branch: learning motion information with more frames and fewer channel numbers.

[0068] The amount of calculation is proportional to the square of the number of channels. The Fast branch is relatively lightweight due to the fewer number of channels, and only accounts for 20% of the overall calculation. The slow branch uses a larger step size τ to collect video frames, usually set τ to 16, if the frame rate of the video is 30, it means that about 2 frames can be collected in 1 second, that is, T = 2. The channel number of the slow branch is D, and the Fast branch uses a smaller step size to collect video frames, the step size is τ / α, where α is usually set to 8, so for a video with a frame rate of 30, 15 frames (αT) can be collected in 1 second. The Fast branch remains lightweight by using a smaller number of channels (βD), and β is usually set to 1 / 8.

[0069] Step S306, extracting a video window feature of the specified frame after image enhancement and with position encoding by a timesformer model, and recording as a second feature, wherein the specified model includes the slowfast model and / or the timesformer model, and the video window feature includes the first feature and / or the second feature.

[0070] In the above embodiment, the Timesformer runs based on the self-attention mechanism in the Transformer, so that it can capture the spatiotemporal dependencies in the entire video. The TimeSformer reduces the amount of computation in two ways: it decomposes the video into disjoint subsets of image block sequences; and it uses a unique way of self-attention to avoid complex calculations between all image block sequences. In the temporal attention, each image block only attends to the image blocks extracted at the corresponding position in the remaining frames. In the spatial attention, the image block only attends to the extracted image blocks of the same frame.

[0071] In an exemplary embodiment of the present disclosure, as shown in Figure 4 The action recognition method of the video further includes:

[0072] Step S402, determining a network predicted action label and an actual action label.

[0073] Step S404, determining an action start time label and an action end time label.

[0074] Step S406, constructing a loss function of the transformer model according to a preset loss weight, the network predicted action label, the actual action label, the action start time label, and the action end time label.

[0075] In an exemplary embodiment of the present disclosure, as shown in Figure 5 The action recognition method of the video further includes:

[0076] Step S502, assuming that the optimal solution of the loss function is smooth near the optimal solution and the optimal solution can be parameterized as t.

[0077] Step S504, deriving the t in a specified range to determine the optimal solution according to the derivation result.

[0078] In the above embodiment, the optimal solution is a Pareto optimal solution, that is, given a feasible point, if then the feasible point is called an absolute optimal solution of the multi-objective programming problem. If not, it is called an effective solution of the multi-objective programming problem, and the effective solution of the multi-objective programming problem.

[0079] In an example embodiment of the present disclosure, the expression of the loss function comprises:

[0080] L(X) = a i (L cls (Y C ,Y GC )+L Tbox (Y Tbox ,Y GTbox ))+b i L p ),

[0081] wherein the X represents an input feature, the Y C represents a network predicted action label, the Y GC represents an actual action label, the Y Tbox represents a network predicted start time label and end time label of a corresponding identified action, the Y GTbox represents an actual start time label and end time label of an action, the a i and the b i represent weight parameters of the loss function, L CLS is a network classification loss function, L Tbox is a network regression loss function, and L p is a gradient loss output by a network NWA module.

[0082] In an example embodiment of the present disclosure, as shown in Figure 6 , in response to a received action recognition task, encoding adjacent video window features collected by two specified models through a transformer model embedded with an NWA module comprises:

[0083] Step S602, in response to a received action recognition task, splitting features extracted from the same window into a plurality of sub-window features.

[0084] Step S604, encoding the sub-window features through a transformer model embedded with an NWA module.

[0085] In an example embodiment of the present disclosure, as shown in Figure 7 , the transformer model embedded with an NWA module comprises a multi-head-in-head transformer, and encoding the sub-window features through the transformer model embedded with an NWA module comprises:

[0086] Step S702, encoding the sub-window features through the multi-head-in-head transformer module.

[0087] The action recognition scheme of a video in an exemplary embodiment of the present disclosure is described in detail below. Figure 8 to Figure 10 The action recognition scheme of a video in an exemplary embodiment of the present disclosure is described in detail below.

[0088] As Figure 8 As shown in the embodiments of the present disclosure, the correlation and effectiveness between adjacent and same video window or block area features are re-encoded, and at the same time, the network is automatically focused on and learns the importance of adjacent and same window or block area to the model.

[0089] In one exemplary embodiment of the present disclosure, the key frame 800 of the video moment is detected as the action target instance of the center of its timeline, and further estimates the time distance from the center action moment to the action start and offset, which is called time boundary. The key frame 800 is cropped by the Clip Feature 802, and the cropped window with position encoding is input to the Multi-Scale Channel Transformer Encode 804 for encoding processing, and then sent to the Multi-Scale Channel Transformer Decode 806 for decoding processing to obtain the feature pyramid 808.

[0090] As Figure 9 As shown in the embodiments of the present disclosure, the correlation and effectiveness between adjacent and same video window or block area features are re-encoded, and at the same time, the network is automatically focused on and learns the importance of adjacent and same window or block area to the model.

[0091] The core technical features of the present disclosure include the following aspects: 1. Video data frame extraction strategy design; 2. Extraction of different video features; 3. NWA BLOCK model construction; 4. NWA model loss calculation.

[0092] (1) Video data frame extraction strategy design: design the frame extraction strategy for the original video, in order to finely classify the positioning of the action, all videos are extracted 32 frames every 16 frames.

[0093] (2) Different video feature extraction: a series of video window features with position encoding are extracted using different models SlowFast and Timesformer, and different image enhancement operations are adopted on the original scene video data in the process of extracting features.

[0094] (3) NWA BLOCK model construction: as shown in Figure 10 , the NWA module is embedded into any visual transformer model structure, and the outer window models the global relationship of the features. The features extracted from the same window are uniformly split into some sub-windows, such as NWA BLOCK1 1002 and NWA BLOCK2 1004. The NWA uses the modified multi-head-in-head transformer to re-encode the sub-adjacent window features.

[0095] (4) NWA model loss calculation: for different NWA features, the Pareto idea is used for guided learning of window features. After modeling all NWA windows, while each window target optimization process approaches the infeasible region, the continuous Pareto frontier is used to constantly let Pareto find the optimal solution to minimize the error of the target. In this patent, different loss functions are designed for action localization detection and recognition NWA Block, as shown in the following formula:

[0096] L(X)=α i (L cls (Y C ,Y GC )+L Tbox (Y Tbox ,Y GTbox ))+β i L p ),

[0097] Where X is the input feature, Y C is the network predicted action label, Y GC is the actual action label Y Tbox is the network predicted label of the start and end time of the recognized action, Y GTbox is the start and end time label of the time action. α i and β i are the weight parameters of different losses, L CLS is the network classification loss function, L Tbox is the network regression loss function, and L p is the gradient loss of the network NWA module output.

[0098] For the loss function L(X) above, Pareto is needed to find the optimal solution, assuming that the optimal solution is smooth around and can be parameterized as variable t, then a small enough range [-e, e] can be found, and the derivative of t is found and the optimal solution t * , so that L(X) is minimized.

[0099] Corresponding to the method embodiments described above, the disclosure also provides an action recognition device for video, which can be used to execute the method embodiments described above.

[0100] Figure 11 is a block diagram of an action recognition device for video in an exemplary embodiment of the disclosure.

[0101] Reference Figure 11 The action recognition device for video 1100 can include:

[0102] The encoding module 1102 is configured to encode the adjacent video window features collected by the two specified models through the transformer model embedded with the NWA module in response to the received action recognition task.

[0103] The determination module 1104 is configured to determine the weight of the video window features on the action recognition task according to the encoding result.

[0104] In an exemplary embodiment of the disclosure, before responding to the received action recognition task, the action recognition device for video 1100 is further configured to:

[0105] extracting specified frames from the original video at a preset interval, and performing image enhancement on the specified frames;

[0106] extracting video window features of the specified frames with image enhancement and position encoding through the slowfast model, and recording the first features;

[0107] extracting video window features of the specified frames with image enhancement and position encoding through the timesformer model, and recording the second features,

[0108] wherein the specified model includes the slowfast model and / or the timesformer model, and the video window features include the first features and / or the second features.

[0109] In an exemplary embodiment of the disclosure, the action recognition device for video 1100 is further configured to:

[0110] determining the network predicted action label and the actual action label;

[0111] determining the action start time label and the action end time label;

[0112] construct a loss function of the transformer model according to the preset loss weight, the network predicted action label, the actual action label, the action start time label and the action end time label.

[0113] In an exemplary embodiment of the present disclosure, the action recognition device 1100 for a video is further configured to:

[0114] assuming that the optimal solution of the loss function is smooth near the optimal solution and the optimal solution can be parameterized as t;

[0115] deriving the t in a specified range to determine the optimal solution according to the derivation result.

[0116] In an exemplary embodiment of the present disclosure, the expression of the loss function comprises:

[0117] L(X) = a i (L cls (Y C ,Y GC )+L Tbox (Y Tbox ,Y GTbox ))+β i L p ),

[0118] wherein the X represents an input feature, the Y C represents a network predicted action label, the Y GC represents an actual action label, the Y Tbox represents a network predicted start time label and end time label of a corresponding recognized action, the Y GTbox represents an actual start time label and end time label of an action, the a i and the β i represent weight parameters of the loss function, L CLS is a network classification loss function, L Tbox is a network regression loss function, and L p is a gradient loss output by a network NWA module.

[0119] In an exemplary embodiment of the present disclosure, the encoding module 1102 is further configured to:

[0120] in response to the received action recognition task, split the features extracted from the same window into a plurality of sub-window features;

[0121] encode the sub-window features by the transformer model embedded with the NWA module.

[0122] In an example embodiment of the present disclosure, the encoding module 1102 is further configured to:

[0123] encode the sub-window features by the multi-head-in-head transformer module.

[0124] Since the functions of the action recognition apparatus 1100 for a video have been described in detail in the corresponding method embodiments, the present disclosure will not be described here again.

[0125] It should be noted that although several modules or units of the device for action performance are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into embodied by multiple modules or units.

[0126] In an example embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0127] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0128] The electronic device 1200 according to this embodiment of the present disclosure will be described below with reference to Figure 12 Figure 12 The displayed electronic device 1200 is only an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0129] As shown in Figure 12 The components of the electronic device 1200 can include, but are not limited to, the above-mentioned at least one processing unit 1210, the above-mentioned at least one storage unit 1220, and a bus 1230 connecting different system components, including the storage unit 1220 and the processing unit 1210.

[0130] The storage unit stores program code which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps described in the above "example method" section of the present specification according to various example embodiments of the present disclosure. For example, the processing unit 1210 can perform the method as shown in the embodiments of the present disclosure. ​

[0131] Storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 12201 and / or cache memory 12202, and may further include a read-only memory (ROM) 12203.

[0132] Storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0133] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0134] Electronic device 1200 can also communicate with one or more external devices 1240 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1200, and / or any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0135] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0136] In exemplary embodiments of the present disclosure, a computer readable storage medium having stored thereon a program product capable of implementing the above-described methods of the specification is also provided. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product including a program code for causing an end device to perform the steps described in the above "Exemplary Methods" section according to various exemplary embodiments of the present application when the program product is run on the end device.

[0137] The program product for implementing the above-described methods according to the embodiments of the present application can take the form of a portable compact disc read-only memory (CD-ROM) and include a program code, and can be run on an end device such as a personal computer. However, the program product of the present application is not limited thereto, and in the present document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus or device.

[0138] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0139] The computer readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which readable program code is borne. Such propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a readable storage medium and that can transmit, propagate or transport for use by or in connection with an instruction execution system, apparatus or device program.

[0140] The program code contained on the readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0141] The program code may, for example, be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider.

[0142] Furthermore, the above-described diagrams are merely schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is readily understood that the processes shown in the above-described diagrams do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0143] Other embodiments of the disclosure will be apparent to those of ordinary skill in the art from a consideration of the specification and practice of the disclosure disclosed herein. The specification and examples given herein are intended as illustrative only and are not intended to limit the true scope and spirit of the disclosure. The true scope and spirit of the disclosure are indicated by the appended claims.

Claims

1. A method of action recognition of a video, characterized by, The method comprises: extracting specified frames from the original video at preset intervals, and performing image enhancement on the specified frames; extracting video window features of the specified frames with image enhancement and position encoding through a slowfast model, and recording the features as first features; extracting video window features of the specified frames with image enhancement and position encoding through a timesformer model, and recording the features as second features; in response to a received action recognition task, encoding adjacent video window features collected by two specified models through a transformer model embedded with an NWA module, wherein the specified models include the slowfast model and / or the timesformer model, and the video window features include the first features and / or the second features; determining the weight of the video window features on the action recognition task according to the encoding result. 2.The method of claim 1, wherein, The method further comprises: determining a network predicted action label and an actual action label; determining an action start time label and an action end time label; constructing a loss function of the transformer model according to a preset loss weight, the network predicted action label, the actual action label, the action start time label, and the action end time label. 3.The method of claim 2, wherein, The method further comprises: assuming that the optimal solution of the loss function is smooth near the optimal solution and the optimal solution can be parameterized as t; deriving the t in a specified range to determine the optimal solution according to the derivation result.

4. The motion recognition method of video according to claim 2 or 3, characterized in that, The expression of the loss function comprises: ， wherein the characterizing an input feature, the characterizing a network predicted action label, the characterizing an actual action label, the characterizing a network predicted start time label and end time label of a corresponding identified action, the characterizing an actual start time label and end time label of an action, the and the characterizing a weight parameter of the loss function, is a network classification loss function, is a network regression loss function, is a gradient loss of a network NWA module output.

5. The method of claim 1-3, wherein, in response to a received action recognition task, encoding adjacent video window features collected by two specified models through a transformer model embedded with an NWA module, in response to a received action recognition task, splitting the features extracted from the same window into a plurality of sub-window features; encoding the sub-window features through the transformer model embedded with the NWA module. 6.The method of claim 5, wherein, The transformer model embedded with the NWA module comprises a multi-head-in-head transformer, and the encoding of the sub-window features through the transformer model embedded with the NWA module comprises: encoding the sub-window features through the multi-head-in-head transformer module.

7. A video motion recognition device, characterized in that, The method comprises: a coding module configured to extract specified frames from the original video at preset intervals, and perform image enhancement on the specified frames; extracting video window features of the specified frames with image enhancement and position encoding through a slowfast model, and recording the features as first features; extracting video window features of the specified frames with image enhancement and position encoding through a timesformer model, and recording the features as second features; in response to a received action recognition task, encoding adjacent video window features collected by two specified models through a transformer model embedded with an NWA module, The specified model comprises the slowfast model and / or the timesformer model, and the video window feature comprises the first feature and / or the second feature. The determining module is configured to determine a weight of the video window feature for the action recognition task according to the encoding result.

8. An electronic device, comprising: Comprise: a memory; and a processor coupled to the memory, the processor being configured to execute a method for action recognition of a video according to any one of claims 1-6 based on instructions stored in the memory. 9.A computer readable storage medium having stored thereon a program which, when executed by a processor, implements a method for action recognition of a video according to any one of claims 1-6.

Images