Video action detection method based on end-to-end framework and electronic device

Abstract

The application provides a video action detection method based on an end-to-end framework and an electronic device, the end-to-end framework comprising a backbone network, a positioning module and a classification module, the method comprising: performing feature extraction on a to-be-detected video segment by the backbone network to obtain a video feature map of the to-be-detected video segment, wherein the video feature map comprises feature maps of all frames in the to-be-detected video segment; extracting a feature map of a key frame from the video feature map by the backbone network, obtaining an actor position feature from the feature map of the key frame, and obtaining an action category feature from the video feature map; determining an actor position according to the actor position feature by the positioning module; and determining an action category corresponding to the actor position according to the action category feature and the actor position by the classification module. The video action detection method provided by the application has low complexity and can achieve good detection performance.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and more specifically, to a video motion detection method and electronic device based on an end-to-end framework. Background Technology

[0002] Video action detection, comprising actor bounding box localization and action classification, is primarily applied in fields such as abnormal behavior detection and autonomous driving. Existing techniques typically utilize two independent stages for video action detection: the first stage employs a pre-trained object detection model on the COCO dataset, trained on a task dataset to obtain a single-class detector for actors (such as humans); the second stage uses the detector trained in the first stage to perform actor bounding box localization (i.e., predicting actor position), and then extracts feature maps of the actor position for action classification (i.e., predicting action category). These two stages each use two independent backbone networks: the first stage uses 2D image data for actor bounding box localization, and the second stage uses 3D video data for action classification.

[0003] Using two separate backbone networks to perform actor bounding box localization and action classification tasks leads to redundant computation and high complexity, thus limiting the application of existing technologies in real-world scenarios. To reduce complexity, a unified backbone network could be used instead of two separate backbone networks. However, using a single backbone network may cause interference between the two tasks. This interference manifests in two ways: First, actor bounding box localization typically uses 2D image models to predict actor positions in keyframes of video clips. Considering adjacent frames within the same video clip at this stage incurs additional computational and storage costs, as well as localization noise. Second, action classification relies on 3D video models to extract temporal information embedded in video clips. Using a single keyframe from actor bounding box localization may result in poor temporal motion representation for action classification. Summary of the Invention

[0004] The purpose of this invention is to provide a video motion detection technology based on an end-to-end framework to solve the problems existing in the prior art.

[0005] One aspect of the present invention provides a video action detection method based on an end-to-end framework, the end-to-end framework including a backbone network, a localization module, and a classification module. The video action detection method includes: the backbone network extracting features from a video segment to be tested, obtaining a video feature map of the video segment, wherein the video feature map includes feature maps of all frames in the video segment; the backbone network extracting feature maps of keyframes from the video feature map, obtaining actor position features from the feature maps of the keyframes, and obtaining action category features from the video feature map; the localization module determining the actor position based on the actor position features; and the classification module determining the action category corresponding to the actor position based on the action category features and the actor position.

[0006] The above method may include: extracting features from the video segment under test using a backbone network at multiple stages to obtain video feature maps for each stage, wherein the spatial scale of the video feature maps at different stages is different; selecting the video feature maps of the last few stages from the multiple stages using the backbone network, extracting feature maps of keyframes from the feature maps of the last few stages, extracting features from the feature maps of the keyframes to obtain actor position features, and using the video feature map of the last stage from the multiple stages as the action category feature. Specifically, a residual network can be used to extract features from the video segment under test at multiple stages, and a feature pyramid network can be used to extract features from the feature maps of the keyframes.

[0007] In the above method, the keyframe can be a frame located in the middle of the video segment to be tested.

[0008] In the above method, the process of the classification module determining the action category corresponding to the actor's position based on the action category features and the actor's position includes: the classification module extracting spatial action features and temporal action features corresponding to the actor's position from the action category features based on the actor's position, fusing the spatial action features and temporal action features corresponding to the actor's position, and determining the action category corresponding to the actor's position based on the fused features.

[0009] In the above method, the process of the classification module extracting spatial and temporal action features corresponding to the actor's position from the action category features based on the actor's position includes: the classification module extracting a fixed-scale feature map of the corresponding region from the action category features based on the actor's position; performing a global average pooling operation on the fixed-scale feature map in the temporal dimension to obtain the spatial action features corresponding to the actor's position; and performing a global average pooling operation on the fixed-scale feature map in the spatial dimension to obtain the temporal action features corresponding to the actor's position.

[0010] In the above method, the localization module determines multiple actor positions, and the classification module extracts spatial action features and temporal action features corresponding to each actor position from the action category features based on each actor position among the multiple actor positions. The method may further include: inputting the spatial embedding vectors corresponding to the multiple actor positions into a self-attention module; performing a convolution operation between the spatial action features corresponding to the multiple actor positions and the output of the self-attention module to update the spatial action features corresponding to each actor position among the multiple actor positions; and inputting the temporal embedding vectors corresponding to the multiple actor positions into the self-attention module; performing a convolution operation between the temporal action features corresponding to the multiple actor positions and the output of the self-attention module to update the temporal action features corresponding to each actor position among the multiple actor positions.

[0011] In the above method, determining the actor's location includes determining the coordinates of the actor's bounding box and the confidence level indicating that the actor's bounding box contains the actor. The method may further include selecting actor locations with a confidence level higher than a predetermined threshold and their corresponding action categories.

[0012] In the above method, the end-to-end framework is trained based on the following objective function:

[0013]

[0014] in, This indicates the loss from actor bounding box localization. This represents the action classification loss. For cross-entropy loss, and These are the bounding box losses, Let be the binary cross-entropy loss, and λ cls , λ L1 , λ giou and λ act This is a constant scalar used to balance the contribution of losses.

[0015] Another aspect of the present invention provides an electronic device including a processor and a memory, the memory storing a computer program executable by the processor, the computer program implementing the above-described video motion detection method based on an end-to-end framework when executed by the processor.

[0016] The technical solutions of the embodiments of the present invention can provide the following beneficial effects:

[0017] Using an end-to-end framework, actor positions and corresponding action categories can be directly generated and output from input video clips.

[0018] In the end-to-end framework, a unified backbone network is used to simultaneously extract actor location features and action category features, simplifying the feature extraction process. Specifically, in the early stages of the backbone network, keyframe feature maps (used for actor bounding box localization) are separated from video feature maps (used for action classification), reducing interference between actor bounding box localization and action classification. The localization and classification modules in the end-to-end framework share the backbone network, eliminating the need for additional ImageNet or COCO pre-training.

[0019] The localization module is trained using a bipartite graph matching method, eliminating the need for post-processing operations such as non-maximum suppression.

[0020] When performing action classification, the classification module further extracts spatial and temporal action features from the action category features, enriching the instance features. Furthermore, it performs embedding interactions on the spatial and temporal action features separately, utilizing lightweight embedding vectors to obtain more discriminative features while improving efficiency and action classification performance.

[0021] Experiments show that, compared with existing video action detection technologies, the video action detection method based on an end-to-end framework provided by this invention has lower detection complexity, is simpler, and can achieve better detection performance.

[0022] It should be understood that the above general description and the following detailed description are for illustrative and explanatory purposes only and are not intended to limit the invention. Attached Figure Description

[0023] Exemplary embodiments will be described in detail with reference to the accompanying drawings, which are intended to depict exemplary embodiments and should not be construed as limiting the intended scope of the claims. Unless explicitly stated otherwise, the drawings are not considered to be drawn to scale.

[0024] Figure 1 The schematic diagram illustrates the structure of an end-to-end frame according to an embodiment of the present invention;

[0025] Figure 2 A flowchart illustrating a video motion detection method according to an embodiment of the present invention is shown schematically;

[0026] Figure 3 This illustration schematically shows a structural diagram of a unified backbone network according to an embodiment of the present invention;

[0027] Figure 4 The diagram illustrates, schematically, the various operations performed in the classification module according to an embodiment of the present invention;

[0028] Figure 5 The schematic diagram illustrates the structure of an interactive module according to an embodiment of the present invention;

[0029] Figure 6 The flowchart illustrating a video motion detection method based on an end-to-end frame according to an embodiment of the present invention is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention more apparent, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] One aspect of the present invention provides a video action detection method that incorporates an end-to-end framework, such as... Figure 1 As shown, this end-to-end framework takes a video clip as input and outputs the actor's position and corresponding action category. The end-to-end framework includes a unified backbone feature extraction network (referred to as the backbone network), used to extract actor position features and action category features from the input video clip. It also includes a localization module and a classification module. The localization module determines the actor's position based on the actor position features, and the classification module determines the action category corresponding to the actor's position based on the action category features and the determined actor position. Using an end-to-end framework, actor positions and corresponding action categories can be directly generated and output from the input video clip, making the video action detection process much simpler.

[0034] Figure 2 The flowchart illustrating a video action detection method according to an embodiment of the present invention is shown schematically. In summary, the method includes constructing and training an end-to-end framework, and using the trained end-to-end framework to determine the actor's position and corresponding action category from a video segment to be tested. The following will combine... Figure 2 Describe each step of the video motion detection method.

[0035] Step S11. Build an end-to-end framework.

[0036] Overall, the end-to-end framework includes a unified backbone network, a localization module, and a classification module.

[0037] The unified backbone network consists of a Residual Network (ResNet) with multiple stages (e.g., 5 stages) and a Feature Pyramid Network (FPN) with multiple layers (e.g., 4 layers). The backbone network receives video segments (e.g., pre-processed video segments) as input to the end-to-end frame and outputs actor position features and action category features. Within the backbone network, ResNet performs feature extraction on the input video segments in multiple stages, resulting in video feature maps for each stage (or the video feature maps extracted at each stage). The spatial scale of the video feature maps differs between stages. A video feature map consists of feature maps from all frames in the video segment and can be represented as... Where C represents the number of channels, T represents time (and also the number of frames in the input video clip), and H and W represent spatial height and width, respectively. Within the backbone network, after obtaining the video feature maps for each stage of ResNet, the feature maps of keyframes extracted from the video feature maps of the last few stages of ResNet (e.g., the last four stages) are extracted as input to the FPN. The FPN then extracts features from the keyframe feature maps to obtain the actor's position features. Additionally, the video feature map extracted from the last stage of ResNet is used as the actor's action category feature. A keyframe refers to a frame located in the middle of the input video clip, such as a frame located in the middle of the video clip. The feature map of the keyframe at that location can be represented as:

[0038] Figure 3 A schematic diagram of the backbone network consisting of ResNet and FPN is shown, where ResNet comprises five stages Res1-Res5 (the first two stages are not shown in the diagram) and FPN comprises three layers. Figure 3 As shown, for the feature maps of keyframes extracted from the video feature maps in the Res3-Res5 stages, the FPN performs further feature extraction to obtain actor position features; in addition, the video feature maps extracted in the Res5 stage are also used as action category features. In this embodiment, the backbone network is described as consisting of a ResNet containing multiple stages and a feature pyramid network containing multiple layers, but it should be understood that the backbone network can also use a network containing only one stage or one layer to perform feature extraction.

[0039] The localization module performs actor bounding box localization. Its input is actor location features (output from the backbone network), and its output is the actor location. The output actor location can include the coordinates of the actor bounding box (referred to as the bounding box) and its corresponding score. The bounding box refers to the bounding box containing the actor; its coordinates indicate the actor's position in the video clip (more specifically, its position in the keyframe), and the score indicates the confidence that the corresponding bounding box contains the actor. A higher confidence score indicates a greater probability that the corresponding bounding box contains the actor. It is important to note that the number of actor locations (i.e., the number of bounding boxes) output by the localization module each time is fixed, and can be one or more. This number should be greater than or equal to the number of actors in all keyframes. For ease of description, N will be used below to represent the number of actor locations, and N will be set to an integer greater than 1.

[0040] The classification module performs action classification. Its inputs are action category features (output from the backbone network), i.e., the video feature map extracted in the last stage of ResNet, and N actor positions (output from the localization module). The output is the action category corresponding to each actor position. Specifically, based on the N actor positions (corresponding to N bounding boxes), the classification module extracts spatial and temporal action features from the action category features for each actor position, resulting in spatial and temporal action features for each actor position. Embedding interactions are performed on the spatial and temporal action features of the N actor positions to obtain the final spatial and temporal action features for each of the N actor positions. The final spatial and temporal action features for each actor position are then fused to obtain the final action category feature corresponding to each actor position. The action category corresponding to each actor position is determined based on the final action category feature. (See below for further details.) Figure 4 Each operation performed in the classification module is described separately:

[0041] 1. Based on the N actor positions (i.e. N bounding boxes) determined by the localization module, extract spatial action features and temporal action features from the action category features for each position.

[0042] As mentioned above, the input to the classification module is the video feature map from the last stage of the ResNet backbone network. (Action category features), where I represents the total number of stages in the ResNet. Based on the N actor positions determined by the localization module, more specifically, based on the coordinates of the N bounding boxes, RoIAlign is used in... Extract fixed-scale feature maps of the corresponding regions, where S×S is the output spatial scale of RoIAlign, thus obtaining the RoI features corresponding to each of the N actor positions. Perform global average pooling on the RoI features corresponding to each actor position along the time dimension to obtain the spatial action features of each actor position. in Let N represent the spatial action features of the nth actor's position, where 1 ≤ n ≤ N; perform global average pooling on the RoI features corresponding to each actor's position in the spatial dimension to obtain the temporal action features of each actor's position. in The temporal action characteristics represent the position of the nth actor.

[0043] In addition to the methods mentioned above, spatial and temporal action features of each actor's location can also be extracted using the following methods: action category features. Perform global average pooling on the time dimension to obtain the spatial feature map. Based on the locations of N actors determined by the positioning module, RoIAlign is used in f s Extract the fixed-size feature map of the corresponding region to obtain the spatial motion features of each of the N actor positions. Action category features Global average pooling is performed in the spatial dimension to efficiently extract the temporal action features of each of the N actor positions.

[0044] 2. Perform embedding interactions on the spatial and temporal action features of the N actor positions respectively to obtain the final spatial and temporal action features of each of the N actor positions.

[0045] Each actor's spatial motion feature has a corresponding spatial embedding vector, and each actor's temporal motion feature also has a corresponding temporal embedding vector. The spatial embedding vector is used to encode spatial attributes, such as shape and pose, while the temporal embedding vector is used to encode temporal dynamic attributes, such as motion dynamics and time scale.

[0046] Input the spatial embedding vectors and spatial action features corresponding to the positions of N actors. Figure 5 The interactive module shown, in which Figure 5 The self-attention module (e.g., the interaction module) is shown. Figure 5 (as shown in the left half) and convolution operations (such as...) Figure 5(As shown in the right half). The spatial embedding vectors corresponding to the N actor positions are processed by a self-attention module to obtain the corresponding outputs. These outputs are then convolved with the spatial action features of the N actor positions using a 1x1 kernel to obtain the final spatial action features for each of the N actor positions. Similarly, the temporal embedding vectors and temporal action features corresponding to the N actor positions are input... Figure 5 The interactive module shown obtains the final temporal action features for each of the N actor positions.

[0047] To capture the relationship information between different actors, a self-attention mechanism is introduced to obtain richer information. Furthermore, the self-attention mechanism is applied using spatial and temporal embedding vectors corresponding to the spatial and temporal action features of each actor's location. Subsequently, the output of the self-attention module is convolved with the spatial and temporal action features to obtain more discriminative features. Compared with directly applying the self-attention mechanism to the spatial and temporal action features, the lighter spatial and temporal embedding vectors can improve efficiency.

[0048] 3. Fuse the final spatial action features and final temporal action features for each of the N actor positions to obtain the final action category features corresponding to each actor position. The fusion operation includes, but is not limited to, summation, concatenation, and cross-attention operations.

[0049] 4. Determine the action category corresponding to each actor position based on the final action category features corresponding to that position. The action category corresponding to each actor position can be identified from the final action category features corresponding to each actor position using a fully connected (FC) layer, which indicates the probability value of each of all action categories.

[0050] The above describes an end-to-end framework (with video clips as input and actor positions and corresponding action categories as output). In this framework, a unified backbone network is used to extract actor position features and action category features simultaneously, which simplifies the feature extraction process. In addition, the feature maps of keyframes are separated from the video feature maps in the early stages of the backbone network, reducing the mutual interference between actor bounding box localization and action classification.

[0051] To train this end-to-end framework, the objective function is also constructed as follows:

[0052]

[0053] The objective function consists of two parts: one part is the actor localization loss, where... The cross-entropy loss represents the loss across two categories (those containing actors and those not containing actors). and Let λ represent the bounding box loss, respectively. cls , λ L1 and λ giou One part represents the constant scalar used to balance the loss contribution; the other part is the action classification loss, where Let λ represent the binary cross-entropy loss used for action classification. act This represents a constant scalar used to balance the contribution of losses.

[0054] Step S12. Train the end-to-end framework.

[0055] During the training phase, a training dataset is acquired to train the framework end-to-end. The Hungarian algorithm is used to perform bipartite graph matching between the coordinates of the N bounding boxes output by the end-to-end framework (more specifically, the localization module within the end-to-end framework) and the actor's ground truth location to find an optimal match. For bounding boxes that match the ground truth location, the actor localization loss is calculated according to formula (1), and the action classification loss is further calculated. Backward gradient propagation is then performed based on these two (more specifically, based on their sum) to update the parameters. For bounding boxes that do not match the ground truth location, only the actor localization loss is calculated according to formula (1), without calculating the action classification loss, for use in backward gradient propagation and parameter updates. The localization module is trained using bipartite graph matching, eliminating the need for post-processing operations such as non-maximum suppression (NMS).

[0056] It should be understood that after the training set is completed, the accuracy of the final end-to-end framework can also be evaluated using the test dataset.

[0057] Step S13. Obtain the video segment to be tested and input it into the trained end-to-end framework. The video segment to be tested can be preprocessed first, and then the preprocessed video segment can be input into the trained end-to-end framework.

[0058] Step S14. The end-to-end framework determines the actor's position and corresponding action category from the video segment under test, and outputs the actor's position and corresponding action category. See also Figure 6 Step S14 includes the following sub-steps:

[0059] S141. The backbone network in the end-to-end framework extracts features from the video segment under test to obtain a video feature map of the video segment under test. The video feature map includes the feature maps of all frames in the video segment under test.

[0060] The backbone network consists of a ResNet with multiple stages and a multi-layer FPN. Within the backbone network, ResNet performs feature extraction on the video segment under test at multiple stages, thereby obtaining video feature maps for each stage. The spatial scale of the video feature maps at different stages is different.

[0061] S142. The backbone network in the end-to-end framework extracts the feature maps of keyframes from the video feature maps, obtains the actor position features from the feature maps of keyframes, and obtains the action category features from the video feature maps.

[0062] After obtaining the video feature maps for each stage of ResNet, the feature maps of keyframes extracted from the video feature maps of the later stages of ResNet are used as input to the FPN. The FPN then extracts features from the feature maps of the keyframes to obtain the actor's position features. Here, a keyframe refers to a frame located in the middle of the video segment under test.

[0063] In addition, the video feature map extracted in the last stage of ResNet is used as the action category feature of the actor.

[0064] S143. The localization module in the end-to-end framework determines N actor positions based on actor position features. The localization module takes actor position features as input and outputs N actor positions. The output actor positions may include the coordinates of the actor's bounding box and its corresponding score. The bounding box refers to the bounding box containing the actor; its coordinates indicate the actor's position in the video clip (more specifically, its position in the keyframe), and the score indicates the confidence that the corresponding bounding box contains the actor. A higher confidence score indicates a greater probability that the corresponding bounding box contains the actor.

[0065] S144. The classification module in the end-to-end framework determines the action category corresponding to each actor position based on the action category features and the determined N actor positions.

[0066] Based on N actor positions, the classification module first extracts spatial and temporal action features from the action category features for each actor position, obtaining the spatial and temporal action features for each actor position. Then, it performs embedding interactions on the spatial and temporal action features of multiple actor positions to obtain the final spatial and temporal action features for each of the multiple actor positions. The classification module also fuses the final spatial and temporal action features for each actor position to obtain the final action category feature corresponding to each actor position. Based on the final action category feature corresponding to each actor position, the module determines the action category corresponding to that actor position.

[0067] Step S15. Select from the actor position and corresponding action category output by the end-to-end framework to obtain the final actor position and corresponding action category.

[0068] As described above, the N actor locations output by the end-to-end framework include the coordinates of the N actor bounding boxes and their corresponding scores (i.e., confidence levels). The actor locations and their corresponding action categories with confidence levels greater than a predetermined threshold (e.g., a threshold of 0.7) are selected as the final results.

[0069] The above embodiments employ an end-to-end framework that can directly generate and output actor positions and corresponding action categories from input video clips. In this end-to-end framework, a unified backbone network is used to simultaneously extract actor position features and action category features, simplifying the feature extraction process. Specifically, in the early stages of the backbone network, keyframe feature maps (used for actor bounding box localization) are separated from video feature maps (used for action classification), reducing mutual interference between actor bounding box localization and action classification. The localization and classification modules in the end-to-end framework share the backbone network, eliminating the need for additional ImageNet or COCO pre-training.

[0070] In the above embodiments, the localization module is trained using a bipartite graph matching method, eliminating the need for post-processing operations such as non-maximum suppression during the evaluation phase. When performing action classification, the classification module further extracts spatial and temporal action features from the action category features, enriching the instance features. Furthermore, it performs embedding interactions on the spatial and temporal action features respectively, utilizing lightweight embedding vectors to obtain more discriminative features while further improving efficiency and action classification performance.

[0071] To verify the effectiveness of the embodiments of the present invention, the video action detection method provided by the present invention was compared with other existing video action detection technologies in terms of detection performance. Table 1 shows the comparison results. The data in Table 1 were obtained by training and testing on the AVA dataset. It can be seen that, compared with other existing technologies, the video action detection method provided by the present invention can significantly reduce the computational requirements, has lower complexity, is simpler, and has a better detection performance index mAP.

[0072] Table 1

[0073] AVA - × K400 15.6 SlowFast, R50 223.3 × K400 24.7 This invention, R50 141.6 √ K400 25.2 SlowFast, R101 302.3 × K600 27.4 This invention, R101 251.7 √ K600 28.3

[0074] Another aspect of the present invention provides a schematic diagram of a computer system suitable for implementing an electronic device according to embodiments of the present invention. The computer system may include: a bus, to which devices coupled can rapidly transmit information; and a processor, coupled to the bus and used to execute a set of actions or operations specified by a computer program. The processor may be implemented alone or in combination with other devices as a mechanical, electrical, magnetic, optical, quantum, or chemical component, etc.

[0075] The computer system may also include a memory coupled to a bus. The memory (e.g., RAM or other dynamic storage device) stores data that can be changed by the computer system, including instructions or computer programs for implementing the video motion detection method described in the above embodiments. When the processor executes the instructions or computer program, the computer system is enabled to implement the video motion detection method described in the above embodiments; for example, it can implement... Figure 2 , Figure 6 The steps are illustrated. The memory can also store temporary data generated during processor execution of instructions or computer programs, as well as various programs and data required for system operation. The computer system also includes read-only memory coupled to the bus and non-volatile storage devices, such as disks or optical discs, for storing data that persists even when the computer system is shut down or loses power.

[0076] Computer systems may also include input devices such as keyboards and sensors, and output devices such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), and printers. Computer systems may also include communication interfaces coupled to a bus, which can provide one-way or two-way communication coupling to external devices. For example, a communication interface may be a parallel port, a serial port, a telephone modem, or a local area network (LAN) card. Computer systems may also include drive devices coupled to the bus and removable devices such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., which are mounted on the drive devices as needed so that computer programs read from them can be installed into storage devices as required.

[0077] It should be understood that although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described herein, and includes various changes and variations without departing from the scope of the present invention.

Claims

1. A video action detection method based on an end-to-end framework, wherein the end-to-end framework includes a unified backbone network, a localization module, and a classification module, the method comprising: The unified backbone network performs feature extraction on the video segment under test to obtain a video feature map of the video segment under test, wherein the video feature map includes feature maps of all frames in the video segment under test. The unified backbone network performs feature extraction on the video segment under test in multiple stages to obtain video feature maps for each stage, wherein the spatial scale of the video feature maps in different stages is different. The unified backbone network selects video feature maps from the last few stages of multiple stages, extracts feature maps of keyframes from these last few stages, performs feature extraction on the feature maps of the keyframes to obtain actor position features, and uses the video feature map of the last stage of multiple stages as the action category feature; The positioning module determines the actor's location based on the actor's location characteristics; as well as The classification module determines the action category corresponding to the actor's position based on the action category features and the actor's position.

2. The method of claim 1, wherein, The residual network is used to perform feature extraction on the video segment under test in multiple stages, and the feature pyramid network is used to extract features from the feature map of the key frame.

3. The method of claim 1, wherein, The keyframe is a frame located in the middle of the video segment to be tested.

4. The method of any one of claims 1-3, wherein, The classification module determines the action category corresponding to the actor's position based on the action category features and the actor's position, including: The classification module extracts spatial and temporal action features corresponding to the actor's position from the action category features based on the actor's position, fuses the spatial and temporal action features corresponding to the actor's position, and determines the action category corresponding to the actor's position based on the fused features.

5. The method of claim 4, wherein, The classification module extracts spatial and temporal action features corresponding to the actor's location from the action category features based on the actor's location, including: The classification module extracts a fixed-scale feature map of the corresponding region from the action category features based on the actor's location; performs global average pooling on the fixed-scale feature map in the time dimension to obtain spatial action features corresponding to the actor's location; and performs global average pooling on the fixed-scale feature map in the spatial dimension to obtain temporal action features corresponding to the actor's location.

6. The method of claim 4, wherein, The positioning module determines the locations of multiple actors, and the classification module extracts spatial action features and temporal action features corresponding to each actor location from the action category features based on each actor location among the multiple actor locations; Furthermore, the method further includes: The spatial embedding vectors corresponding to the multiple actor positions are input into the self-attention module. The spatial action features corresponding to the multiple actor positions are convolved with the output of the self-attention module to update the spatial action features corresponding to each of the multiple actor positions. The temporal embedding vectors corresponding to the multiple actor positions are input into the self-attention module. The temporal action features corresponding to the multiple actor positions are convolved with the output of the self-attention module to update the temporal action features corresponding to each of the multiple actor positions.

7. The method of any one of claims 1-3, wherein, Determining the actor's location includes determining the coordinates of the actor's bounding box and indicating the confidence that the actor's bounding box contains the actor; Furthermore, the method further includes: Select the actor's location with a confidence level higher than a predetermined threshold and the action category corresponding to the actor's location.

8. The method of claim 7, wherein, The end-to-end framework is trained based on the following objective function: in, This indicates the loss from actor bounding box localization. This represents the action classification loss. For cross-entropy loss, and These are the bounding box coordinate regression loss and the bounding box localization loss, respectively. For binary cross-entropy loss, and , , and This is a constant scalar used to balance the contribution of losses.

9. An electronic device, wherein, The electronic device includes a processor and a memory, the memory storing a computer program executable by the processor, the computer program implementing the method as described in any one of claims 1-8 when executed by the processor.

Images