Timing action detection method and device, electronic equipment and storage medium
By extracting and fusing the original and residual features of video frames in temporal action detection, the problem of low detection accuracy in existing technologies is solved, achieving higher detection accuracy and model transferability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
- Filing Date
- 2023-03-01
- Publication Date
- 2026-06-02
AI Technical Summary
Existing temporal action detection methods do not fully utilize the temporal information between video frames during feature extraction, resulting in low detection accuracy and a lack of transferability of the network model.
By extracting the original features and residual features of video frames and fusing them, the fused features are used to detect action categories and temporal boundaries, and a temporal action detection model is used for analysis.
It improves the accuracy of temporal action detection, enhances the temporal characteristics and transferability of features, and is suitable for temporal action detection in smart devices.
Smart Images
Figure CN116386134B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision technology, and more specifically, to a method, apparatus, electronic device, and storage medium for detecting temporal actions. Background Technology
[0002] With the rapid development of devices capable of video recording, video storage is constantly increasing, making the analysis of video content increasingly urgent. Furthermore, since only a small portion of a video often contains events of interest, with the majority being background, the accuracy of action detection in videos is relatively low. In continuous video, Temporal Action Detection (TAD) refers to a technology that simultaneously identifies and accurately locates actions, identifying the video by precisely pinpointing the occurrence segment and category of each action.
[0003] Current mainstream temporal action detection methods use TSN (temporal segment networks) to extract features from videos to obtain dual-stream features (RGB stream + optical flow), and then use different network models to model them to generate boundaries and action classification.
[0004] However, existing methods only consider action changes (shape changes and action temporal changes) between adjacent frames when extracting raw two-stream features. But actions often last for long frames, sometimes even thousands. Simply using the raw feature stream cannot fully represent the changes in action over the entire time series. Modeling temporal information focuses on the network module level, resulting in network models that lack transferability and fail to fully utilize the inherent temporal nature of the features, leading to low accuracy in temporal action determination and detection.
[0005] As can be seen from the above, how to improve the accuracy of timing action detection still needs to be addressed. Summary of the Invention
[0006] This application provides a method, apparatus, electronic device, and storage medium for detecting sequential actions, which can solve the problem of low accuracy in sequential action detection in related technologies. The technical solution is as follows:
[0007] According to one aspect of the embodiments of this application, a temporal action detection method is provided, the method comprising: acquiring a video to be detected; the video to be detected comprising multiple video frames; extracting original features and residual features of the current video frame in the video to be detected; fusing the original features and residual features of the current video frame to obtain fused features; and detecting the category and temporal boundary of actions in the video to be detected based on the fused features to obtain a temporal action detection result.
[0008] According to one aspect of the embodiments of this application, a temporal action detection device is provided, the device comprising: a video acquisition module for acquiring a video to be detected; a feature extraction module for extracting original features and residual features of the current video frame in the video to be detected; a feature fusion module for fusing the original features and residual features of the current video frame to obtain fused features; and an action detection module for detecting the category and temporal boundary of actions in the video to be detected based on the fused features to obtain a temporal action detection result.
[0009] According to one aspect of the embodiments of this application, a storage medium stores a computer program thereon, which, when executed by a processor, implements the timing action detection method as described above.
[0010] According to one aspect of the embodiments of this application, a computer program product includes a computer program stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to implement the timing action detection method as described above when executed.
[0011] The beneficial effects of the technical solution provided in this application are:
[0012] In the above technical solution, the original features and residual features of the current video frame in the video to be detected are extracted. The residual features are obtained by calculating the difference between the original features of the current video frame and the original features of the interval video frames, and taking the absolute value. The original features and residual features of the current video frame are then fused to obtain fused features. A temporal action detection model is used to detect these fused features to obtain the temporal action detection result. This effectively solves the problems in related technologies where feature extraction lacks transferability and does not fully utilize the temporal nature of the features themselves, resulting in low accuracy in temporal action determination and low accuracy in temporal action detection. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.
[0014] Figure 1 This is a schematic diagram illustrating the implementation environment involved in this application according to an exemplary embodiment;
[0015] Figure 2 This is a flowchart illustrating a timing action detection method according to an exemplary embodiment;
[0016] Figure 3 This is a flowchart illustrating step 330 according to an exemplary embodiment;
[0017] Figure 4 This is a flowchart illustrating another timing action detection method according to an exemplary embodiment;
[0018] Figure 5 This is a structural block diagram of a timing action detection device according to an exemplary embodiment;
[0019] Figure 6 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;
[0020] Figure 7 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation
[0021] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.
[0022] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.
[0023] As mentioned earlier, related technologies only consider the changes in action (shape changes and action temporal changes) between adjacent frames when extracting the original feature stream. However, actions usually last for a long number of frames, and long actions can even last for thousands of frames. Simply using the original feature stream cannot fully represent the changes in action over the entire time series, resulting in insufficient temporal information modeling and thus low accuracy in temporal action detection.
[0024] It is evident that the relevant technologies still suffer from low accuracy in timing action detection.
[0025] Therefore, the temporal action detection method provided in this application can effectively extract the temporal characteristics of features and has good transferability, thereby improving the detection accuracy of temporal actions. Accordingly, the temporal action detection method is applicable to a temporal action detection device, which can be deployed on a smart device. For example, the smart device can be a computer device that deploys the von Neumann architecture, such as a desktop device, a laptop, or a server.
[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0027] Figure 1 This is a schematic diagram of the implementation environment involved in a timing action detection method. The implementation environment includes a data acquisition terminal 110 and a server terminal 130.
[0028] Specifically, the acquisition terminal 110 acquires the video to be tested. The acquisition terminal can be an electronic device such as a smartphone, tablet, laptop, desktop computer, or other device with video acquisition function, and there is no limitation on it here.
[0029] The acquisition terminal 110 and the server terminal 130 can establish a communication connection through wired or wireless means to realize data transmission between them. For example, the transmitted data can be the video to be detected.
[0030] The server-side 130 can also be considered as the cloud, cloud platform, platform-side, server-side, etc. This server-side 130 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center consisting of multiple servers, in order to better provide backend services to the massive data collection terminal 110. For example, the backend services include time-series action detection services.
[0031] With the interaction between the acquisition terminal 110 and the server terminal 130, in an application scenario, the acquisition terminal 110 acquires time-series motion video and transmits the acquired time-series motion video to the server terminal 130. After receiving the time-series motion video transmitted by the acquisition terminal 110, the server terminal 130 can perform time-series motion detection on the time-series motion video.
[0032] Of course, in another application scenario, the acquisition terminal 110 can also simultaneously acquire and detect time-series motion videos, which is not limited here.
[0033] Please see Figure 2 This application provides a timing action detection method, applicable to electronic devices. The electronic device can be... Figure 1 The image shows the acquisition terminal 110 in the implementation environment, which can also be... Figure 1 The server 130 in the implementation environment is shown.
[0034] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.
[0035] like Figure 2 As shown, the method may include the following steps:
[0036] Step 310: Obtain the video to be tested.
[0037] The embodiments of this application can be applied to electronic devices, such as cameras, video recorders, or other electronic devices with video capture capabilities.
[0038] Step 330: Extract the original features and residual features of the current video frame in the video to be detected.
[0039] In one possible implementation, the original features of the current video frame in the video to be detected are dual-stream features. Specifically, these dual-stream features are RGB stream features and optical flow features. RGB stream features, also called frame stream features, are the static features of a single frame in the video to be detected. Optical flow refers to the motion trajectory of objects in the video to be detected; areas with more pronounced motion have higher brightness. By extracting optical flow features, background noise and other motion-irrelevant features in the video to be detected can be removed. Optionally, optical flow feature extraction methods include one or more of the Lucas-Cornard method and the Horn-Shank method.
[0040] The inventors realized that since the actions in videos are usually continuous over long frames, simply considering the changes in actions between adjacent frames would result in insufficient feature modeling. Therefore, they proposed a method for generating residual features to enhance the temporal feature modeling of the network model.
[0041] In one exemplary embodiment, such as Figure 3 As shown, step 330 may include the following steps:
[0042] Step 331: Based on multiple video frames in the video to be detected, select the video frame whose distance from the current video frame is a set window length as the interval video frame.
[0043] Specifically, this application sets a fixed window K to extract video frames that are K distances from the current video frame as interval video frames. It is worth noting that the value of K is set based on the best value obtained from multiple experiments; that is, it can be flexibly set according to different experimental scenarios and models.
[0044] Step 333: Using the TSN network and I3D network, the spatiotemporal information in the current video frame and the interval video frames is extracted to obtain the original features of the current video frame and the original features of the interval video frames.
[0045] In one exemplary embodiment, based on the obtained current video frame and its interval video frames, a TSN (temporal-segment-networks) network is used to perform dual-stream feature extraction to obtain the original features T1 of the current video frame and the original features T of the interval video frames. k .
[0046] Step 335: Calculate the difference between the original features of the current video frame and the original features of the interval video frames, and take the absolute value to obtain the residual features of the current video frame.
[0047] Tc=|T1-T k |
[0048] Where T1 represents the original features of the current video frame, T k Tc represents the original features of the interval video frames, and Tc represents the residual features of the current video frame.
[0049] Step 350: Fuse the original features and residual features of the current video frame to obtain the fused features.
[0050] The inventors realized that simple feature addition would affect the original feature distribution, so they adopted different feature fusion methods for different models.
[0051] One possible implementation is the transformer model.
[0052] Specifically, the original features extracted by multiple attention modules in the transformer model are concatenated with the residual features through feature channels to obtain the input features;
[0053] The input features are decoded by the attention decoding module to obtain the fused features.
[0054] Through the above steps, the residual features of the current video in the video to be detected are extracted, and the residual features are fused with the original features to obtain fused features. This enhances the temporal feature modeling of features in the video to be detected and serves as a new input for the temporal action detection model, exhibiting better transferability.
[0055] Step 370: Detect the category and temporal boundary of the action in the video to be detected based on the fusion features to obtain the temporal action detection result.
[0056] In one possible implementation, the temporal action detection result is obtained by calling a temporal action detection model, which is a trained machine learning model capable of detecting the category and temporal boundaries of actions in the video to be detected.
[0057] Using the fused features obtained above as input, the temporal action detection model is used to detect the input, and the temporal action detection result of the video to be detected is obtained. The temporal action detection result represents the category and temporal boundary of the action in the video to be detected.
[0058] It is worth mentioning that the embodiments of this application are a plug-and-play method that can directly replace the feature extraction module in existing temporal action detection models, such as transformer-based BCNet and convolution-based BMN. Combined with publicly available databases such as THUMOS14 and ActivityNet-v1.3, the temporal action detection process can be completed, as shown in Tables 1 and 2.
[0059] Table 1 shows the experimental results on the THUMOS14 database.
[0060]
[0061]
[0062] Table 2 shows the experimental results on the ActivityNet-v1.3 database.
[0063]
[0064]
[0065] Tables 1 and 2 show the experimental results of this application on two public databases, and the experiments were conducted in conjunction with the BCNet network. As can be seen from the tables, this application has different performance improvements on different databases and different evaluation metrics.
[0066] Table 3. Experimental results of combining with BMN network
[0067] Method 0.3 0.4 0.5 0.6 0.7 BMN 59.3 54.3 45.1 35.3 25.8 BMN+ This application 61.0 56.1 46.2 36.8 26.3
[0068] Table 3 shows the experimental results of this application on the BMN network. It can be seen that even when combined with the new network, the accuracy of the experiment is improved, indicating that this application has good network portability.
[0069] In one exemplary embodiment, such as Figure 4 As shown, after step 370, the above method may further include the following steps:
[0070] Step 390: Based on the temporal action detection results of the video to be detected, locate the target video segment in the video to be detected.
[0071] Optionally, temporal action detection, as a subtask of video search, can identify videos by accurately locating the occurrence segment and category of each action in the video. This helps video search tasks quickly identify target videos and segments from videos such as traffic videos, surveillance videos, video summary extraction, and video object recognition, thus completing the video search task.
[0072] The following are embodiments of the apparatus described in this application, which can be used to execute the timing action detection method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the timing action detection method involved in this application.
[0073] Please see Figure 5 This application provides a timing action detection device 500, including but not limited to: a video acquisition module 510, a feature extraction module 520, a feature fusion module 530, and an action detection module 540.
[0074] The video acquisition module 510 is used to acquire the video to be detected.
[0075] The feature extraction module 520 is used to extract the original features and residual features of the current video frame in the video to be detected.
[0076] The feature fusion module 530 is used to fuse the original features and residual features of the current video frame to obtain fused features.
[0077] The action detection module 540 is used to detect the category and temporal boundary of actions in the video to be detected based on the fused features, and obtain the temporal action detection results.
[0078] In an exemplary embodiment, the timing action detection device 500 further includes a video positioning module, used to locate a target video segment in the video to be detected based on the timing action detection results of the video to be detected.
[0079] It should be noted that the timing action detection device provided in the above embodiments is only illustrated by the division of the above functional modules when performing timing action detection. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the timing action detection device will be divided into different functional modules to complete all or part of the functions described above.
[0080] Furthermore, the timing action detection device and the timing action detection method provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.
[0081] Figure 6 A schematic diagram of the structure of an electronic device is shown according to an exemplary embodiment.
[0082] It should be noted that this electronic device is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this electronic device should not be interpreted as requiring or needing to have [certain features]. Figure 6 One or more components in the exemplary electronic device 2000 shown.
[0083] The hardware configuration of Electronic Device 2000 can vary considerably due to differences in configuration or performance, such as... Figure 6 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.
[0084] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.
[0085] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices.
[0086] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 6 As shown, this does not constitute a specific limitation.
[0087] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.
[0088] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0089] Application 253 is a computer program that performs at least one specific task based on operating system 251, and may include at least one module ( Figure 6(Not shown), each module may contain a computer program for the electronic device 2000. For example, the timing action detection device may be considered as an application program 253 deployed on the electronic device 2000.
[0090] Data 255 can be photos, pictures, etc. stored on a disk, or it can be a video to be detected, stored in memory 250.
[0091] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer programs stored in the memory 250, thereby performing operations and processing on massive amounts of data 255 stored in the memory 250. For example, a timing action detection method may be implemented by the central processing unit 270 reading a series of computer programs stored in the memory 250.
[0092] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.
[0093] Please see Figure 7 This application provides an electronic device 4000, which may include: smart door lock, drone, desktop computer, laptop computer, tablet computer, server, etc.
[0094] exist Figure 7 The electronic device 4000 includes at least one processor 4001, at least one communication bus 4002, and at least one memory 4003.
[0095] The processor 4001 and memory 4003 are connected, for example, via a communication bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.
[0096] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0097] The communication bus 4002 may include a path for transmitting information between the aforementioned components. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0098] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0099] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002.
[0100] When the computer program is executed by the processor 4001, it implements the timing action detection method in the above embodiments.
[0101] Furthermore, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the timing action detection method in the above embodiments.
[0102] This application provides a computer program product comprising a computer program stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to perform the timing action detection method described in the above embodiments.
[0103] Compared with related technologies, on the one hand, the embodiments of this application, as a plug-and-play method, can conveniently and quickly replace the feature extraction operations in related technologies and combine with existing commonly used models for temporal action detection to complete the temporal action detection process; on the other hand, without increasing the amount of computation, this application improves the accuracy of temporal action detection by combining different network models, and has both simplicity and transferability, making it more practical.
[0104] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0105] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for detecting sequential actions, characterized in that, The method includes: Acquire the video to be detected; the video to be detected includes multiple video frames; Extract the original features of the video frames in the video to be detected; Extract residual features from video frames in the video to be detected, wherein, based on a set window length, determine the interval video frames that are a distance of the set window length from the current video frame; calculate the absolute value of the difference between the original features of the current video frame and the original features of the interval video frames to obtain the residual features of the current video frame. The original features and residual features of the current video frame are fused to obtain the fused features; Based on the fusion features, the category and temporal boundary of the action in the video to be detected are detected to obtain the temporal action detection result.
2. The method as described in claim 1, characterized in that, The step of extracting the residual features of the current video frame in the video to be detected includes: Based on multiple video frames in the video to be detected, video frames whose distance from the current video frame is a set window length are used as interval video frames. Using TSN and I3D networks, spatiotemporal information is extracted from the current video frame and the interval video frames to obtain the original features of the current video frame and the interval video frames. Calculate the difference between the original features of the current video frame and the original features of the video frames in between, and take the absolute value to obtain the residual features of the current video frame.
3. The method as described in claim 1, characterized in that, The original features include dual-stream features, which include RGB stream features and optical stream features.
4. The method as described in claim 1, characterized in that, The step of fusing the original features and residual features of the current video frame to obtain fused features includes: The original features and residual features of the current video frame are concatenated to obtain the fused features.
5. The method as described in claim 4, characterized in that, The step of concatenating the original features and residual features of the current video frame to obtain the fused features includes: The original features extracted by multiple attention coding modules are concatenated with the residual features through feature channels to obtain the input features; The input features are decoded by the attention decoding module to obtain the fused features.
6. The method as described in claim 1, characterized in that, The temporal action detection result is obtained by calling the temporal action detection model, which is a machine learning model that has been trained and has the ability to detect the category and temporal boundary of actions in the video to be detected. The extraction of residual features from the current video frame in the video to be detected is achieved by inserting the feature residual module into the temporal action detection model.
7. The method as described in any one of claims 1 to 6, characterized in that, After detecting the category and temporal boundary of actions in the video to be detected based on the fusion features to obtain the temporal action detection result, the method further includes: Based on the temporal action detection results of the video to be detected, the target video segment is located in the video to be detected.
8. A timing action detection device, characterized in that, The device includes: The video acquisition module is used to acquire the video to be detected. The feature extraction module is used to extract the original features and residual features of the current video frame in the video to be detected; wherein, the step of extracting the residual features includes: determining the interval video frames that are separated from the current video frame by the set window length based on a set window length; calculating the absolute value of the difference between the original features of the current video frame and the original features of the interval video frames to obtain the residual features of the current video frame; The feature fusion module is used to fuse the original features and residual features of the current video frame to obtain fused features; The action detection module is used to detect the category and temporal boundaries of actions in the video to be detected based on the fused features, and obtain the temporal action detection results.
9. An electronic device, characterized in that, include: At least one processor, at least one memory, and at least one communication bus, wherein, The memory stores a computer program, and the processor reads the computer program from the memory via the communication bus; When the computer program is executed by the processor, it implements the timing action detection method according to any one of claims 1 to 7.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the timing action detection method as described in any one of claims 1 to 7.