Timing action detection method and device, computer device and storage medium

Abstract

The application provides a timing action detection method and device, computer equipment and a storage medium, and belongs to the field of artificial intelligence. The method comprises the following steps: performing feature extraction on a plurality of continuous image frames to obtain a plurality of timing features with different lengths; obtaining a target feature based on the longest timing feature in the plurality of timing features; for any timing feature in the plurality of timing features, obtaining first time positioning information based on the timing feature, wherein the first time positioning information represents the action start time and the action end time corresponding to each target moment in the timing feature; and correcting the first time positioning information based on the timing feature, the first time positioning information and the target feature to obtain second time positioning information. The above technical solution does not need to set anchor box values for each target moment to predict the corresponding positioning box, thereby reducing the processing amount and accelerating the action detection speed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, apparatus, computer device, and storage medium for detecting sequential motions. Background Art

[0002] Sequential action recognition technology has been widely used in fields such as intelligent security, elderly and vulnerable population monitoring, and robotics research. Sequential action recognition often requires sequential action detection. Sequential action detection involves identifying action segments within a specific timeframe to determine the start and end points of each action. Due to the fuzzy nature of action boundaries, sequential action detection is a challenging task.

[0003] Many temporal action detection methods refer to anchor-based object detection methods. However, these methods usually require setting anchor values ​​for video segments before detection, resulting in large processing load and slow processing speed. Summary of the Invention

[0004] The present invention provides a method, apparatus, computer device, and storage medium for detecting temporal action, which eliminates the need to set anchor box values ​​for each target moment to predict the corresponding positioning box, thereby reducing processing volume and accelerating action detection. The technical solution is as follows:

[0005] In one aspect, a method for detecting a time sequence action is provided, the method comprising:

[0006] Perform feature extraction on multiple consecutive image frames to obtain multiple time series features of different lengths;

[0007] Acquire a target feature based on the longest time series feature among the multiple time series features;

[0008] For any one of the multiple time series features, obtaining first time positioning information based on the time series feature, where the first time positioning information represents an action start time and an action end time corresponding to each target time in the time series feature;

[0009] Based on the time series feature, the first time positioning information and the target feature, the first time positioning information is corrected to obtain second time positioning information.

[0010] In an optional implementation, before extracting features from a plurality of consecutive image frames to obtain a plurality of time series features of different lengths, the method further includes:

[0011] Extracting multiple consecutive image frames from the video to be processed;

[0012] The multiple image frames are adjusted to a target size to obtain multiple image frames of the target size.

[0013] In an optional implementation, the temporal action detection method is implemented based on a temporal action detection model, wherein the temporal action detection model includes a feature extraction layer, a first processing layer, and a second processing layer;

[0014] The feature extraction layer is used to extract features from the plurality of continuous image frames to obtain the plurality of time series features of different lengths;

[0015] The feature extraction layer is further configured to obtain a target feature based on the longest time series feature among the multiple time series features;

[0016] The first processing layer is configured to process any one of the multiple time series features to obtain the first time positioning information;

[0017] The second processing layer is configured to modify the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain the second time positioning information.

[0018] In an optional implementation, the training steps of the temporal action detection model include:

[0019] Based on the feature extraction layer, feature extraction is performed on a plurality of consecutive sample image frames to obtain a plurality of sample time series features of different lengths;

[0020] Acquire a sample target feature based on the feature extraction layer and the longest time series feature among the multiple sample time series features;

[0021] For any sample time series feature, obtaining first sample time location information and first sample classification information based on the first processing layer and the sample time series feature, wherein the first sample time location information represents the action start time and action end time corresponding to each sample moment in the sample time series feature, and the first sample classification information represents the likelihood that each sample moment in the sample time series feature belongs to a different action category;

[0022] Based on the second processing layer, the sample timing characteristics, the first sample time positioning information, and the sample target characteristics, the first sample time positioning information is corrected to obtain second sample time positioning information;

[0023] The temporal action detection model is trained based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information.

[0024] In an optional implementation, the method further includes:

[0025] Based on the sample timing feature, the first sample time positioning information, and the sample target feature, obtaining an activation learning loss and a constraint learning loss, wherein the activation learning loss and the constraint learning loss are used to constrain features containing action start information and features containing action end information;

[0026] The temporal action detection model is trained based on the activation learning loss and the constraint learning loss.

[0027] In an optional implementation, the training of the temporal action detection model based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information includes:

[0028] Obtaining a first coordinate regression loss value based on the first sample time positioning information;

[0029] Obtaining a first classification loss value based on the first sample classification information;

[0030] Obtaining a second coordinate regression loss value based on the second sample time positioning information;

[0031] Obtaining a second classification loss value based on the second sample classification information;

[0032] Adjust parameters of the temporal action detection model based on the first coordinate regression loss value, the first classification loss value, the second coordinate regression loss value, and the second classification loss value.

[0033] In another aspect, a time sequence motion detection device is provided, the device comprising:

[0034] A feature extraction module is used to extract features from multiple consecutive image frames to obtain multiple time series features of different lengths;

[0035] The feature extraction module is further configured to obtain a target feature based on the longest time series feature among the multiple time series features;

[0036] A first positioning information acquisition module is configured to acquire, for any one of the multiple time series features, first time positioning information based on the time series feature, where the first time positioning information represents an action start time and an action end time corresponding to each target time in the time series feature;

[0037] The second positioning information acquisition module is configured to correct the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain second time positioning information.

[0038] In an optional implementation, the feature extraction module is used to perform convolution processing on the multiple consecutive image frames to obtain original time series features, which include feature values ​​belonging to the time dimension; based on the original time series features, generate multiple time series features of different lengths.

[0039] In an optional implementation, the feature extraction module is configured to obtain a first feature containing positioning-related information from the time series features; and based on the first feature, obtain the action start time and action end time corresponding to each target moment.

[0040] In an optional implementation, the apparatus further includes:

[0041] The first classification information acquisition module is configured to acquire a second feature containing classification-related information from the time series feature; based on the second feature, acquire first classification information of each target moment, wherein the first classification information represents the possibility that each target moment belongs to a different action category.

[0042] In an optional implementation, the feature extraction module is used to obtain the longest time series feature from the multiple time series features; and perform upsampling and convolution processing on the longest time series feature to obtain the target feature.

[0043] In an optional implementation, the first positioning information acquisition module includes:

[0044] a salient feature acquisition submodule, configured to perform pooling processing on the time series features and the target features based on the first time positioning information to obtain salient features, wherein the salient features represent the maximum eigenvalue corresponding to the action start time range and the maximum eigenvalue corresponding to the action end time range of the time series features and the target features at each target moment;

[0045] an offset acquisition submodule, configured to acquire, based on the significant features, a first offset and a second offset corresponding to each target moment, wherein the first offset represents an offset from the action start moment corresponding to each target moment, and the second offset represents an offset from the action end moment corresponding to each target moment;

[0046] The positioning information correction submodule is used to correct the action start time and action end time corresponding to each target moment based on the first offset and the second offset corresponding to each target moment to obtain the second time positioning information.

[0047] In an optional implementation, the significant feature acquisition submodule includes:

[0048] A first feature acquisition unit is configured to extract a central feature from the time series feature, where the central feature represents a feature value corresponding to each target moment in the time series feature;

[0049] a second feature acquisition unit, configured to perform pooling processing on the time series features based on the first time positioning information to obtain a first boundary feature, where the first boundary feature represents a maximum feature value corresponding to a range of action start times and a maximum feature value corresponding to a range of action end times at each target moment in the time series features;

[0050] a third feature acquisition unit, configured to perform pooling processing on the target feature based on the first time positioning information to obtain a second boundary feature, where the second boundary feature represents a maximum feature value corresponding to a range of action start times and a maximum feature value corresponding to a range of action end times of each target moment in the target feature;

[0051] A fourth feature acquisition unit is configured to acquire the salient feature based on the central feature, the first boundary feature, and the second boundary feature.

[0052] In an optional implementation, the first feature acquisition unit is configured to extract features from each target moment in the time series feature to obtain the central feature.

[0053] In an optional implementation, the second feature acquisition unit includes:

[0054] A first feature processing subunit, configured to perform convolution processing on the time series feature to obtain a first input feature;

[0055] a first determining subunit, configured to determine, based on the first time positioning information, a first action start range and a first action end range corresponding to each target moment, wherein the first action start range represents a range within which the first action start time corresponding to each target moment belongs, and the first action end range represents a range within which the first action end time corresponding to each target moment belongs;

[0056] The first feature acquisition subunit is configured to perform pooling processing on the first input feature based on the first action start range and the first action end range to acquire the first boundary feature.

[0057] In an optional implementation, the first boundary feature includes a first action start boundary feature and a first action end boundary feature at each target moment;

[0058] The first feature acquisition subunit is used to split the first input feature into a third feature and a fourth feature, where the third feature includes action start information and the fourth feature includes action end information; the maximum feature value within the third feature corresponding to the first action start range of each target moment is determined as the first action start boundary feature of each target moment; the maximum feature value within the fourth feature corresponding to the first action end range of each target moment is determined as the first action end boundary feature of each target moment.

[0059] In an optional implementation, the second feature acquisition unit includes:

[0060] A second feature processing subunit is used to perform convolution processing on the target feature to obtain a second input feature;

[0061] a second determining subunit, configured to determine, based on the first time positioning information, a second action start range and a second action end range corresponding to each target moment, wherein the second action start range represents a range within which the second action start time corresponding to each target moment belongs, and the second action end range represents a range within which the action end time corresponding to each target moment belongs;

[0062] The second feature acquisition subunit is configured to perform pooling processing on the second input feature based on the second action start range and the second action end range to acquire the second boundary feature.

[0063] In an optional implementation, the second boundary feature includes a second action start boundary feature and a second action end boundary feature at each target moment;

[0064] The second feature acquisition subunit is used to split the second input feature into a fifth feature and a sixth feature, the fifth feature including action start information, and the sixth feature including action end information; the maximum feature value within the fifth feature corresponding to the second action start range of each target moment is determined as the second action start boundary feature of each target moment; the maximum feature value within the sixth feature corresponding to the second action end range of each target moment is determined as the second action end boundary feature of each target moment.

[0065] In an optional implementation, the apparatus further includes:

[0066] The image frame acquisition module is used to extract multiple continuous image frames from the video to be processed; and adjust the multiple image frames to a target size to obtain multiple image frames of the target size.

[0067] In an optional implementation, the time-series action detection device is implemented based on a time-series action detection model, and the time-series action detection model includes a feature extraction layer, a first processing layer, and a second processing layer;

[0068] The feature extraction layer is used to extract features from the plurality of continuous image frames to obtain the plurality of time series features of different lengths;

[0069] The feature extraction layer is further configured to obtain a target feature based on the longest time series feature among the multiple time series features;

[0070] The first processing layer is configured to process any one of the multiple time series features to obtain the first time positioning information;

[0071] The second processing layer is configured to modify the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain the second time positioning information.

[0072] In an optional implementation, the training step of the temporal action detection model includes:

[0073] Based on the feature extraction layer, feature extraction is performed on a plurality of consecutive sample image frames to obtain a plurality of sample time series features of different lengths;

[0074] Acquire a sample target feature based on the feature extraction layer and the longest time series feature among the multiple sample time series features;

[0075] For any sample time series feature, obtaining first sample time location information and first sample classification information based on the first processing layer and the sample time series feature, wherein the first sample time location information represents the action start time and action end time corresponding to each sample moment in the sample time series feature, and the first sample classification information represents the likelihood that each sample moment in the sample time series feature belongs to a different action category;

[0076] Based on the second processing layer, the sample timing characteristics, the first sample time positioning information, and the sample target characteristics, the first sample time positioning information is corrected to obtain second sample time positioning information;

[0077] The temporal action detection model is trained based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information.

[0078] In an optional implementation, the training step of the temporal action detection model further includes:

[0079] Based on the sample timing feature, the first sample time positioning information, and the sample target feature, obtaining an activation learning loss and a constraint learning loss, wherein the activation learning loss and the constraint learning loss are used to constrain features containing action start information and features containing action end information;

[0080] The temporal action detection model is trained based on the activation learning loss and the constraint learning loss.

[0081] In an optional implementation, the training of the temporal action detection model based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information includes:

[0082] Obtaining a first coordinate regression loss value based on the first sample time positioning information;

[0083] Obtaining a first classification loss value based on the first sample classification information;

[0084] Obtaining a second coordinate regression loss value based on the second sample time positioning information;

[0085] Obtaining a second classification loss value based on the second sample classification information;

[0086] Adjust parameters of the temporal action detection model based on the first coordinate regression loss value, the first classification loss value, the second coordinate regression loss value, and the second classification loss value.

[0087] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the timed motion detection method in the embodiment of the present application.

[0088] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed in the time sequence motion detection method in the embodiment of the present application.

[0089] In another aspect, a computer program product or computer program is provided, comprising computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the time-series motion detection method provided in each of the aforementioned aspects or various optional implementations of each aspect.

[0090] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0091] The solution provided by the embodiment of the present application extracts multiple time series features of different lengths from multiple image frames, performs rough predictions on each of the multiple time series features, obtains rough time positioning information, and then corrects the rough time positioning information of the multiple time series features based on the target features to obtain more accurate time positioning information. This solution does not require setting anchor frame values ​​for each target moment to predict the corresponding positioning frame, which reduces the amount of processing and speeds up the motion detection speed. Moreover, the longest time series feature among the multiple time series features contains the most features, and the target feature obtained based on the longest time series feature has a finer granularity in time series, and retains more features than other time series features. When the rough time positioning information of the multiple time series features is corrected based on the target feature, the accuracy can be improved while reducing the amount of processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0093] Figure 1 1 is a schematic diagram of an implementation environment of a time sequence motion detection method provided according to an embodiment of the present application;

[0094] Figure 2 is a flow chart of a method for detecting a time sequence action according to an embodiment of the present application;

[0095] Figure 3 is a flowchart of another method for detecting sequential motion according to an embodiment of the present application;

[0096] Figure 4 This is an architecture diagram of a temporal action detection model provided according to an embodiment of the present application;

[0097] Figure 5 is a schematic diagram of obtaining significant features according to an embodiment of the present application;

[0098] Figure 6 This is a visual effect diagram provided according to an embodiment of the present application;

[0099] Figure 7 This is a feature visualization effect diagram provided according to an embodiment of the present application;

[0100] Figure 8is a block diagram of a timing action detection device provided according to an embodiment of the present application;

[0101] Figure 9 This is a structural block diagram of a terminal provided according to an embodiment of the present application;

[0102] Figure 10 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0103] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0104] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0105] The following is a brief introduction to the technologies used in the embodiments of this application:

[0106] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0107] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0108] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Furthermore, it performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0109] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0110] mAP (Mean Average Precision) is a metric used to measure recognition accuracy in object detection. For multi-category object detection, each category can be plotted against a curve based on recall and precision. AP is the area under this curve, and mAP is the average of the AP values ​​for each category.

[0111] IOU (Intersection-over-Union) represents the overlap ratio or degree of overlap between the candidate bounding box and the ground truth bounding box. It is the ratio of the intersection to the union of the two. The higher the correlation between the candidate bounding box and the ground truth bounding box, the higher the IOU. Ideally, the candidate bounding box completely overlaps with the ground truth bounding box, with the IOU being 1.

[0112] The timing action detection method provided in the embodiment of the present application can be applied to a computer device. Optionally, the computer device is a terminal or a server. The following first takes the computer device as an example to introduce the implementation environment of the timing action detection method provided in the embodiment of the present application. Figure 1 Schematic diagram of an implementation environment of a method for detecting sequential motion according to an embodiment of the present application. Figure 1 , the implementation environment includes a terminal 101 and a server 102.

[0113] The terminal 101 and the server 102 can be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0114] Optionally, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0115] Optionally, server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 102 is used to provide background services for applications that support time-series motion detection. Optionally, server 102 undertakes the main work and terminal 101 undertakes the secondary work; or, server 102 undertakes the secondary work and terminal 101 undertakes the main work; or, server 102 and terminal 101 adopt a distributed computing architecture for collaborative computing.

[0116] In this implementation environment, the application supporting time-series motion detection may be a monitoring application, a social application, a multimedia application, and the like.

[0117] For example, the application that supports time-series motion detection is a monitoring application. Terminal 101 shoots a monitoring video and then sends the captured monitoring video to server 102. After receiving the monitoring video, server 102 uses the time-series motion detection method provided in the embodiment of the present application to determine the start time and end time of each action of the person in the monitoring video, thereby determining the action clip of the person in the monitoring video, and then sends the action clip of the person to terminal 101 for display by terminal 101.

[0118] Optionally, the server 102 is composed of an access server, a sequential action detection server, and a database server. The access server is used to provide access services for the terminal 101. The sequential action detection server is used to provide sequential action detection services. The sequential action detection server can be one or more servers. When there are multiple sequential action detection servers, there are at least two sequential action detection servers for providing different services, and / or there are at least two sequential action detection servers for providing the same service, such as providing the same service in a load balancing manner, which is not limited in this embodiment of the present application. Optionally, a sequential action detection model is provided in the sequential action detection server, so that sequential action detection is performed based on the sequential action detection model.

[0119] Terminal 101 may generally refer to one of multiple terminals. This embodiment uses terminal 101 as an example. Those skilled in the art will appreciate that the number of terminals may be greater or lesser. For example, there may be only one terminal, or there may be dozens, hundreds, or even more terminals. This embodiment of the application does not limit the number or device type of terminals.

[0120] In another optional implementation, the computer device is a terminal. Optionally, the terminal is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited thereto. The terminal has installed and runs an application that supports timed motion detection. Optionally, the application is a monitoring application, a social application, or a multimedia application. Illustratively, the terminal is a terminal used by a user, and the application running on the terminal has a user account logged in.

[0121] For example, the terminal is installed with a multimedia application. The terminal obtains the video to be processed based on the multimedia application, and then uses the time-series action detection method provided in the embodiment of the present application to detect the character actions included in the video, and mark the start time and end time of each action. After that, the terminal user can edit the video based on the start time and end time of each action to obtain action clips.

[0122] Figure 2 is a flow chart of a method for detecting a time sequence action according to an embodiment of the present application. Figure 2 As shown, in the embodiment of the present application, the method for detecting sequential motions is applied to a computer device as an example. The method includes the following steps:

[0123] 201. The computer device extracts features from a plurality of consecutive image frames to obtain a plurality of time series features of different lengths.

[0124] In an embodiment of the present application, the multiple continuous image frames are all the image frames included in the video to be detected, or are partially continuous image frames included in the video to be detected, and the embodiment of the present application does not limit this. The multiple time series features retain the features of the time dimension in the multiple image frames, and the multiple time series features are obtained by convolution of convolution kernels of different sizes by a computer device. Due to the different convolution kernels used, the speed of reduction of the time dimension is different. If the receptive field size of the time dimension is larger than the spatial dimension, the edge information of different objects will be merged. Otherwise, the dynamic scene will not be captured. Therefore, different convolution kernels can be used to obtain time series features of different lengths, and time series features of different lengths correspond to different time lengths.

[0125] 202. The computer device obtains a target feature based on the longest time series feature among the multiple time series features.

[0126] In the embodiment of the present application, the longest time series feature retains the most features, and the computer device can obtain the target feature corresponding to the feature that retains the most features by processing the longest time series feature. During the time series action detection process, the computer device can perform auxiliary processing based on the target feature when processing the above multiple time series features.

[0127] 203. For any time series feature among the multiple time series features, the computer device obtains first time positioning information based on the time series feature, where the first time positioning information represents the action start time and action end time corresponding to each target time in the time series feature.

[0128] In an embodiment of the present application, for any of the multiple time series features, the computer device determines at least one target moment based on the length of the time series feature. For each target moment, the computer device can roughly predict the action start time and action end time corresponding to each target moment based on the time series feature to obtain the first time positioning information. The action start time represents the start time of the action corresponding to each target moment, and the action end time represents the end time of the action corresponding to each target moment.

[0129] 204. The computer device modifies the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain second time positioning information.

[0130] In an embodiment of the present application, the computer device further extracts features related to the start time and end time of the action from the above-mentioned timing features and target features based on the first time positioning information obtained above. By introducing the target features, the computer device can more accurately predict the start time and end time of the action corresponding to each target moment, thereby correcting the first time positioning information and obtaining more accurate second time positioning information.

[0131] The solution provided by the embodiment of the present application extracts multiple time series features of different lengths from multiple image frames, performs rough predictions on each of the multiple time series features, obtains rough time positioning information, and then corrects the rough time positioning information of the multiple time series features based on the target features to obtain more accurate time positioning information. This solution does not require setting anchor frame values ​​for each target moment to predict the corresponding positioning frame, which reduces the amount of processing and speeds up the motion detection speed. Moreover, the longest time series feature among the multiple time series features contains the most features, and the target feature obtained based on the longest time series feature has a finer granularity in time series, and retains more features than other time series features. When the rough time positioning information of the multiple time series features is corrected based on the target feature, the accuracy can be improved while reducing the amount of processing.

[0132] Figure 3 is a flow chart of another method for detecting sequential actions according to an embodiment of the present application. Figure 3 As shown, in the embodiment of the present application, the method for detecting sequential motions is applied to a computer device as an example. The method includes the following steps:

[0133] 301. The computer device extracts features from a plurality of consecutive image frames to obtain a plurality of time series features of different lengths.

[0134] In an embodiment of the present application, a computer device can obtain a video to be processed that requires time-series motion detection, and then extract a plurality of continuous image frames from the video to be processed, wherein the plurality of image frames are all the image frames included in the video to be processed, or are part of the image frames included in the video to be processed. Optionally, after extracting a plurality of continuous image frames from the video to be processed, the computer device adjusts the extracted plurality of image frames to a target size to obtain a plurality of image frames of the target size. The target size is the input size of the time-series motion detection model, and the computer device subsequently needs to be able to perform time-series motion detection based on the time-series motion detection model, so the plurality of image frames are first adjusted to the target size required by the time-series motion detection model.

[0135] For example, the computer device extracts N consecutive image frames from the video to be processed, and then processes the size of the N consecutive image frames into 96×96, that is, image frames with a width and height of 96 pixels. Wherein, N is a positive integer greater than one. Of course, the computer device can also adjust the N image frames to other sizes according to the requirements of the temporal action detection model, and the embodiments of the present application are not limited to this.

[0136] In an optional implementation, the temporal action detection model includes a feature extraction layer. Based on the feature extraction layer of the temporal action detection model, the computer device can perform feature extraction on the multiple image frames, thereby obtaining multiple temporal features of different lengths. The feature extraction layer can perform convolution processing on the multiple image frames, compressing the features of the spatial dimension while retaining the features of the temporal dimension, thereby obtaining the original temporal features. The computer device then processes the original temporal features to generate multiple temporal features of different lengths.

[0137] Optionally, the computer device can generate multiple time series features of different lengths based on an image pyramid network, can also generate multiple time series features of different lengths based on a 1D (one-dimensional) convolutional network with a step size of 2 and the original time series features, can also pool the original time series features based on a 1D convolutional network with a step size of 1 to generate multiple time series features of different lengths, and can also resize the original time series features based on a 1D convolutional network with a step size of 1 to generate multiple time series features of different lengths. The embodiment of the present application does not limit the method of generating multiple time series features of different lengths.

[0138] For example, Figure 4 As shown, Figure 4 This is an architecture diagram of a temporal action detection model provided according to an embodiment of the present application. Figure 4 As shown, the temporal action detection model includes a feature extraction layer, a first processing layer and a second processing layer. Among them, the feature extraction layer includes I 3D (Two-Stream Inflated 3DConvNet, two-stream expanded 3D convolutional network) and an image pyramid network. The computer device inputs N consecutive image frames into the I 3D network, and then compresses the feature dimensions of each image frame into a one-dimensional original temporal feature through convolution processing based on the I 3D network, that is, compresses the features of the spatial dimension and retains the features of the time dimension. Then, based on the image pyramid network, the original temporal feature is processed to generate a pyramid feature. The pyramid includes multiple layers, and the pyramid features of each layer are temporal features of different lengths, such as f l ∈T l ×C,f l Represents the pyramid feature of the lth layer, that is, the lth temporal feature, T l represents the length of the time series feature, C represents the number of channels of the time series feature, and l represents any positive integer.

[0139] 302. The computer device obtains a target feature based on the longest time series feature among the multiple time series features.

[0140] In the embodiment of the present application, since the length of the time series features at the bottom of the pyramid network is relatively short, the time series features at the bottom contain fewer features. Therefore, the computer device can also obtain a target feature, which retains more features than other time series features. The target feature is obtained by: the computer device obtains the longest time series feature from the above multiple time series features, and then performs upsampling and convolution processing on the longest time series feature to obtain the target feature.

[0141] For example, see Figure 4 As shown, the computer device obtains the longest temporal feature f from the pyramid feature l-i ∈T l-i ×C, that is, the (li)th time series feature, l and i represent positive integers. l-i Indicates the length of the longest time series feature, and C indicates the number of channels of the longest time series feature. The computer device then performs upsampling and convolution processing on the longest time series feature to obtain the target feature f frame ∈T×2C, T represents the length of the target feature, 2C represents the number of channels of the target feature. If the pyramid feature includes 10 temporal features, l=5, then when i=4, f l-i ∈T l-i ×C is the longest time series feature.

[0142] 303. For any time series feature among the multiple time series features, the computer device obtains first time positioning information based on the time series feature, where the first time positioning information represents the action start time and action end time corresponding to each target time in the time series feature.

[0143] In an embodiment of the present application, the computer device can process each time series feature separately based on the first processing layer of the time series action detection model. For any of the multiple time series features, the computer device obtains a first feature from the time series feature, the first feature containing positioning-related information, and then obtains the action start time and action end time corresponding to each target moment based on the first feature.

[0144] Optionally, the computer device can also use distance to measure the interval between the target moment and the action start moment and the action end moment, wherein the distance and time are converted according to a certain ratio, for example, one unit distance is equal to 0.1 seconds, 0.05 seconds, or 0.01 seconds. Accordingly, the computer device can obtain the action start distance and action end distance corresponding to each target moment based on the first feature. The computer device then determines the action start time and action end time corresponding to the target moment based on the action start distance and action end distance corresponding to each target moment.

[0145] Optionally, the computer device can also obtain a second feature from the above-mentioned timing feature, where the second feature contains classification-related information. Based on the second feature, the computer device obtains first classification information for each target moment, where the first classification information represents the possibility that each target moment belongs to a different action category.

[0146] For example, see Figure 4 As shown, the lth time series feature f l For example, computer equipment from f l Extract the first feature containing positioning related information From f l Extract the second feature containing classification related information Computer equipment will Input the first regressor, process the first feature based on the first regressor, and obtain the first time positioning information (Coarse Boundary, also known as rough boundary), where represents the action start time corresponding to the jth target moment in the lth temporal feature, Indicates the end time of the action corresponding to the jth target moment in the lth time series feature. The computer device will Input the first classifier, process the second feature based on the first classifier, and obtain the first classification information (also called rough classification information) of each target moment. Represents the first classification information of the jth target moment in the lth time series feature.

[0147] It should be noted that the computer device can process each temporal feature separately based on the second processing layer of the temporal action detection model. Accordingly, the computer device inputs the temporal action positioning result obtained based on the first processing layer of the temporal action detection model, that is, the first time positioning information, into the second processing layer of the temporal action detection model. The temporal feature and the target feature are also input into the second processing layer.

[0148] 304. The computer device performs pooling processing on the time series feature and the target feature based on the first time positioning information to obtain a significant feature, where the significant feature represents the maximum feature value corresponding to the action start time range and the maximum feature value corresponding to the action end time range of each target moment in the time series feature and the target feature.

[0149] In the embodiment of the present application, the computer device performs pooling processing on the temporal features and the target features through the saliency optimization unit in the second processing layer to obtain saliency features. Accordingly, this step includes sub-steps 3041 to 3044.

[0150] 3041. The computer device extracts a central feature from the time series feature, where the central feature represents the feature value corresponding to each target moment in the time series feature.

[0151] In an embodiment of the present application, the computer device can extract features of each target moment in the time series feature to obtain the above-mentioned central feature.

[0152] For example, see Figure 5 As shown, Figure 5 FIG. 1 is a schematic diagram of obtaining significant features according to an embodiment of the present application. Figure 5 As shown in Figure (a), the first feature used for positioning in the time series feature is expressed as Figure 5 In (a) of the figure, line 1 indicates that the computer device extracts features of each target moment in the time series feature to obtain the central feature.

[0153] 3042. The computer device performs pooling processing on the time series feature based on the first time positioning information to obtain a first boundary feature, where the first boundary feature represents the maximum feature value corresponding to the action start time range and the maximum feature value corresponding to the action end time range of each target moment in the time series feature.

[0154] In an embodiment of the present application, the computer device can perform convolution processing on the temporal feature to obtain a first input feature. Then, based on the first time positioning information, the first action start range and the first action end range corresponding to each target moment are determined. The first action start range represents the range to which the first action start time corresponding to each target moment belongs, and the first action end range represents the range to which the first action end time corresponding to each target moment belongs. The computer device then performs pooling processing on the first input feature based on the first action start range and the first action end range to obtain a first boundary feature.

[0155] For example, see Figure 5 As shown, Figure 5 Line 2 in (a) of the figure indicates that the computer device performs convolution processing on the time series feature to obtain the first input feature T l ×2C. The first positioning information is expressed as represents the action start time corresponding to the k-th target moment, Indicates the end time of the action corresponding to each target moment in the kth time series feature. During the pooling process, the computer device determines the first action start range corresponding to the kth target moment based on the first positioning information. And the range to which the corresponding first action end time belongs in, Indicates the time length of the positioning result corresponding to the k-th target moment. δa and δ b Used to control the size of the action start range and the action end range. Optionally, set δ in the experiment a =4,δ b =10, of course, other values ​​can be set, and this embodiment of the present application does not limit this.

[0156] In an optional implementation, the first boundary feature includes the first action start boundary feature and the first action end boundary feature of each target moment. Accordingly, the computer device performs pooling processing on the first input feature based on the first action start range and the first action end range, and the step of obtaining the first boundary feature includes: the computer device splits the first input feature into a third feature and a fourth feature, the third feature includes action start information, and the fourth feature includes action end information, and then the computer device determines the maximum feature value corresponding to the first action start range of each target moment in the third feature as the first action start boundary feature of each target moment, and then the computer device determines the maximum feature value corresponding to the first action end range of each target moment in the fourth feature as the first action end boundary feature of each target moment.

[0157] For example, see Figure 5 As shown, Figure 5 In (a), the first input feature is split into the third feature f according to the number of channels. s ∈T×C and the fourth feature f e ∈T×C. Then during the pooling process, as Figure 5 The computer device shown in Figure (b) is in the first action start range The maximum value of the third feature is obtained by the max processing function to obtain the maximum feature value, and the boundary feature of the first action start at each target moment is obtained. Computer equipment in the first action end range The maximum value of the fourth feature is obtained by the max processing function, and the first action end boundary feature of each target moment is obtained.

[0158] 3043. The computer device performs pooling processing on the target feature based on the first time positioning information to obtain a second boundary feature, where the second boundary feature represents the maximum feature value corresponding to the action start time range and the maximum feature value corresponding to the action end time range of each target moment in the target feature.

[0159] In an embodiment of the present application, the computer device performs convolution processing on the target feature to obtain a second input feature, and then the computer device determines the second action start range and second action end range corresponding to each target moment based on the first time positioning information, the second action start range representing the range to which the second action start time corresponding to each target moment belongs, and the second action end range representing the range to which the action end time corresponding to each target moment belongs. The computer device performs pooling processing on the second input feature based on the second action start range and the second action end range to obtain the second boundary feature.

[0160] For example, see Figure 5 As shown, Figure 5 In FIG. 1 , the computer device performs convolution processing on the target feature to obtain a second input feature. The computer device performs pooling processing on the target feature based on the first time positioning information. The method of pooling processing is not further described.

[0161] In an optional implementation, the second boundary feature includes a second action start boundary feature and a second action end boundary feature at each target moment. Accordingly, the computer device performs pooling processing on the second input feature based on the second action start range and the second action end range, and the step of obtaining the second boundary feature includes: the computer device splits the second input feature into a fifth feature and a sixth feature, the fifth feature includes action start information, and the sixth feature includes action end information, and then the computer device determines the maximum feature value corresponding to the second action start range of each target moment in the fifth feature as the second action start boundary feature of each target moment; and then the computer device determines the maximum feature value corresponding to the second action end range of each target moment in the sixth feature as the second action end boundary feature of each target moment.

[0162] For example, see Figure 5 As shown, Figure 5 The second input feature in (a) is split into the fifth feature according to the number of channels and the sixth characteristic Then the computer equipment follows Figure 5 As shown in Figure (b), based on the second action start range and the second action end range, the second action start boundary feature at each target moment is determined. and the second action end boundary characteristics at the target moment

[0163] It should be noted that the first action start range at each target time may be the same as or different from the second action start range at each target time. The first action end range at each target time may be the same as or different from the second action end range at each target time. This embodiment of the application does not limit this.

[0164] 3044. The computer device obtains a significant feature based on the central feature, the first boundary feature, and the second boundary feature.

[0165] In the embodiment of the present application, the central feature comprises one channel, the first boundary feature comprises two channels, and the second boundary feature comprises two channels. The computer device concatenates the central feature, the first boundary feature, and the second boundary feature along the channel dimension to form a multi-channel feature. The computer device then performs convolution processing on the multi-channel feature to compress the number of channels and obtain a significant feature.

[0166] For example, see Figure 5 As shown, Figure 5 In (a) of FIG, the computer device is arranged according to the first boundary feature (including and two channels), the center feature, and the second boundary feature (including and The two channels are spliced ​​in sequence, and then the spliced ​​features are compressed into significant features

[0167] 305. The computer device obtains a first offset and a second offset corresponding to each target moment based on the significant feature, wherein the first offset represents an offset of the action start moment corresponding to each target moment, and the second offset represents an offset of the action end moment corresponding to each target moment.

[0168] In an embodiment of the present application, the computer device processes the features including positioning-related information in the salient features through the second regressor in the second processing layer to obtain the offset of the action start time corresponding to each target moment and the offset of the action end time corresponding to each target moment. The first offset refers to the offset of each target moment relative to the action start time corresponding to each target moment in the first time positioning information. The second offset refers to the offset of each target moment relative to the action end time corresponding to each target moment in the first time positioning information.

[0169] In an optional implementation, the computer device can also obtain second classification information of each target moment based on the significant feature, and the second classification information can more accurately represent the possibility that each target moment belongs to a different action category.

[0170] For example, see Figure 4 As shown, the computer device converts the lth time series feature f l , first-time positioning information And the target feature f frame ∈T×2C is input into the second processing layer of the temporal action detection model, and the computer device performs pooling processing on the temporal features and target features based on the saliency optimization unit in the second processing layer to obtain saliency features The computer device calculates the location-related information in the significant feature based on the second regressor Processing is performed to determine the offset corresponding to each target moment (Refined Boundary, also known as refined boundary), where Indicates the first offset, The computer device classifies the significant feature including the classification related information based on the second classifier. Process and determine the second classification information (Refined class), Represents the second classification information of the j-th target moment in the l-th time series feature.

[0171] 306. The computer device corrects the action start time and the action end time corresponding to each target moment based on the first offset and the second offset corresponding to each target moment to obtain second time positioning information.

[0172] In an embodiment of the present application, the computer device corrects the action start time corresponding to each target time in the first time positioning information based on the above-mentioned first offset, and corrects the action end time corresponding to each target time in the first time positioning information based on the second offset to obtain the second time positioning information.

[0173] For example, the second time positioning information determined by the computer device based on the above information is: the action start time Action end time in, represents the action start time corresponding to the jth target moment in the lth temporal feature, Indicates the action end time corresponding to the jth target time in the lth time series feature. Indicates the first offset, Indicates the second offset. Represents the lth time series feature f l The time length of the positioning result corresponding to the j-th target moment.

[0174] In an optional implementation, the computer device can also obtain the quality of the action start time and action end time corresponding to each target moment based on the second regressor in the second processing layer. The physical meaning of the quality is the temporal overlap tIoU (temporal IoU) between the temporal action positioning and the actual label positioning. The higher the quality value, the higher the overlap between the positioning and the real label, and therefore the higher the quality of the positioning. Correspondingly, the quality can also represent the accuracy of the trained temporal action detection model, that is, the higher the quality value, the higher the quality of the positioning output by the trained temporal action detection model, and thus the higher the accuracy of the temporal action detection model.

[0175] For example, the final classification result output by the computer device is in, represents the first classification information of the jth target moment in the lth time series feature, represents the second classification information of the jth target moment in the lth time series feature, η l，j Represents the quality value of the jth target moment in the lth time series feature.

[0176] It should be noted that the training steps of the temporal action detection model include: a computer device, based on a feature extraction layer, performs feature extraction on multiple consecutive sample image frames to obtain multiple sample temporal features of varying lengths. Then, the computer device obtains a sample target feature based on the feature extraction layer and the longest temporal feature among the multiple sample temporal features. Then, for any sample temporal feature, the computer device, based on the first processing layer and the sample temporal features, obtains first sample temporal location information and first sample classification information. The first sample temporal location information represents the start and end times of the action corresponding to each sample moment in the sample temporal feature, and the first sample classification information represents the likelihood that each sample moment in the sample temporal feature belongs to a different action category. Then, based on the second processing layer, the sample temporal features, the first sample temporal location information, and the sample target features, the computer device modifies the first sample temporal location information to obtain second sample temporal location information. Finally, the computer device trains the temporal action detection model based on the first sample temporal location information, the first sample classification information, the second sample temporal location information, and the second sample classification information.

[0177] In an optional implementation, the computer device trains the temporal motion detection model based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information, including the following steps: the computer device obtains a first coordinate regression loss value based on the first sample time positioning information; the computer device obtains a first classification loss value based on the first sample classification information; the computer device obtains a second coordinate regression loss value based on the second sample time positioning information; the computer device obtains a second classification loss value based on the second sample classification information; the computer device adjusts the parameters of the temporal motion detection model based on the first coordinate regression loss value, the first classification loss value, the second coordinate regression loss value, and the second classification loss value. Optionally, the computer device can also obtain a quality loss value, and adjust the parameters of the temporal motion detection model based on the quality loss value and the above-mentioned loss values.

[0178] For example, see Figure 4 As shown, the first coordinate regression loss value is expressed as The first classification loss value is expressed as The second coordinate regression loss value is expressed as The second classification loss value is expressed as The mass loss value is expressed as l q .

[0179] It should be noted that since the pooling process uses the max maximum value process to obtain significant boundary features, in order to ensure that the maximum value obtained by the max process is a significant boundary feature, the embodiment of the present application also imposes relevant constraints. Accordingly, the computer device obtains the activation learning loss and the constraint learning loss based on the sample time series features, the first sample time positioning information, and the sample target features, and then trains the time series action detection model based on the activation learning loss and the constraint learning loss.

[0180] Among them, the activation learning loss can explicitly constrain the features containing action start information, such as f s , and features containing action end information, such as f e The computer device first maps the values ​​of the above two features to the range of 0-1 by mapping: Among them, tanh() represents the activation function, mean() represents the average function, and f s represents the feature containing the action start information, f e Indicates a feature containing information about the end of an action. represents f s The mapping, represents f e 's mapping.

[0181] Then, for Each value in If the current target moment is in the start area of ​​the action, set The label is 1, otherwise it is 0; Each value in If the current target moment is in the end area of ​​the action, set The label is 1, otherwise it is 0. Given a label of 1 or 0, the computer device constrains and The action start area refers to the area where the action starts, as determined by the computer device, and the action end area refers to the area where the action ends, as determined by the computer device. Of course, the computer device can also use other methods to determine the action start area and the action end area, such as determining the earliest action start time and the latest action end time, and then using the middle time between these two times as the dividing line, with the area before the middle time being the action start area and the area after the middle time being the action end area.

[0182] Among them, the constraint learning loss can implicitly constrain the features containing action start information, such as f s , and features containing action end information, such as f e The computer device first uses the above 1 and 0 labels to segment a complete action A in the sample video into action A1 and action A2. It then finds a background segment Bg from the background and splices it into a new video segment in the order of A1, Bg, and A2. This video segment is sent to the temporal action detection model to obtain the action start feature and action end feature pairs of A1, Bg, and A2: Because A1 and A2 are continuous actions, the computer device uses the following prior condition: A1's end feature should be close to A2's start feature. Bg's start and end features should be as different from A1's as possible, meaning the distance between Bg and A1 should be as far as possible. This is prior condition 2. Based on these two prior conditions, the computer device uses triplet loss (a loss function) to impose the following constraints: in 1 and 0 represent the labels given by the activation learning loss.

[0183] It should be noted that to further demonstrate the effectiveness of the temporal action detection method provided in this embodiment, a performance comparison of this method with other methods on the THUMOS14 and ActivityNet1.3 datasets is also provided. The evaluation metrics are mAP and IOU, and the evaluation results are shown in Table 1.

[0184] Table 1

[0185]

[0186] As can be seen from Table 1, the temporal action detection method provided in this application is better than other algorithms in all mAP indicators on THUMOS14. On ActivityNet1.3, the temporal action detection method provided in this application achieves the best performance at IoU=0.75 and on average.

[0187] Also, see Figure 6 As shown, Figure 6 This is a visual effect diagram provided according to an embodiment of the present application. Figure 6 As shown, for the video clip containing the long jump action, the correct marked time is 18.1 seconds and 26.1 seconds. Based on the timing detection method provided by this application, the first positioning information output by the computer device is 20.2 seconds and 26.7 seconds, and the second positioning information output by the computer device is 18.1 seconds and 26.2 seconds. This shows that the computer device obtains more accurate second positioning information by correcting the first positioning information. The positioning information output by the RC3D method is 17.9 seconds and 26.6 seconds, which is not as accurate as the result output by the timing detection method provided by this application.

[0188] Also, see Figure 7 As shown, Figure 7 This is a feature visualization effect diagram provided according to an embodiment of the present application. Figure 7 As shown in the figure, for a video clip containing a shot put action, the computer device averages the features in the channel dimension and displays them. When using BCL (Box Constraints Learning), the features learned by the model have obvious peaks in the action start and end areas, which can better represent the significance of the action start and end.

[0189] The solution provided by the embodiment of the present application extracts multiple time series features of different lengths from multiple image frames, performs rough predictions on each of the multiple time series features, obtains rough time positioning information, and then corrects the rough time positioning information of the multiple time series features based on the target features to obtain more accurate time positioning information. This solution does not require setting anchor frame values ​​for each target moment to predict the corresponding positioning frame, which reduces the amount of processing and speeds up the motion detection speed. Moreover, the longest time series feature among the multiple time series features contains the most features, and the target feature obtained based on the longest time series feature has a finer granularity in time series, and retains more features than other time series features. When the rough time positioning information of the multiple time series features is corrected based on the target feature, the accuracy can be improved while reducing the amount of processing.

[0190] Figure 8This is a block diagram of a sequential motion detection device according to an embodiment of the present application. The device is used to perform the steps in the sequential motion detection method described above. Figure 8 The device includes: a feature extraction module 801, a first positioning information acquisition module 802 and a second positioning information acquisition module 803.

[0191] The feature extraction module 801 is used to extract features from a plurality of consecutive image frames to obtain a plurality of time series features of different lengths;

[0192] The feature extraction module 801 is further configured to obtain a target feature based on the longest time series feature among the multiple time series features;

[0193] A first positioning information acquisition module 802 is configured to acquire, for any one of the multiple time series features, first time positioning information based on the time series feature, the first time positioning information representing an action start time and an action end time corresponding to each target time in the time series feature;

[0194] The second positioning information acquisition module 803 is configured to modify the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain second time positioning information.

[0195] In an optional implementation, the feature extraction module 801 is used to perform convolution processing on the multiple consecutive image frames to obtain original time series features, which include feature values ​​belonging to the time dimension; based on the original time series features, multiple time series features of different lengths are generated.

[0196] In an optional implementation, the feature extraction module 801 is configured to obtain a first feature containing positioning-related information from the time series feature; and based on the first feature, obtain the action start time and action end time corresponding to each target moment.

[0197] In an optional implementation, the device further includes:

[0198] The first classification information acquisition module is configured to acquire a second feature containing classification-related information from the time series feature; based on the second feature, acquire first classification information of each target moment, wherein the first classification information represents the possibility that each target moment belongs to a different action category.

[0199] In an optional implementation, the feature extraction module 801 is configured to obtain the longest time series feature from the multiple time series features; and perform upsampling and convolution processing on the longest time series feature to obtain the target feature.

[0200] In an optional implementation, the first positioning information obtaining module 802 includes:

[0201] a salient feature acquisition submodule, configured to perform pooling processing on the time series feature and the target feature based on the first time positioning information to obtain a salient feature, wherein the salient feature represents the maximum eigenvalue corresponding to the action start time range and the maximum eigenvalue corresponding to the action end time range of each target moment in the time series feature and the target feature;

[0202] An offset acquisition submodule, configured to acquire, based on the significant feature, a first offset and a second offset corresponding to each target moment, wherein the first offset represents an offset from the action start moment corresponding to each target moment, and the second offset represents an offset from the action end moment corresponding to each target moment;

[0203] The positioning information correction submodule is used to correct the action start time and the action end time corresponding to each target moment based on the first offset and the second offset corresponding to each target moment to obtain the second time positioning information.

[0204] In an optional implementation, the salient feature acquisition submodule includes:

[0205] A first feature acquisition unit is configured to extract a central feature from the time series feature, where the central feature represents a feature value corresponding to each target moment in the time series feature;

[0206] A second feature acquisition unit is configured to perform pooling processing on the time series feature based on the first time positioning information to obtain a first boundary feature, where the first boundary feature represents a maximum feature value corresponding to a range of action start times and a maximum feature value corresponding to a range of action end times at each target moment in the time series feature;

[0207] a third feature acquisition unit, configured to perform pooling processing on the target feature based on the first time positioning information to obtain a second boundary feature, wherein the second boundary feature represents a maximum feature value corresponding to a range of action start times and a maximum feature value corresponding to a range of action end times at each target moment in the target feature;

[0208] The fourth feature acquisition unit is configured to acquire the salient feature based on the center feature, the first boundary feature, and the second boundary feature.

[0209] In an optional implementation, the first feature acquisition unit is configured to extract features from each target moment in the time series feature to obtain the central feature.

[0210] In an optional implementation, the second feature acquisition unit includes:

[0211] A first feature processing subunit is used to perform convolution processing on the time series feature to obtain a first input feature;

[0212] a first determining subunit, configured to determine, based on the first time positioning information, a first action start range and a first action end range corresponding to each target moment, the first action start range representing a range within which the first action start time corresponding to each target moment belongs, and the first action end range representing a range within which the first action end time corresponding to each target moment belongs;

[0213] The first feature acquisition subunit is configured to perform pooling processing on the first input feature based on the first action start range and the first action end range to acquire the first boundary feature.

[0214] In an optional implementation, the first boundary feature includes a first action start boundary feature and a first action end boundary feature at each target moment;

[0215] The first feature acquisition subunit is used to split the first input feature into a third feature and a fourth feature, where the third feature includes action start information and the fourth feature includes action end information; the maximum feature value within the third feature corresponding to the first action start range of each target moment is determined as the first action start boundary feature of each target moment; the maximum feature value within the fourth feature corresponding to the first action end range of each target moment is determined as the first action end boundary feature of each target moment.

[0216] In an optional implementation, the second feature acquisition unit includes:

[0217] A second feature processing subunit is used to perform convolution processing on the target feature to obtain a second input feature;

[0218] a second determining subunit, configured to determine, based on the first time positioning information, a second action start range and a second action end range corresponding to each target moment, the second action start range representing a range within which the second action start time corresponding to each target moment belongs, and the second action end range representing a range within which the action end time corresponding to each target moment belongs;

[0219] The second feature acquisition subunit is configured to perform pooling processing on the second input feature based on the second action start range and the second action end range to acquire the second boundary feature.

[0220] In an optional implementation, the second boundary feature includes a second action start boundary feature and a second action end boundary feature at each target moment;

[0221] The second feature acquisition subunit is used to split the second input feature into a fifth feature and a sixth feature, the fifth feature including action start information, and the sixth feature including action end information; the maximum feature value within the fifth feature corresponding to the second action start range of each target moment is respectively determined as the second action start boundary feature of each target moment; the maximum feature value within the sixth feature corresponding to the second action end range of each target moment is respectively determined as the second action end boundary feature of each target moment.

[0222] In an optional implementation, the device further includes:

[0223] The image frame acquisition module is used to extract multiple continuous image frames from the video to be processed; adjust the multiple image frames to a target size to obtain multiple image frames of the target size.

[0224] In an optional implementation, the time-series action detection device is implemented based on a time-series action detection model, which includes a feature extraction layer, a first processing layer, and a second processing layer;

[0225] The feature extraction layer is used to extract features from the multiple continuous image frames to obtain multiple time series features of different lengths;

[0226] The feature extraction layer is further used to obtain a target feature based on the longest time series feature among the multiple time series features;

[0227] The first processing layer is configured to process any one of the plurality of time series features to obtain the first time positioning information;

[0228] The second processing layer is used to modify the first time positioning information based on the time series feature, the first time positioning information and the target feature to obtain the second time positioning information.

[0229] In an optional implementation, the training steps of the temporal action detection model include:

[0230] Based on the feature extraction layer, feature extraction is performed on multiple consecutive sample image frames to obtain multiple sample time series features of different lengths;

[0231] Acquire a sample target feature based on the feature extraction layer and the longest time series feature among the multiple sample time series features;

[0232] For any sample time series feature, based on the first processing layer and the sample time series feature, obtain first sample time location information and first sample classification information, the first sample time location information representing the action start time and action end time corresponding to each sample moment in the sample time series feature, and the first sample classification information representing the likelihood that each sample moment in the sample time series feature belongs to a different action category;

[0233] Based on the second processing layer, the sample timing characteristics, the first sample time positioning information and the sample target characteristics, the first sample time positioning information is corrected to obtain second sample time positioning information;

[0234] The temporal action detection model is trained based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information.

[0235] In an optional implementation, the training step of the temporal action detection model further includes:

[0236] Based on the sample timing feature, the first sample time positioning information, and the sample target feature, obtaining an activation learning loss and a constraint learning loss, wherein the activation learning loss and the constraint learning loss are used to constrain features including action start information and features including action end information;

[0237] The temporal action detection model is trained based on the activation learning loss and the constraint learning loss.

[0238] In an optional implementation, the training of the temporal action detection model based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information includes:

[0239] Obtaining a first coordinate regression loss value based on the first sample time positioning information;

[0240] Based on the first sample classification information, obtaining a first classification loss value;

[0241] Obtaining a second coordinate regression loss value based on the second sample time positioning information;

[0242] Based on the second sample classification information, obtaining a second classification loss value;

[0243] Based on the first coordinate regression loss value, the first classification loss value, the second coordinate regression loss value, and the second classification loss value, the parameters of the temporal action detection model are adjusted.

[0244] The solution provided by the embodiment of the present application extracts multiple time series features of different lengths from multiple image frames, performs rough predictions on each of the multiple time series features, obtains rough time positioning information, and then corrects the rough time positioning information of the multiple time series features based on the target features to obtain more accurate time positioning information. This solution does not require setting anchor frame values ​​for each target moment to predict the corresponding positioning frame, which reduces the amount of processing and speeds up the motion detection speed. Moreover, the longest time series feature among the multiple time series features contains the most features, and the target feature obtained based on the longest time series feature has a finer granularity in time series, and retains more features than other time series features. When the rough time positioning information of the multiple time series features is corrected based on the target feature, the accuracy can be improved while reducing the amount of processing.

[0245] It should be noted that the above-mentioned embodiments provide a time-series motion detection device, and only illustrate the division of the above-mentioned functional modules when performing time-series motion detection. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the time-series motion detection device provided in the above-mentioned embodiments and the time-series motion detection method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0246] In the embodiments of the present application, the computer device is a terminal or a server. When the computer device is a terminal, the terminal serves as the execution subject to implement the technical solution provided in the embodiments of the present application. When the computer device is a server, the server serves as the execution subject to implement the technical solution provided in the embodiments of the present application. The embodiments of the present application do not limit this.

[0247] Figure 9 The following is a block diagram of a terminal 900 according to an embodiment of the present application. Terminal 900 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 900 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0248] Typically, the terminal 900 includes a processor 901 and a memory 902 .

[0249] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0250] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is executed by the processor 901 to implement the time sequence action detection method provided in the method embodiment of the present application.

[0251] In some embodiments, terminal 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0252] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0253] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuits related to Near Field Communication (NFC), which is not limited in this application.

[0254] The display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 905 is a touch screen display, it is also capable of collecting touch signals on or above the surface of the display screen 905. These touch signals can be input as control signals to the processor 901 for processing. In this case, the display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 905, disposed on the front panel of the terminal 900; in other embodiments, there can be at least two display screens 905, disposed on different surfaces of the terminal 900 or in a foldable design; in still other embodiments, the display screen 905 can be a flexible display, disposed on a curved or foldable surface of the terminal 900. Furthermore, the display screen 905 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0255] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0256] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 900. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0257] Power supply 908 is used to power various components in terminal 900. Power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is charged via a wired line, while a wireless rechargeable battery is charged via a wireless coil. The rechargeable battery can also support fast charging technology.

[0258] In some embodiments, the terminal 900 further includes one or more sensors 909 , including but not limited to: an acceleration sensor 910 , a gyroscope sensor 911 , a pressure sensor 912 , an optical sensor 913 , and a proximity sensor 914 .

[0259] The accelerometer 910 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 900. For example, the accelerometer 910 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 910. The accelerometer 910 can also be used to collect game or user motion data.

[0260] The gyroscope sensor 911 can detect the orientation and rotation angle of the terminal 900. It can also work with the accelerometer 910 to collect the user's 3D movements of the terminal 900. Based on the data collected by the gyroscope sensor 911, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0261] The pressure sensor 912 can be provided on the side frame of the terminal 900 and / or below the display screen 905. When the pressure sensor 912 is provided on the side frame of the terminal 900, it can detect the user's gripping signal of the terminal 900, and the processor 901 can perform left-hand or right-hand recognition or shortcut operations based on the gripping signal collected by the pressure sensor 912. When the pressure sensor 912 is provided below the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0262] Optical sensor 913 is used to detect ambient light intensity. In one embodiment, processor 901 can control the display brightness of display screen 905 based on the ambient light intensity detected by optical sensor 913. Specifically, when the ambient light intensity is high, the display brightness of display screen 905 is increased; when the ambient light intensity is low, the display brightness of display screen 905 is decreased. In another embodiment, processor 901 can also dynamically adjust the shooting parameters of camera assembly 906 based on the ambient light intensity detected by optical sensor 913.

[0263] The proximity sensor 914, also known as a distance sensor, is typically located on the front panel of the terminal 900. The proximity sensor 914 is used to detect the distance between the user and the front of the terminal 900. In one embodiment, when the proximity sensor 914 detects that the distance between the user and the front of the terminal 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from the screen-on state to the screen-off state. When the proximity sensor 914 detects that the distance between the user and the front of the terminal 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from the screen-off state to the screen-on state.

[0264] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0265] Figure 10This is a structural diagram of a server provided in accordance with an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1001 to implement the time sequence motion detection method provided by each of the above method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0266] An embodiment of the present application also provides a computer-readable storage medium, which is applied to a computer device, and the computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the computer device in the timed motion detection method of the above embodiment.

[0267] The present application also provides a computer program product or computer program, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the time-series motion detection method provided in each of the above embodiments.

[0268] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0269] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for detecting a time sequence action, characterized in that: The method comprises: Perform feature extraction on multiple consecutive image frames to obtain multiple time series features of different lengths; Acquire a target feature based on the longest time series feature among the multiple time series features; For any one of the multiple time series features, obtaining first time positioning information based on the time series feature, where the first time positioning information represents an action start time and an action end time corresponding to each target time in the time series feature; Based on the time series feature, the first time positioning information, and the target feature, the first time positioning information is corrected to obtain second time positioning information; The correcting the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain the second time positioning information includes: Based on the first time positioning information, pooling the time series features and the target features to obtain a significant feature, where the significant feature represents a maximum feature value corresponding to a range of action start times and a maximum feature value corresponding to a range of action end times at each target moment in the time series features and the target features; Based on the significant features, obtaining a first offset and a second offset corresponding to each target moment, wherein the first offset represents an offset from the action start moment corresponding to each target moment, and the second offset represents an offset from the action end moment corresponding to each target moment; Based on the first offset and the second offset corresponding to each target moment, the action start moment and the action end moment corresponding to each target moment are corrected to obtain the second time positioning information.

2. The method according to claim 1, characterized in that The feature extraction is performed on a plurality of continuous image frames to obtain a plurality of time series features of different lengths, including: Performing convolution processing on the plurality of continuous image frames to obtain original time series features, wherein the original time series features include feature values ​​belonging to the time dimension; Based on the original time series features, the multiple time series features of different lengths are generated.

3. The method according to claim 1, characterized in that The acquiring first time positioning information based on the time series feature includes: Acquire a first feature containing positioning related information from the time series feature; Based on the first feature, the action start time and the action end time corresponding to each target time are obtained.

4. The method according to claim 1, wherein The method further comprises: Obtaining a second feature containing classification-related information from the time series feature; Based on the second feature, first classification information of each target moment is obtained, where the first classification information represents the possibility that each target moment belongs to a different action category.

5. The method according to claim 1, wherein The acquiring the target feature based on the longest time series feature among the multiple time series features includes: Obtaining the longest time series feature from the multiple time series features; The longest time series feature is subjected to upsampling and convolution processing to obtain the target feature.

6. The method according to claim 1, characterized in that The performing pooling processing on the temporal features and the target features based on the first time positioning information to obtain significant features includes: Extracting a central feature from the time series feature, wherein the central feature represents a feature value corresponding to each target moment in the time series feature; Based on the first time positioning information, pooling processing is performed on the time series features to obtain a first boundary feature, where the first boundary feature represents a maximum feature value corresponding to a range of action start times and a maximum feature value corresponding to a range of action end times at each target moment in the time series features; Based on the first time positioning information, the target feature is pooled to obtain a second boundary feature, where the second boundary feature represents a maximum feature value corresponding to a range of action start times and a maximum feature value corresponding to a range of action end times at each target moment in the target feature; The salient feature is obtained based on the central feature, the first boundary feature, and the second boundary feature.

7. The method according to claim 6, characterized in that The extracting a central feature from the time series feature includes: Feature extraction is performed on each target moment in the time series feature to obtain the central feature.

8. The method according to claim 6, characterized in that The performing pooling processing on the time series features based on the first time positioning information to obtain a first boundary feature includes: Performing convolution processing on the time series feature to obtain a first input feature; Determining, based on the first time positioning information, a first action start range and a first action end range corresponding to each target moment, wherein the first action start range represents a range within which the first action start moment corresponding to each target moment belongs, and the first action end range represents a range within which the first action end moment corresponding to each target moment belongs; Pooling processing is performed on the first input feature based on the first action start range and the first action end range to obtain the first boundary feature.

9. The method according to claim 8, characterized in that The first boundary features include the first action start boundary features and the first action end boundary features at each target moment; The performing pooling processing on the first input feature based on the first action start range and the first action end range to obtain the first boundary feature includes: Splitting the first input feature into a third feature and a fourth feature, wherein the third feature includes action start information and the fourth feature includes action end information; determining the maximum feature value corresponding to the first action start range at each target moment within the third feature as the first action start boundary feature at each target moment; The maximum feature value within the fourth feature corresponding to the first action end range of each target moment is determined as the first action end boundary feature of each target moment.

10. The method according to claim 6, characterized in that The performing pooling processing on the target feature based on the first time positioning information to obtain a second boundary feature includes: Performing convolution processing on the target feature to obtain a second input feature; Determining, based on the first time positioning information, a second action start range and a second action end range corresponding to each target moment, wherein the second action start range represents a range within which the second action start time corresponding to each target moment belongs, and the second action end range represents a range within which the action end time corresponding to each target moment belongs; Pooling processing is performed on the second input feature based on the second action start range and the second action end range to obtain the second boundary feature.

11. The method according to claim 10, characterized in that The second boundary features include a second action start boundary feature and a second action end boundary feature at each target moment; The performing pooling processing on the second input feature based on the second action start range and the second action end range to obtain the second boundary feature includes: Splitting the second input feature into a fifth feature and a sixth feature, wherein the fifth feature includes action start information and the sixth feature includes action end information; determining the maximum feature value within the fifth feature corresponding to the second action start range at each target moment as the second action start boundary feature at each target moment; The maximum feature value within the sixth feature corresponding to the second action end range at each target moment is determined as the second action end boundary feature at each target moment.

12. The method according to any one of claims 1 to 11, characterized in that Before extracting features from a plurality of consecutive image frames to obtain a plurality of time series features of different lengths, the method further includes: Extracting multiple consecutive image frames from the video to be processed; The multiple image frames are adjusted to a target size to obtain multiple image frames of the target size.

13. The method according to any one of claims 1 to 11, characterized in that The temporal action detection method is implemented based on a temporal action detection model, which includes a feature extraction layer, a first processing layer, and a second processing layer; The feature extraction layer is used to extract features from the plurality of continuous image frames to obtain the plurality of time series features of different lengths; The feature extraction layer is further configured to obtain a target feature based on the longest time series feature among the multiple time series features; The first processing layer is configured to process any one of the multiple time series features to obtain the first time positioning information; The second processing layer is configured to modify the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain the second time positioning information.

14. The method according to claim 13, characterized in that The training steps of the temporal action detection model include: Based on the feature extraction layer, feature extraction is performed on a plurality of consecutive sample image frames to obtain a plurality of sample time series features of different lengths; Acquire a sample target feature based on the feature extraction layer and the longest time series feature among the multiple sample time series features; For any sample time series feature, obtaining first sample time location information and first sample classification information based on the first processing layer and the sample time series feature, wherein the first sample time location information represents the action start time and action end time corresponding to each sample moment in the sample time series feature, and the first sample classification information represents the likelihood that each sample moment in the sample time series feature belongs to a different action category; Based on the second processing layer, the sample timing characteristics, the first sample time positioning information, and the sample target characteristics, the first sample time positioning information is corrected to obtain second sample time positioning information; The temporal action detection model is trained based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information.

15. The method according to claim 14, characterized in that The method further comprises: Based on the sample timing feature, the first sample time positioning information, and the sample target feature, obtaining an activation learning loss and a constraint learning loss, wherein the activation learning loss and the constraint learning loss are used to constrain features containing action start information and features containing action end information; The temporal action detection model is trained based on the activation learning loss and the constraint learning loss.

16. The method according to claim 14, characterized in that The training of the temporal action detection model based on the first sample time positioning information, the first sample classification information, the second sample time positioning information, and the second sample classification information includes: Obtaining a first coordinate regression loss value based on the first sample time positioning information; Obtaining a first classification loss value based on the first sample classification information; Obtaining a second coordinate regression loss value based on the second sample time positioning information; Obtaining a second classification loss value based on the second sample classification information; Adjust parameters of the temporal action detection model based on the first coordinate regression loss value, the first classification loss value, the second coordinate regression loss value, and the second classification loss value.

17. A time sequence motion detection device, characterized in that: The device comprises: A feature extraction module is used to extract features from multiple consecutive image frames to obtain multiple time series features of different lengths; The feature extraction module is further configured to obtain a target feature based on the longest time series feature among the multiple time series features; A first positioning information acquisition module is configured to acquire, for any one of the multiple time series features, first time positioning information based on the time series feature, where the first time positioning information represents an action start time and an action end time corresponding to each target time in the time series feature; A second positioning information acquisition module is configured to modify the first time positioning information based on the time series feature, the first time positioning information, and the target feature to obtain second time positioning information; The first positioning information acquisition module includes: a salient feature acquisition submodule, configured to perform pooling processing on the time series features and the target features based on the first time positioning information to obtain salient features, wherein the salient features represent the maximum eigenvalue corresponding to the action start time range and the maximum eigenvalue corresponding to the action end time range of the time series features and the target features at each target moment; an offset acquisition submodule, configured to acquire, based on the significant features, a first offset and a second offset corresponding to each target moment, wherein the first offset represents an offset from the action start moment corresponding to each target moment, and the second offset represents an offset from the action end moment corresponding to each target moment; The positioning information correction submodule is used to correct the action start time and action end time corresponding to each target moment based on the first offset and the second offset corresponding to each target moment to obtain the second time positioning information.

18. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor and executes the time sequence motion detection method according to any one of claims 1 to 16.

19. A storage medium, characterized in that: The storage medium is used to store at least one computer program, and the at least one computer program is used to execute the time sequence motion detection method according to any one of claims 1 to 16.

Images