Motion detection method and device

Abstract

The present application relates to an action detection method and device in the field of artificial intelligence. The action detection method includes: obtaining a video to be processed; determining a first boundary probability sequence based on the video to be processed, the first boundary probability sequence being used to indicate the first probability that each of the K video frames is the boundary of an action segment, and the first probability that each of the video frames is the boundary of an action segment is determined based on a first set of video frames; determining a second boundary probability sequence based on the video to be processed, the second boundary probability sequence being used to indicate the second probability that each of the K video frames is the boundary of an action segment, and the second probability that each of the video frames is the boundary of an action segment is determined based on a second set of video frames; and determining a target action segment set based on the first boundary probability sequence and the second boundary probability sequence. This method can improve the detection performance of temporal action detection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a motion detection method and device thereof. Background Art

[0002] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0003] With the development of mobile internet, video media has become an increasingly important form of information media, and deep learning technology is increasingly being applied to video media. For example, temporal action detection is a task that involves detecting the start and end times, action categories, and other aspects of a target action in a video. However, current temporal action detection technologies have poor detection performance.

[0004] Therefore, how to improve the detection performance of temporal motion detection has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides a motion detection method and device thereof, which help to improve the detection performance of time-series motion detection.

[0006] In a first aspect, a method for detecting an action is provided, the method comprising: obtaining a video to be processed, the video to be processed comprising K video frames, K being a positive integer; determining a first boundary probability sequence based on the video to be processed, the first boundary probability sequence being used to indicate a first probability that each of the K video frames is a boundary of an action segment, the first probability that each video frame is a boundary of an action segment being determined based on a first video frame set corresponding to each video frame, the first video frame set comprising video frames that are i frames away from each video frame, i being an integer, and the value of i ranging from 1 to N, N being a preset threshold; determining a first boundary probability sequence based on the video to be processed, the first boundary probability sequence being used to indicate a first probability that each of the K video frames is a boundary of an action segment, the first probability that each video frame is a boundary of an action segment being determined based on a first video frame set corresponding to each video frame, the first video frame set ... the first video frame set, the first video frame set being determined based on the first video frame set, the first video frame set being determined based on the first video frame set, the first video frame A second boundary probability sequence is determined based on the processed video, where the second boundary probability sequence is used to indicate the second probability that each of the K video frames is a boundary of an action segment, and the second probability that each video frame is a boundary of an action segment is determined based on a second video frame set corresponding to each video frame, where the second video frame set includes video frames that are j frames away from each video frame, where j is an integer and the value of j is 1 to M, where M is an integer greater than N and less than K; a target action segment set is determined based on the first boundary probability sequence and the second boundary probability sequence, where the target action segment set includes one or more target action segments.

[0007] In an embodiment of the present application, the first boundary probability sequence is determined based on the local information of each video frame (i.e., the first video frame set corresponding to each video frame), and the recall rate of temporal action detection using the first boundary probability sequence is higher. The second boundary probability sequence is determined based on the global information of each video frame (i.e., the second video frame set corresponding to each video frame), and the accuracy of temporal action detection using the second boundary probability sequence is higher. Therefore, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence can improve the accuracy of temporal action detection while ensuring the recall rate, that is, improve the detection capability of action boundaries, thereby improving the detection performance of temporal action detection.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: determining multiple candidate action segments of the video to be processed, each of the multiple candidate action segments corresponds to a candidate action; determining an action attention matrix based on the multiple candidate action segments, the action attention matrix being used to represent the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions; performing confidence regression on the multiple candidate action segments using the action attention matrix to obtain the confidence of the multiple candidate action segments; wherein, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence includes: determining the target action segment set from the multiple candidate action segments based on the confidence of the multiple candidate action segments, the first boundary probability sequence and the second boundary probability sequence.

[0009] In an embodiment of the present application, the action attention matrix determined based on the multiple candidate action segments includes the associated information of the multiple candidate actions (i.e., the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions). The action attention matrix is ​​used to perform confidence regression on the multiple candidate action segments. The associated information of the multiple candidate actions can be incorporated into the confidence regression process. Therefore, the accuracy of the confidence regression of the multiple candidate action segments can be improved, thereby improving the accuracy of temporal action detection.

[0010] In combination with the first aspect, in certain implementations of the first aspect, determining the action attention matrix based on the multiple candidate action segments includes: using a neural network model to determine the action attention matrix based on the multiple candidate action segments; wherein the neural network model is trained by the following method: using a target loss function to calculate the loss value of the neural network model based on the confidence of the multiple candidate action segments, the target loss function can balance the weights of the confidence of the candidate action segments of different scales when calculating the loss value; and updating the parameters of the neural network model based on the loss value.

[0011] In an embodiment of the present application, the number of samples of candidate action segments of different scales is different (generally, the smaller the scale, the fewer the sample numbers of the candidate action segments). The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value. The target loss function is used to calculate the loss value of the neural network model, and the parameters of the neural network model are updated according to the loss value. This can improve the detection ability of the neural network model for small-scale actions, thereby improving the detection performance of temporal action detection.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the target loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value.

[0013] In an embodiment of the present application, by making the confidence weights of the candidate action segments of different scales the same when calculating the loss value, the detection capability of the neural network model for small-scale actions can be further improved, thereby further improving the detection performance of temporal action detection.

[0014] In a second aspect, a method for action detection is provided, which includes: obtaining a video to be processed; determining multiple candidate action segments of the video to be processed, each of the multiple candidate action segments corresponds to a candidate action; determining an action attention matrix based on the multiple candidate action segments, the action attention matrix being used to represent the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions; performing confidence regression on the multiple candidate action segments using the action attention matrix to obtain the confidence of the multiple candidate action segments; determining a target action segment set based on the confidence of the multiple candidate action segments, the target action segment set including one or more target action segments.

[0015] In an embodiment of the present application, the action attention matrix determined based on the multiple candidate action segments includes the associated information of the multiple candidate actions (i.e., the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions. The action attention matrix is ​​used to perform confidence regression on the multiple candidate action segments. The associated information of the multiple candidate actions can be incorporated into the confidence regression process. Therefore, the accuracy of the confidence regression of the multiple candidate action segments can be improved, thereby improving the accuracy of temporal action detection.

[0016] In combination with the second aspect, in certain implementations of the second aspect, determining the action attention matrix based on the multiple candidate action segments includes: using a neural network model to determine the action attention matrix based on the multiple candidate action segments; wherein the neural network model is trained by the following method: using a target loss function to calculate the loss value of the neural network model based on the confidence of the multiple candidate action segments, the target loss function can balance the weights of the confidence of the candidate action segments of different scales when calculating the loss value; and updating the parameters of the neural network model based on the loss value.

[0017] In an embodiment of the present application, the number of samples of candidate action segments of different scales is different (generally, the smaller the scale, the fewer the sample numbers of the candidate action segments). The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value. The target loss function is used to calculate the loss value of the neural network model, and the parameters of the neural network model are updated according to the loss value. This can improve the detection ability of the neural network model for small-scale actions, thereby improving the detection performance of temporal action detection.

[0018] In combination with the second aspect, in certain implementations of the second aspect, the target loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value.

[0019] In an embodiment of the present application, by making the confidence weights of the candidate action segments of different scales the same when calculating the loss value, the detection capability of the neural network model for small-scale actions can be further improved, thereby further improving the detection performance of temporal action detection.

[0020] In combination with the second aspect, in some implementations of the second aspect, the video to be processed includes K video frames, K is a positive integer, and the method further includes: determining a first boundary probability sequence based on the video to be processed, the first boundary probability sequence is used to indicate the first probability that each of the K video frames is a boundary of an action segment, and the first probability that each video frame is a boundary of an action segment is determined based on a first video frame set corresponding to each video frame, the first video frame set includes video frames that are i frames away from each video frame, i is an integer, and the value of i ranges from 1 to N, and N is a preset threshold; determining a second boundary probability sequence based on the video to be processed, the second boundary probability sequence is used to indicate the first probability that each of the K video frames is a boundary of an action segment, and the first probability that each video frame is a boundary of an action segment is determined based on a first video frame set corresponding to each video frame, and the first video frame set includes video frames that are i frames away from each video frame, i is an integer, and the value of i ranges from 1 to N, and N is a preset threshold; determining a second boundary probability sequence based on the video to be processed, The probability sequence is used to indicate the second probability that each of the K video frames is the boundary of an action segment, and the second probability that each video frame is the boundary of an action segment is determined based on a second video frame set corresponding to each video frame, and the second video frame set includes video frames that are j frames away from each video frame, j is an integer, and the value of j is 1 to M, and M is an integer greater than N and less than K; wherein, determining the target action segment set based on the confidence of the multiple candidate action segments includes: determining the target action segment set from the multiple candidate action segments based on the confidence of the multiple candidate action segments, the first boundary probability sequence and the second boundary probability sequence.

[0021] In an embodiment of the present application, the first boundary probability sequence is determined based on the local information of each video frame (i.e., the first video frame set corresponding to each video frame), and the recall rate of temporal action detection using the first boundary probability sequence is higher. The second boundary probability sequence is determined based on the global information of each video frame (i.e., the second video frame set corresponding to each video frame), and the accuracy of temporal action detection using the second boundary probability sequence is higher. Therefore, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence can improve the accuracy of temporal action detection while ensuring the recall rate, that is, improve the detection capability of action boundaries, thereby improving the detection performance of temporal action detection.

[0022] According to a third aspect, an action detection device is provided, comprising: an acquisition module for acquiring a video to be processed, wherein the video to be processed includes K video frames, where K is a positive integer; a first execution module for determining a first boundary probability sequence based on the video to be processed, wherein the first boundary probability sequence is used to indicate a first probability that each of the K video frames is a boundary of an action segment, and the first probability that each video frame is a boundary of an action segment is determined based on a first video frame set corresponding to each video frame, wherein the first video frame set includes video frames that are i frames away from each video frame, where i is an integer and the value of i ranges from 1 to N, where N is a preset threshold; and the first execution module , used to determine a second boundary probability sequence according to the video to be processed, the second boundary probability sequence is used to indicate the second probability that each of the K video frames is the boundary of an action segment, the second probability that each video frame is the boundary of an action segment is determined according to a second video frame set corresponding to each video frame, the second video frame set includes video frames that are j frames away from each video frame, j is an integer, and the value of j is 1 to M, M is an integer greater than N and less than K; a determination module is used to determine a target action segment set according to the first boundary probability sequence and the second boundary probability sequence, the target action segment set including one or more target action segments.

[0023] In an embodiment of the present application, the first boundary probability sequence is determined based on the local information of each video frame (i.e., the first video frame set corresponding to each video frame), and the recall rate of temporal action detection using the first boundary probability sequence is higher. The second boundary probability sequence is determined based on the global information of each video frame (i.e., the second video frame set corresponding to each video frame), and the accuracy of temporal action detection using the second boundary probability sequence is higher. Therefore, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence can improve the accuracy of temporal action detection while ensuring the recall rate, that is, improve the detection capability of action boundaries, thereby improving the detection performance of temporal action detection.

[0024] In combination with the third aspect, in certain implementations of the third aspect, the device also includes a second execution module, and the second execution is used to: determine multiple candidate action segments of the video to be processed, each of the multiple candidate action segments corresponds to a candidate action; determine an action attention matrix based on the multiple candidate action segments, and the action attention matrix is ​​used to represent the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions; use the action attention matrix to perform confidence regression on the multiple candidate action segments to obtain the confidence of the multiple candidate action segments; wherein the determination module is specifically used to: determine the target action segment set based on the confidence of the multiple candidate action segments, the first boundary probability sequence and the second boundary probability sequence.

[0025] In an embodiment of the present application, the action attention matrix determined based on the multiple candidate action segments includes the associated information of the multiple candidate actions (i.e., the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions. The action attention matrix is ​​used to perform confidence regression on the multiple candidate action segments. The associated information of the multiple candidate actions can be incorporated into the confidence regression process. Therefore, the accuracy of the confidence regression of the multiple candidate action segments can be improved, thereby improving the accuracy of temporal action detection.

[0026] In combination with the third aspect, in certain implementations of the third aspect, the second execution module is specifically used to: use a neural network model to determine an action attention matrix based on the multiple candidate action segments; wherein the neural network model is trained by the following method: using a target loss function to calculate the loss value of the neural network model based on the confidence of the multiple candidate action segments, the target loss function can balance the weights of the confidence of the candidate action segments of different scales when calculating the loss value; and updating the parameters of the neural network model based on the loss value.

[0027] In an embodiment of the present application, the number of samples of candidate action segments of different scales is different (generally, the smaller the scale, the fewer the sample numbers of the candidate action segments). The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value. The target loss function is used to calculate the loss value of the neural network model, and the parameters of the neural network model are updated according to the loss value. This can improve the detection ability of the neural network model for small-scale actions, thereby improving the detection performance of temporal action detection.

[0028] In combination with the third aspect, in certain implementations of the third aspect, the target loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value.

[0029] In an embodiment of the present application, by making the confidence weights of the candidate action segments of different scales the same when calculating the loss value, the detection capability of the neural network model for small-scale actions can be further improved, thereby further improving the detection performance of temporal action detection.

[0030] In a fourth aspect, an action detection device is provided, comprising: an acquisition module for acquiring a video to be processed; a second execution module for determining multiple candidate action segments of the video to be processed, each of the multiple candidate action segments corresponding to a candidate action; the second execution module for determining an action attention matrix based on the multiple candidate action segments, the action attention matrix being used to represent the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions; the second execution module for performing confidence regression on the multiple candidate action segments in combination with the action attention matrix to obtain the confidence of the multiple candidate action segments; a determination module for determining a target action segment set based on the confidence of the multiple candidate action segments, the target action segment set including one or more target action segments.

[0031] In an embodiment of the present application, the action attention matrix determined based on the multiple candidate action segments includes the associated information of the multiple candidate actions (i.e., the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions. The action attention matrix is ​​used to perform confidence regression on the multiple candidate action segments. The associated information of the multiple candidate actions can be incorporated into the confidence regression process. Therefore, the accuracy of the confidence regression of the multiple candidate action segments can be improved, thereby improving the accuracy of temporal action detection.

[0032] In combination with the fourth aspect, in certain implementations of the fourth aspect, the second execution module is specifically used to: use a neural network model to determine an action attention matrix based on the multiple candidate action segments; wherein the neural network model is trained by the following method: using a target loss function to calculate the loss value of the neural network model based on the confidence of the multiple candidate action segments, the target loss function can balance the weights of the confidence of the candidate action segments of different scales when calculating the loss value; and updating the parameters of the neural network model based on the loss value.

[0033] In an embodiment of the present application, the number of samples of candidate action segments of different scales is different (generally, the smaller the scale, the fewer the sample numbers of the candidate action segments). The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value. The target loss function is used to calculate the loss value of the neural network model, and the parameters of the neural network model are updated according to the loss value. This can improve the detection ability of the neural network model for small-scale actions, thereby improving the detection performance of temporal action detection.

[0034] In combination with the fourth aspect, in certain implementations of the fourth aspect, the target loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value.

[0035] In an embodiment of the present application, by making the confidence weights of the candidate action segments of different scales the same when calculating the loss value, the detection capability of the neural network model for small-scale actions can be further improved, thereby further improving the detection performance of temporal action detection.

[0036] In combination with the fourth aspect, in some implementations of the fourth aspect, the video to be processed includes K video frames, K is a positive integer, and the device also includes a first execution module, which is used to: determine a first boundary probability sequence based on the video to be processed, the first boundary probability sequence is used to indicate the first probability that each of the K video frames is a boundary of an action segment, and the first probability that each video frame is a boundary of an action segment is determined based on a first video frame set corresponding to each video frame, the first video frame set includes video frames that are i frames away from each video frame, i is an integer, and the value of i ranges from 1 to N, and N is a preset threshold; determine based on the video to be processed A second boundary probability sequence, wherein the second boundary probability sequence is used to indicate the second probability that each of the K video frames is the boundary of an action segment, and the second probability that each video frame is the boundary of an action segment is determined based on a second video frame set corresponding to each video frame, and the second video frame set includes video frames that are j frames away from each video frame, j is an integer, and the value of j is 1 to M, and M is an integer greater than N and less than K; wherein the determination module is specifically used to: determine a target action segment set from the multiple candidate action segments according to the confidence of the multiple candidate action segments, the first boundary probability sequence and the second boundary probability sequence.

[0037] In an embodiment of the present application, the first boundary probability sequence is determined based on the local information of each video frame (i.e., the first video frame set corresponding to each video frame), and the recall rate of temporal action detection using the first boundary probability sequence is higher. The second boundary probability sequence is determined based on the global information of each video frame (i.e., the second video frame set corresponding to each video frame), and the accuracy of temporal action detection using the second boundary probability sequence is higher. Therefore, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence can improve the accuracy of temporal action detection while ensuring the recall rate, that is, improve the detection capability of action boundaries, thereby improving the detection performance of temporal action detection.

[0038] In a fifth aspect, a motion detection device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in any one of the implementation modes of the first aspect or the second aspect.

[0039] The processor in the fifth aspect can be either a central processing unit (CPU) or a combination of a CPU and a neural network processor. The neural network processor here can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU). Among them, the TPU is a dedicated integrated circuit for artificial intelligence accelerators fully customized by Google for machine learning.

[0040] In a sixth aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of the first aspect or the second aspect.

[0041] In a seventh aspect, a computer program product comprising instructions is provided, which, when run on a computer, enables the computer to execute the method in any one of the implementations of the first or second aspect above.

[0042] In an eighth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes a method in any one of the implementations of the first or second aspect.

[0043] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementation methods of the first aspect or the second aspect.

[0044] The chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0045] In an embodiment of the present application, the first boundary probability sequence is determined based on the local information of each video frame (i.e., the first video frame set corresponding to each video frame), and the recall rate of temporal action detection using the first boundary probability sequence is higher. The second boundary probability sequence is determined based on the global information of each video frame (i.e., the second video frame set corresponding to each video frame), and the accuracy of temporal action detection using the second boundary probability sequence is higher. Therefore, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence can improve the accuracy of temporal action detection while ensuring the recall rate, that is, improve the detection capability of action boundaries, thereby improving the detection performance of temporal action detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of the present application.

[0047] Figure 2 A schematic diagram of the system architecture provided in an embodiment of the present application.

[0048] Figure 3 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application.

[0049] Figure 4 A schematic diagram of the structure of another convolutional neural network provided in an embodiment of the present application.

[0050] Figure 5 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application.

[0051] Figure 6 A schematic diagram of a system architecture provided in an embodiment of the present application.

[0052] Figure 7 It is a schematic flowchart of an action detection method according to an embodiment of the present application.

[0053] Figure 8 It is a schematic flowchart of an action detection method according to another embodiment of the present application.

[0054] Figure 9 This is a schematic flowchart of a motion detection method according to another embodiment of the present application.

[0055] Figure 10 It is a schematic block diagram of an action detection method in one embodiment of the present application.

[0056] Figure 11 It is a schematic block diagram of a multi-branch boundary detector in one embodiment of the present application.

[0057] Figure 12 It is a schematic block diagram of a scale-balanced action confidence regressor in one embodiment of the present application.

[0058] Figure 13 It is a schematic flowchart of an action detection method according to an embodiment of the present application.

[0059] Figure 14 It is a schematic flowchart of an action detection method according to another embodiment of the present application.

[0060] Figure 15 Schematic diagram of the hardware structure of the motion detection device according to an embodiment of the present application.

[0061] Figure 16 It is a schematic diagram of the hardware structure of the model training device of an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will describe the technical solutions in the embodiments of this application in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0064] The following is a detailed explanation of the above artificial intelligence thematic framework from the two dimensions of "intelligent information chain" (horizontal axis) and "information technology (IT) value chain" (vertical axis).

[0065] The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom."

[0066] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0067] (1) Infrastructure:

[0068] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and provides support through a basic platform.

[0069] The infrastructure can communicate with the outside world through sensors, and the computing power of the infrastructure can be provided by smart chips.

[0070] The smart chips here can be hardware acceleration chips such as central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), and field programmable gate array (FPGA).

[0071] The basic platform of the infrastructure can include related platform guarantees and support such as distributed computing framework and network, and can include cloud storage and computing, interconnected networks, etc.

[0072] For example, for infrastructure, data can be obtained through sensors and external communication, and then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0073] (2) Data:

[0074] The data layer above the infrastructure represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0075] (3) Data processing:

[0076] The above data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.

[0077] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0078] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0079] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0080] (4) General abilities:

[0081] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0082] (5) Smart products and industry applications:

[0083] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0084] The embodiments of the present application can be applied in many fields of artificial intelligence, such as smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities and other fields.

[0085] Specifically, the embodiments of the present application can be applied in scenarios such as intelligent monitoring, short video extraction, abnormal behavior detection, and autonomous driving. Alternatively, the embodiments of the present application can also be applied in other scenarios where it is necessary to automatically detect (or extract) target video clips (e.g., video clips of interest) in video media. The embodiments of the present application can also be combined with other video technologies, such as video classification or video compression.

[0086] The following briefly introduces the application scenarios of the action detection method in the embodiment of the present application, taking intelligent monitoring and short video extraction as examples.

[0087] Intelligent monitoring:

[0088] When a surveillance camera is monitoring, the motion detection method in the embodiment of the present application can be used to perform time-series motion detection on the captured video images to detect (one or more) motion segments in the video images. When abnormal motion segments appear in the video images, an alarm signal can be issued or corresponding measures can be taken, which can improve the security and stability of the intelligent monitoring system and enhance the user experience.

[0089] Short video extraction:

[0090] For a long video, short videos within the long video can be automatically extracted. For example, the motion detection method in the embodiments of the present application can be used to perform time-series motion detection on the long video, and one or more short videos (e.g., action clips) within the long video can be detected, thereby achieving short video extraction (of the long video).

[0091] Since the embodiments of the present application involve the application of a large number of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0092] (1) Neural Network

[0093] A neural network can be composed of neural units, which can be represented by x s The output of the operation unit with the intercept 1 as input can be:

[0094]

[0095] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0096] (2) Deep Neural Networks

[0097] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0098] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0099] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0100] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0101] (3) Convolutional Neural Network

[0102] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. Convolution kernels can be initialized as matrices of random size, and during CNN training, they can learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0103] (4) Recurrent neural networks (RNNs) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this ordinary neural network solves many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word because the previous and next words in a sentence are not independent. RNN is called a recurrent neural network because the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN.

[0104] Since we already have convolutional neural networks, why do we still need recurrent neural networks? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, such as cats and dogs. However, in the real world, many elements are interconnected, such as the changes in stock prices over time. Or, for example, someone says, "I love traveling, and my favorite place is Yunnan. I must visit it someday." Humans should know to fill in the blank with "Yunnan." This is because humans make inferences based on context, but how can machines do the same? That's where RNNs come in. RNNs aim to give machines the ability to remember, like humans do. Therefore, the output of an RNN depends on both the current input and historical memory.

[0105] (5) Loss function

[0106] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0107] (6) Backpropagation algorithm

[0108] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0109] like Figure 2 As shown, the embodiment of the present application provides a system architecture 100. Figure 2 In the embodiment of the present invention, the data acquisition device 160 is used to collect training data. For the action detection method of the embodiment of the present invention, the training data may include a training video and the ground truth of the training data. The training video includes one or more action segments, and the ground truth of the training data may include the start time, end time, and action category of each action segment in the training video that has been manually pre-labeled.

[0110] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .

[0111] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the training video and compares the output action clips with the true values ​​of the action clips included in the training video until the difference between the action clips output by the training device 120 and the true values ​​of the action clips included in the training video is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0112] The above-mentioned target model / rule 101 can be used to implement the action detection method of the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be the action detection device in the implementation of the present application, and the action detection device may include one (or more) neural network models. It should be noted that, in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application.

[0113] The target model / rule 101 obtained by training the training device 120 can be applied to different systems or devices, such as Figure 2 The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) or virtual reality (VR), a vehicle terminal, etc. It can also be a server or a cloud. Figure 2 In the embodiment of the present application, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data may include: video input by the client device.

[0114] The preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing based on the input data (such as input video) received by the I / O interface 112. In an embodiment of the present application, the preprocessing module 113 and the preprocessing module 114 may not be present (or only one of the preprocessing modules may be present), and the computing module 111 may be used directly to process the input data.

[0115] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0116] Finally, the I / O interface 112 returns the processing result, such as the action fragment obtained above, to the client device 140 for providing to the user.

[0117] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0118] exist Figure 2 In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0119] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0120] like Figure 2As shown, the target model / rule 101 is obtained by training according to the training device 120. In the embodiment of the present application, the target model / rule 101 can be the action detection device in the present application. The action detection device can include multiple neural networks. Specifically, the action detection device can include CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNN), etc.

[0121] Since CNN is a very common neural network, the following Figure 3 This section focuses on a detailed introduction to the structure of CNNs. As mentioned in the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. Deep learning architectures use machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images.

[0122] The structure of the convolutional neural network specifically used in the embodiment of the present application can be as follows: Figure 3 As shown. Figure 3 In FIG, a convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230 .

[0123] In the embodiment of the present application, the video frame can be considered as an image. Therefore, the structure of the convolutional neural network is introduced by taking image processing as an example. For example, the input layer 210 can obtain the image to be processed, and pass the obtained image to be processed to the convolution layer / pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result can be obtained. Figure 3 The internal layer structure of CNN 200 in

[15] is introduced in detail.

[0124] Convolutional layer / pooling layer 220:

[0125] Convolutional layer:

[0126] like Figure 3The convolutional layer / pooling layer 220 shown may include layers 221-226. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer. In another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0127] The following will take convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0128] Convolution layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from an input image matrix. A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), thereby extracting specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used. Instead, multiple weight matrices of the same size (rows × columns) are applied, i.e., multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image, where the dimension is determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix is ​​used to extract edge information, another weight matrix is ​​used to extract specific colors from an image, and yet another weight matrix is ​​used to blur unwanted noise in an image. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by these weight matrices of the same size are also of the same size. The extracted convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0129] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0130] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0131] Pooling layer:

[0132] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 3 Each layer 221-226 illustrated in Figure 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In the image processing process, the sole purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain a smaller image. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0133] Neural Network Layer 230:

[0134] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 3 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type, for example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0135] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 3 The propagation from 210 to 240 is forward propagation) and the reverse propagation (such as Figure 3 The propagation from 240 to 210 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will be updated to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0136] The structure of the convolutional neural network used in the embodiment of the present application can be as follows: Figure 4 As shown. Figure 4 In FIG, a convolutional neural network (CNN) 200 may include an input layer 110, a convolutional layer / pooling layer 120 (where the pooling layer is optional), and a neural network layer 130. Figure 3 compared to, Figure 4 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 120 are operated in parallel, and the features extracted from each layer are input to the full neural network layer 130 for processing.

[0137] It should be noted that Figure 3 and Figure 4 The convolutional neural network shown is only an example of two possible convolutional neural networks adopted in an embodiment of the present application. In specific applications, the convolutional neural network adopted in the embodiment of the present application can also exist in the form of other network models.

[0138] Figure 5 The hardware structure of a chip provided in the embodiment of the present application includes a neural network processor 50. The chip can be set as follows Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 3 and Figure 4 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 5 is implemented in the chip shown.

[0139] The neural network processor NPU 50 is mounted as a coprocessor on the host central processing unit (CPU), and the host CPU assigns tasks. The core of the NPU is the arithmetic circuit 503. The controller 504 controls the arithmetic circuit 503 to extract data from the memory (weight memory or input memory) and perform calculations.

[0140] In some implementations, the arithmetic circuit 503 includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0141] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 502 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 501 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 508.

[0142] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0143] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, for example, for use in a subsequent layer in a neural network.

[0144] The unified memory 506 is used to store input data and output data.

[0145] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0146] The bus interface unit (BIU) 510 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0147] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0148] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0149] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0150] in, Figure 3 and Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0151] Introduced above Figure 2 The execution device 110 in the embodiment of the present application can execute each step of the action detection method. Figure 3 and Figure 4 The CNN model shown and Figure 5 The chip shown can also be used to execute the various steps of the motion detection method of the embodiment of the present application. The motion detection method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0152] like Figure 6As shown, an embodiment of the present application provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 210, and a data storage system 250, wherein the local device 301 and the local device 302 are connected to the execution device 210 via a communication network.

[0153] Execution device 210 can be implemented by one or more servers. Optionally, execution device 210 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and other devices. Execution device 210 can be deployed at a single physical site or distributed across multiple physical sites. Execution device 210 can use data in data storage system 250 or invoke program code in data storage system 250 to implement the motion detection method of the embodiments of the present application.

[0154] Specifically, the execution device 210 may perform the following process:

[0155] Obtain a video segment to be processed, where the video segment to be processed includes K video frames, where K is a positive integer; determine a first boundary probability sequence based on the video segment to be processed, where the first boundary probability sequence is used to indicate a first probability that each of the K video frames is a boundary of an action segment, where the first probability that each video frame is a boundary of an action segment is determined based on a first video frame set corresponding to each video frame, where the first video frame set includes video frames that are i frames away from each video frame, where i is an integer and ranges from 1 to N, where N is a preset threshold; determine a second boundary probability sequence based on the video segment to be processed, where the second boundary probability sequence is used to indicate a second probability that each of the K video frames is a boundary of an action segment, where the second probability that each video frame is a boundary of an action segment is determined based on a second video frame set corresponding to each video frame, where the second video frame set includes video frames that are j frames away from each video frame, where j is an integer and ranges from 1 to M, where M is an integer greater than N and less than K; determine a target action segment set based on the first boundary probability sequence and the second boundary probability sequence.

[0156] Alternatively, the execution device 210 may also perform the following process:

[0157] Obtain a video clip to be processed; determine multiple candidate action clips of the video clip to be processed, each of the multiple candidate action clips corresponds to a candidate action; determine an action attention matrix based on the multiple candidate action clips, the action attention matrix including association information of the multiple candidate actions; perform confidence regression on the multiple candidate action clips in combination with the action attention matrix to obtain the confidence of the multiple candidate action clips; determine a target action clip set based on the confidence of the multiple candidate action clips.

[0158] By executing the above process, the device 210 can be built into an action detection device, which may include one or more neural networks. The action detection device can be used to detect one or more target video segments (e.g., action segments) in a video.

[0159] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0160] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0161] In one implementation, the local device 301 and the local device 302 obtain relevant parameters of the motion detection device from the execution device 210, deploy the motion detection device on the local device 301 and the local device 302, and use the motion detection device to detect one or more target video segments (for example, action segments) in the video.

[0162] In another implementation, the action detection device can be directly deployed on the execution device 210. The execution device 210 obtains the input video and input sentences from the local device 301 and the local device 302, and detects one or more target video segments (for example, action segments) in the video according to the action detection device.

[0163] The above-mentioned execution device 210 can also be a cloud device. In this case, the execution device 210 can be deployed in the cloud; or, the above-mentioned execution device 210 can also be a terminal device. In this case, the execution device 210 can be deployed on the user terminal side. The embodiment of the present application does not limit this.

[0164] In this application, a video is a video frame sequence composed of video frames, each of which can also be considered as a picture or image, that is, a video can also be considered as a picture sequence composed of pictures, or an image sequence composed of images.

[0165] A video can also be divided into one or more video clips, each of which is composed of one or more video frames. For example, the video can be divided into multiple video clips based on the content, or it can be divided into multiple video clips based on the time coordinates in the video. In the embodiments of the present application, the method for dividing a video into video clips is not limited.

[0166] The following combination Figures 7 to 14 The technical solutions of the embodiments of the present application are described in detail.

[0167] Figure 7 This is a schematic flow chart of the motion detection method of the present application. Figure 7 The action detection method 700 may include steps 710, 720, 730, and 740. In some examples, the action detection method may be performed by Figure 2 The execution device 110 in Figure 5 The chip shown and Figure 6 The execution is executed by the execution device 210 and other devices.

[0168] It should be noted that Figure 7 The method 700 can be used to perform temporal action detection on a video (i.e., a video to be processed), and can also be used to train a temporal action detection device or model (i.e., a device or model for performing temporal action detection). The embodiments of the present application are not limited to this.

[0169] S710: Obtain the video to be processed.

[0170] The video to be processed may include K video frames, where K is a positive integer.

[0171] The video frame may be considered as an image. For example, the video frame may be an optical flow image, or the video frame may be an RGB color model (RGB) image, or the video frame may include both an optical flow image and an RGB image.

[0172] Furthermore, a dual-stream network may be used to extract video features of the video to be processed to obtain a video feature sequence of the video to be processed.

[0173] For example, the video feature sequence of the video to be processed may include K video features, and the K video features may be used to indicate video features of K video frames in the video to be processed, respectively.

[0174] The specific method of extracting video features using the dual-stream network can be referred to the existing technology and will not be repeated here.

[0175] S720: Determine a first boundary probability sequence according to the video to be processed.

[0176] The first boundary probability sequence may be used to indicate a first probability that each of the K video frames is a boundary of an action segment.

[0177] Optionally, the boundary of the action clip may include the starting moment of the action clip, or the boundary of the action clip may also include the ending moment of the action clip, or the boundary of the action clip may also include the starting moment of the action clip and the ending moment of the action clip.

[0178] That is to say, the first probability that each of the K video frames is the boundary of the action segment may refer to: the probability that each video frame is the starting moment of the action segment, or the probability that each video frame is the ending moment of the action segment, or the probability that each video frame is the starting moment of the action segment, and the probability that each video frame is the ending moment of the action segment.

[0179] Optionally, the first probability that each video frame is the boundary of an action segment can be determined based on a first video frame set corresponding to each video frame, and the first video frame set can include video frames that are i frames away from each video frame, where i is an integer and the value of i ranges from 1 to N, and N is a preset threshold.

[0180] For example, when the video to be processed includes K video frames and the preset threshold N is 2, the first video frame set corresponding to the kth video frame in the video to be processed may include:

[0181] The k-2th video frame in the video to be processed, the k-1th video frame in the video to be processed, the k+1th video frame in the video to be processed, and the k+2th video frame in the video to be processed, where k ranges from 1 to K.

[0182] That is, the first probability that the kth video frame in the first boundary probability sequence is the boundary of the action segment can be determined based on the k-2th video frame, the k-1th video frame, the k+1th video frame and the k+2th video frame.

[0183] For another example, when the video to be processed includes K video frames and the preset threshold N is 1, the first video frame set corresponding to the kth video frame in the video to be processed may include:

[0184] The k-1th video frame in the video to be processed and the k+1th video frame in the video to be processed, where k ranges from 1 to K.

[0185] That is, the first probability that the kth video frame in the first boundary probability sequence is the boundary of the action segment may be determined based on the k-1th video frame and the k+1th video frame.

[0186] Optionally, the first boundary probability sequence may include multiple probability sequences.

[0187] For example, the first boundary probability sequence may include a probability sequence A and a probability sequence B, wherein the probability sequence A may be determined based on a video frame set a, and the video frame set a may include video frames within 2 frames from each of the video frames (i.e., the preset threshold N is 2); the probability sequence B may be determined based on a video frame set b, and the video frame set b may include video frames within 1 frame from each of the video frames (i.e., the preset threshold N is 1).

[0188] It should be noted that, in the above example, the number of probability sequences included in the first boundary probability sequence and the value of the preset threshold are only examples and not limitations. In the embodiment of the present application, the number of probability sequences included in the first boundary probability sequence and the value of the preset threshold are not limited.

[0189] S730: Determine a second boundary probability sequence according to the video to be processed.

[0190] The second boundary probability sequence may be used to indicate a second probability that each of the K video frames is a boundary of an action segment.

[0191] Similarly, the second probability that each of the K video frames is the boundary of the action segment may refer to: the probability that each video frame is the starting moment of the action segment, or the probability that each video frame is the ending moment of the action segment, or the probability that each video frame is the starting moment of the action segment and the probability that each video frame is the ending moment of the action segment.

[0192] Optionally, the second probability that each video frame is the boundary of an action segment can be determined based on a second video frame set corresponding to each video frame, and the second video frame set can include video frames that are j frames away from each video frame, where j is an integer and the value of j is 1 to M, and M is an integer greater than N and less than K.

[0193] Similar to the above-mentioned first boundary probability sequence, for example, when the video to be processed includes K video frames and the preset threshold N is 2, the value of M can be 5. At this time, the second probability that the kth video frame in the second boundary probability sequence is the boundary of the action segment can be determined based on the k-5th video frame, the k-4th video frame, the k-3th video frame, the k-2th video frame, the k-1th video frame, the k+1th video frame, the k+2th video frame, the k+3th video frame, the k+4th video frame and the k+5th video frame.

[0194] For another example, when the video to be processed includes K video frames and the preset threshold N is 1, the value of M can be 3. At this time, the second probability that the kth video frame in the second boundary probability sequence is the boundary of the action segment can be determined based on the k-3th video frame, the k-2th video frame, the k-1th video frame, the k+1th video frame, the k+2th video frame and the k+3th video frame.

[0195] Similarly, the second boundary probability sequence may include multiple probability sequences.

[0196] For example, the second boundary probability sequence may include a probability sequence A and a probability sequence B, wherein the probability sequence A may be determined based on a video frame set a, and the video frame set a may include video frames within 10 frames from each of the video frames (for example, the preset threshold N is 3, and M is 10), and the probability sequence B may be determined based on a video frame set b, and the video frame set b may include video frames within 5 frames from each of the video frames (for example, the preset threshold N is 2, and M is 5).

[0197] It should be noted that, in the above example, the number of probability sequences included in the second boundary probability sequence, the value of the preset threshold and the value of M are only examples and not limitations. In the embodiment of the present application, the number of probability sequences included in the second boundary probability sequence, the value of the preset threshold and the value of M are not limited.

[0198] S740: Determine a target action segment set according to the first boundary probability sequence and the second boundary probability sequence.

[0199] The target action segment set may include one or more target action segments.

[0200] For example, existing post-processing methods can be used to generate an action segment set based on the high-probability start moments and high-probability end moments in the first boundary probability sequence and the second boundary probability sequence; then, redundant action segments in the action segment set can be removed by using a non-maximum suppression method, and the action segments in the action segment set (after removing redundant action segments) can be sorted to obtain one or more high-quality action segments (i.e., the target action segment set).

[0201] The specific method for determining the target action segment set in the above S740 can refer to the bottom-up temporal action detection method in the prior art, which will not be described in detail here.

[0202] Optionally, the method 700 may further include step 722, step 724, and step 726, as follows:

[0203] S722: Determine a plurality of candidate action segments of the video to be processed.

[0204] Each of the multiple candidate action segments corresponds to a candidate action.

[0205] For example, multiple candidate action segments of the video to be processed may be determined based on a mechanism such as a sliding window or an anchor frame.

[0206] The specific method for determining the multiple candidate action segments in the above S722 can refer to the top-down temporal action detection method in the prior art, which will not be described in detail here.

[0207] S724: Determine an action attention matrix based on the multiple candidate action segments.

[0208] In which, the action attention matrix can be used to represent the temporal relationship of multiple candidate actions and / or the semantic relationship of multiple candidate actions.

[0209] Specifically, the temporal relationship may refer to the temporal relationship between the plurality of candidate actions, for example, the sequence between the plurality of candidate actions or the temporal association relationship between the plurality of candidate actions.

[0210] Taking a rowing video as an example, one candidate action clip in the video corresponds to the candidate action of "getting on the boat", and the other candidate action clip corresponds to the candidate action of "rowing with an oar". The temporal relationship between these two candidate actions can be understood as the temporal relationship between the two actions of "getting on the boat" and "rowing with an oar".

[0211] The semantic relationship may refer to the relationship between the semantic features of the multiple candidate actions, for example, the relationship between the specific actions referred to by the multiple candidate actions.

[0212] Taking a running video as an example, one candidate action clip in the video corresponds to the candidate action of "starting", and the other candidate action clip corresponds to the candidate action of "running midway". Then the semantic relationship between these two candidate actions can be understood as the relationship (semantic relationship) between the two specific actions of "starting" and "running midway".

[0213] For example, you can use Figure 9 The method described in S930 in method 900 is used to determine the action attention matrix.

[0214] S726: Use the action attention matrix to perform confidence regression on the multiple candidate action segments to obtain the confidence of the multiple candidate action segments.

[0215] The confidence level may be determined based on the degree of overlap (or completeness) between the action segment and the real action segment (i.e., the true value) in the video to be processed. For example, the degree of overlap (or completeness) may be determined based on the intersection over union (IoU) between the action segment and the real action segment.

[0216] Alternatively, the confidence level may be based on the probability of the start time (serving as the start time) and the probability of the end time (serving as the end time) of the action segment.

[0217] Optionally, the above S724 may further include:

[0218] Using a neural network model, an action attention matrix is ​​determined based on the multiple candidate action segments.

[0219] Similarly, the above S726 may further include:

[0220] Using the neural network model, confidence regression is performed on the multiple candidate action segments according to (i.e., using) the action attention matrix to obtain the confidence of the multiple candidate action segments.

[0221] It should be noted that the use of the action attention matrix in the above S726 can incorporate the associated information of multiple candidate actions (corresponding to the multiple candidate action clips) in the video to be processed (i.e., the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions) in the process of performing confidence regression on the multiple candidate action clips, thereby improving the accuracy of the confidence of the multiple candidate action clips, thereby improving the accuracy of temporal action detection.

[0222] Optionally, the above neural network model can be trained by the following method:

[0223] Using the target loss function, the loss value of the neural network model is calculated according to the confidence of the multiple candidate action segments; and the parameters of the neural network model are updated according to the loss value.

[0224] The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value.

[0225] For example, the objective loss function may reduce the weight of the confidence of the large-scale candidate action segments when calculating the loss value.

[0226] For another example, the target loss function may also increase the weight of the confidence of the small-scale candidate action segments when calculating the loss value.

[0227] For another example, the target loss function may also reduce the weight of the confidence of the candidate action segments at a large scale and increase the weight of the confidence of the candidate action segments at a small scale when calculating the loss value.

[0228] It should be noted that those skilled in the art will appreciate that the scale (of an action segment) can be used to measure the duration of an action. For example, the scale (of an action segment) can be the ratio of the duration of the action segment to the total duration of the video to be processed (i.e., the video containing the action segment). It can be seen that the scale (of an action segment) is a number between 0 and 1.

[0229] In the embodiments of the present application, the definitions of large-scale (action segments) and small-scale (action segments) can refer to the prior art.

[0230] Optionally, a large-scale (action segment) and a small-scale (action segment) can be distinguished based on a preset threshold. For example, an action segment with a scale greater than (or equal to) 0.7 can be considered a large-scale action segment, and an action segment with a scale less than (or equal to) 0.3 can be considered a small-scale action segment.

[0231] In an embodiment of the present application, the number of samples of candidate action segments of different scales is different (generally, the smaller the scale, the fewer the sample numbers of the candidate action segments). The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value. The target loss function is used to calculate the loss value of the neural network model, and the parameters of the neural network model are updated according to the loss value. This can improve the detection ability of the neural network model for small-scale actions, thereby improving the detection performance of temporal action detection.

[0232] Optionally, the target loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value. For example, the target loss function can be Figure 9 The scale-equalized loss function in method 900.

[0233] Since the confidence weights of the candidate action segments of different scales are made the same when calculating the loss value, the detection capability of the neural network model for small-scale actions can be further improved, thereby further improving the detection performance of temporal action detection.

[0234] Furthermore, when the method 700 further includes step 722, step 724, and step 726, the above S740 may include:

[0235] A target action segment set is determined from the multiple candidate action segments according to the confidences of the multiple candidate action segments, the first boundary probability sequence, and the second boundary probability sequence.

[0236] For example, the scores of the multiple candidate action segments can be determined based on the confidence of the multiple candidate action segments, the first boundary probability sequence and the second boundary probability sequence; then, the redundant action segments in the action segment set can be removed by the non-maximum suppression method, and the action segments in the action segment set (after removing the redundant action segments) can be sorted to obtain one or more high-quality action segments (i.e., the target action segment set).

[0237] The specific method for determining the target action segment set can be referred to in the prior art and will not be described in detail here.

[0238] In an embodiment of the present application, the first boundary probability sequence is determined based on the local information of each video frame (i.e., the first video frame set corresponding to each video frame), and the recall rate of temporal action detection using the first boundary probability sequence is higher. The second boundary probability sequence is determined based on the global information of each video frame (i.e., the second video frame set corresponding to each video frame), and the accuracy of temporal action detection using the second boundary probability sequence is higher. Therefore, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence can improve the accuracy of temporal action detection while ensuring the recall rate, that is, improve the detection capability of action boundaries, thereby improving the detection performance of temporal action detection.

[0239] Figure 8 This is another schematic flow chart of the motion detection method of the present application. Figure 8 The action detection method 800 may include steps 810, 820, 830, and 840. In some examples, the action detection method may be Figure 2 The execution device 110 in Figure 5 The chip shown and Figure 6 The execution is executed by the execution device 210 and other devices.

[0240] It should be noted that Figure 8 The method 800 can be used to perform temporal action detection on a video (i.e., a video to be processed), and can also be used to train a temporal action detection device or model (i.e., a device or model for performing temporal action detection). The embodiments of the present application are not limited to this.

[0241] S810: Obtain the video to be processed.

[0242] The video to be processed may include K video frames, where K is a positive integer.

[0243] The video frame may be considered as an image. For example, the video frame may be an optical flow image, or the video frame may be an RGB color model (RGB) image, or the video frame may include both an optical flow image and an RGB image.

[0244] Furthermore, a dual-stream network may be used to extract video features of the video to be processed to obtain a video feature sequence of the video to be processed.

[0245] For example, the video feature sequence of the video to be processed may include K video features, and the K video features may be used to indicate video features of K video frames in the video to be processed, respectively.

[0246] The specific method of extracting video features using the dual-stream network can be referred to the existing technology and will not be repeated here.

[0247] S820: Determine a plurality of candidate action segments of the video to be processed.

[0248] Each of the multiple candidate action segments corresponds to a candidate action.

[0249] For example, multiple candidate action segments of the video to be processed may be determined based on a mechanism such as a sliding window or an anchor frame.

[0250] The specific method for determining the multiple candidate action segments in the above S722 can refer to the top-down temporal action detection method in the prior art, which will not be described in detail here.

[0251] S830: Determine an action attention matrix based on the multiple candidate action segments.

[0252] In which, the action attention matrix can be used to represent the temporal relationship of multiple candidate actions and / or the semantic relationship of multiple candidate actions.

[0253] For a detailed description of temporal and semantic relationships, please refer to Figure 7 The embodiments of method 700 are not described in detail here.

[0254] For example, you can use Figure 9 The method described in S930 in method 900 is used to determine the action attention matrix.

[0255] S840: Use the action attention matrix to perform confidence regression on the multiple candidate action segments to obtain confidences of the multiple candidate action segments.

[0256] The confidence level may be determined based on the degree of overlap (or completeness) between the action segment and the real action segment (i.e., the true value) in the video to be processed. For example, the degree of overlap (or completeness) may be determined based on the intersection over union (IoU) between the action segment and the real action segment.

[0257] Alternatively, the confidence level may be based on the probability of the start time (serving as the start time) and the probability of the end time (serving as the end time) of the action segment.

[0258] Optionally, the above S830 may further include:

[0259] Using a neural network model, an action attention matrix is ​​determined based on the multiple candidate action segments.

[0260] Similarly, the above S840 may further include:

[0261] Using the neural network model, confidence regression is performed on the multiple candidate action segments according to the action attention matrix to obtain the confidence of the multiple candidate action segments.

[0262] It should be noted that, by using the action attention matrix in the above S840, the associated information of multiple candidate actions (corresponding to the multiple candidate action clips) in the video to be processed (i.e., the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions) can be incorporated into the process of confidence regression of the multiple candidate action clips, thereby improving the accuracy of the confidence of the multiple candidate action clips, thereby improving the accuracy of temporal action detection.

[0263] Optionally, the above neural network model can be trained by the following method:

[0264] S822, using the target loss function, calculate the loss value of the neural network model according to the confidence of the multiple candidate action segments; and update the parameters of the neural network model according to the loss value.

[0265] The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value.

[0266] For example, the objective loss function may reduce the weight of the confidence of the large-scale candidate action segments when calculating the loss value.

[0267] For another example, the target loss function may also increase the weight of the confidence of the small-scale candidate action segments when calculating the loss value.

[0268] For another example, the target loss function may also reduce the weight of the confidence of the candidate action segments at a large scale and increase the weight of the confidence of the candidate action segments at a small scale when calculating the loss value.

[0269] It should be noted that those skilled in the art will appreciate that the scale (of an action segment) can be used to measure the duration of an action. For example, the scale (of an action segment) can be the ratio of the duration of the action segment to the total duration of the video to be processed (i.e., the video containing the action segment). It can be seen that the scale (of an action segment) is a number between 0 and 1.

[0270] In the embodiments of the present application, the definitions of large-scale (action segments) and small-scale (action segments) can refer to the prior art.

[0271] Optionally, a large-scale (action segment) and a small-scale (action segment) can be distinguished based on a preset threshold. For example, an action segment with a scale greater than (or equal to) 0.7 can be considered a large-scale action segment, and an action segment with a scale less than (or equal to) 0.3 can be considered a small-scale action segment.

[0272] In an embodiment of the present application, the number of samples of candidate action segments of different scales is different (generally, the smaller the scale, the fewer the sample numbers of the candidate action segments). The target loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value. The target loss function is used to calculate the loss value of the neural network model, and the parameters of the neural network model are updated according to the loss value. This can improve the detection ability of the neural network model for small-scale actions, thereby improving the detection performance of temporal action detection.

[0273] Optionally, the target loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value. For example, the target loss function can be Figure 9 The scale-equalized loss function in method 900.

[0274] Since the confidence weights of the candidate action segments of different scales are made the same when calculating the loss value, the detection capability of the neural network model for small-scale actions can be further improved, thereby further improving the detection performance of temporal action detection.

[0275] S850: Determine a target action segment set according to the confidence levels of the multiple candidate action segments.

[0276] The target action segment set may include one or more target action segments.

[0277] For example, existing post-processing methods can be used to perform boundary regression, confidence evaluation, and confidence sorting on the multiple candidate action segments according to their confidence levels to obtain one or more high-quality action segments (i.e., a target action segment set).

[0278] The specific method for determining the multiple candidate action segments in the above S840 can refer to the top-down temporal action detection method in the prior art, which will not be described in detail here.

[0279] Optionally, the method 800 may further include steps 824 and 826, which are specifically as follows:

[0280] S824: Determine a first boundary probability sequence according to the video to be processed.

[0281] The first boundary probability sequence may be used to indicate a first probability that each of the K video frames is a boundary of an action segment.

[0282] Optionally, the boundary of the action clip may include the starting moment of the action clip, or the boundary of the action clip may also include the ending moment of the action clip, or the boundary of the action clip may also include the starting moment of the action clip and the ending moment of the action clip.

[0283] That is to say, the first probability that each of the K video frames is the boundary of the action segment may refer to: the probability that each video frame is the starting moment of the action segment, or the probability that each video frame is the ending moment of the action segment, or the probability that each video frame is the starting moment of the action segment, and the probability that each video frame is the ending moment of the action segment.

[0284] Optionally, the first probability that each video frame is the boundary of an action segment can be determined based on a first video frame set corresponding to each video frame, and the first video frame set can include video frames that are i frames away from each video frame, where i is an integer and the value of i ranges from 1 to N, and N is a preset threshold.

[0285] For example, when the video to be processed includes K video frames and the preset threshold N is 2, the first video frame set corresponding to the kth video frame in the video to be processed may include:

[0286] The k-2th video frame in the video to be processed, the k-1th video frame in the video to be processed, the k+1th video frame in the video to be processed, and the k+2th video frame in the video to be processed, where k ranges from 1 to K.

[0287] That is, the first probability that the kth video frame in the first boundary probability sequence is the boundary of the action segment can be determined based on the k-2th video frame, the k-1th video frame, the k+1th video frame and the k+2th video frame.

[0288] For another example, when the video to be processed includes K video frames and the preset threshold N is 1, the first video frame set corresponding to the kth video frame in the video to be processed may include:

[0289] The k-1th video frame in the video to be processed and the k+1th video frame in the video to be processed, where k ranges from 1 to K.

[0290] That is, the first probability that the kth video frame in the first boundary probability sequence is the boundary of the action segment may be determined based on the k-1th video frame and the k+1th video frame.

[0291] Optionally, the first boundary probability sequence may include multiple probability sequences.

[0292] For example, the first boundary probability sequence may include a probability sequence A and a probability sequence B, wherein the probability sequence A may be determined based on a video frame set a, and the video frame set a may include video frames within 2 frames from each of the video frames (i.e., the preset threshold N is 2); the probability sequence B may be determined based on a video frame set b, and the video frame set b may include video frames within 1 frame from each of the video frames (i.e., the preset threshold N is 1).

[0293] It should be noted that, in the above example, the number of probability sequences included in the first boundary probability sequence and the value of the preset threshold are only examples and not limitations. In the embodiment of the present application, the number of probability sequences included in the first boundary probability sequence and the value of the preset threshold are not limited.

[0294] S826: Determine a second boundary probability sequence according to the video to be processed.

[0295] The second boundary probability sequence is used to indicate a second probability that each of the K video frames is a boundary of an action segment.

[0296] Similarly, the second probability that each of the K video frames is the boundary of the action segment may refer to: the probability that each video frame is the starting moment of the action segment, or the probability that each video frame is the ending moment of the action segment, or the probability that each video frame is the starting moment of the action segment and the probability that each video frame is the ending moment of the action segment.

[0297] Optionally, the second probability that each video frame is a boundary of an action segment may be determined based on a second video frame set corresponding to each video frame, where the second video frame set may include video frames that are j frames away from each video frame, where j is an integer ranging from 1 to M, and M is an integer greater than N and less than K.

[0298] Similar to the above-mentioned first boundary probability sequence, for example, when the video to be processed includes K video frames and the preset threshold N is 2, the value of M can be 5. At this time, the second probability that the kth video frame in the second boundary probability sequence is the boundary of the action segment can be determined based on the k-5th video frame, the k-4th video frame, the k-3th video frame, the k-2th video frame, the k-1th video frame, the k+1th video frame, the k+2th video frame, the k+3th video frame, the k+4th video frame and the k+5th video frame.

[0299] For another example, when the video to be processed includes K video frames and the preset threshold N is 1, the value of M can be 3. At this time, the second probability that the kth video frame in the second boundary probability sequence is the boundary of the action segment can be determined based on the k-3th video frame, the k-2th video frame, the k-1th video frame, the k+1th video frame, the k+2th video frame and the k+3th video frame.

[0300] Similarly, the second boundary probability sequence may include multiple probability sequences.

[0301] For example, the second boundary probability sequence may include a probability sequence A and a probability sequence B, wherein the probability sequence A may be determined based on a video frame set a, and the video frame set a may include video frames within 10 frames from each of the video frames (for example, the preset threshold N is 3, and M is 10), and the probability sequence B may be determined based on a video frame set b, and the video frame set b may include video frames within 5 frames from each of the video frames (for example, the preset threshold N is 2, and M is 5).

[0302] It should be noted that, in the above example, the number of probability sequences included in the second boundary probability sequence, the value of the preset threshold and the value of M are only examples and not limitations. In the embodiment of the present application, the number of probability sequences included in the second boundary probability sequence, the value of the preset threshold and the value of M are not limited.

[0303] Furthermore, when the method 800 further includes step 824 and step 826, the above S850 may include:

[0304] A target action segment set is determined from the multiple candidate action segments according to the confidences of the multiple candidate action segments, the first boundary probability sequence, and the second boundary probability sequence.

[0305] For example, the scores of the multiple candidate action segments can be determined based on the confidence of the multiple candidate action segments, the first boundary probability sequence and the second boundary probability sequence; then, the redundant action segments in the action segment set can be removed by the non-maximum suppression method, and the action segments in the action segment set (after removing the redundant action segments) can be sorted to obtain one or more high-quality action segments (i.e., the target action segment set).

[0306] In an embodiment of the present application, the first boundary probability sequence is determined based on the local information of each video frame (i.e., the first video frame set corresponding to each video frame), and the recall rate of temporal action detection using the first boundary probability sequence is higher. The second boundary probability sequence is determined based on the global information of each video frame (i.e., the second video frame set corresponding to each video frame), and the accuracy of temporal action detection using the second boundary probability sequence is higher. Therefore, determining the target action segment set based on the first boundary probability sequence and the second boundary probability sequence can improve the accuracy of temporal action detection while ensuring the recall rate, that is, improve the detection capability of action boundaries, thereby improving the detection performance of temporal action detection.

[0307] The specific method for determining the target action segment set can be referred to in the prior art and will not be described in detail here.

[0308] In an embodiment of the present application, the action attention matrix determined based on the multiple candidate action segments includes the associated information of the multiple candidate actions (i.e., the temporal relationship of the multiple candidate actions and / or the semantic relationship of the multiple candidate actions). The action attention matrix is ​​used to perform confidence regression on the multiple candidate action segments. The associated information of the multiple candidate actions can be incorporated into the confidence regression process. Therefore, the accuracy of the confidence regression of the multiple candidate action segments can be improved, thereby improving the accuracy of temporal action detection.

[0309] Figure 9 This is another schematic flowchart of the motion detection method of the present application. Figure 9 The action detection method 900 may include steps 910, 920, 930, and 940. In some examples, the action detection method may be Figure 2 The execution device 110 in Figure 5 The chip shown and Figure 6 The execution is executed by the execution device 210 and other devices.

[0310] It should be noted that Figure 9The method 900 can be used to perform temporal action detection on a video (i.e., a video to be processed), and can also be used to train a temporal action detection device or model (i.e., a device or model for performing temporal action detection). The embodiments of the present application are not limited to this.

[0311] S910: Extract features from the video to be processed to obtain a video feature sequence of the video to be processed.

[0312] The video to be processed may include D video frames, where D is a positive integer.

[0313] The video frame may be considered as an image. For example, the video frame may be an optical flow image, or the video frame may be an RGB color model (RGB) image, or the video frame may include both an optical flow image and an RGB image.

[0314] Optionally, a dual-stream network may be used to extract video features of the video to be processed to obtain a video feature sequence of the video to be processed.

[0315] For example, the video feature sequence of the video to be processed may include D video features, and the D video features may be used to indicate video features of D video frames in the video to be processed, respectively.

[0316] The specific method of extracting video features using the dual-stream network can be referred to the existing technology and will not be repeated here.

[0317] S920: Use a multi-branch boundary detector to determine a boundary probability sequence of the video to be processed.

[0318] like Figure 11 As shown, the multi-branch boundary detector may include a local branch and a global branch.

[0319] Among them, the local branch can determine the local boundary probability sequence based on the local video frame set corresponding to each video frame, and the local video frame set may include video frames that are i frames away from each video frame, i is an integer, and the value of i ranges from 1 to N, and N is a preset threshold.

[0320] The specific method for determining the local boundary probability sequence of the local branch can be referred to Figure 7 The embodiment of S720 in method 700 or the bottom-up temporal motion detection method in the prior art will not be described in detail here.

[0321] The global branch can determine a global boundary probability sequence based on a global video frame set corresponding to each video frame, where the global video frame set may include video frames that are j frames away from each video frame, where j is an integer and the value of j ranges from 1 to M, where M is an integer greater than N and less than K.

[0322] For example, the global branch can be used Figure 11 The U-shaped network structure shown in the figure can fuse the information between the current frame and the frames with a larger interval before and after, and can incorporate contextual semantic information into the process of boundary detection, thereby improving the accuracy of boundary detection.

[0323] It can be seen that Figure 11 The receptive field of the global branch in is larger than that of the local branch.

[0324] The specific method for determining the global boundary probability sequence of the global branch can be referred to Figure 7 The embodiment of S730 in method 700 will not be described in detail here.

[0325] Optionally, a multi-branch boundary detector is used to perform boundary detection on the video feature sequence of the video to be processed, so as to obtain a local boundary probability sequence and a global boundary probability sequence.

[0326] It can be seen from the above embodiments that the boundary probability sequence may include a local boundary probability sequence and a global boundary probability sequence.

[0327] S930 : Determine the confidence of the candidate action segment in the video to be processed using a scale-balanced action confidence regressor.

[0328] Optionally, a confidence regression network with an action attention module can be used to perform confidence assessment on the multiple candidate action segments.

[0329] For example, a scale-balanced action confidence regressor may be used to construct an action attention matrix based on the video feature sequence, and the confidence of the candidate action segments in the video to be processed may be determined using the action attention matrix.

[0330] like Figure 12 Shown is a possible implementation of the scale-balanced action confidence regressor.

[0331] For example, Figure 12 As shown, a neural network model (i.e., a scale-balanced action confidence regressor) can be used to construct the action attention matrix, and the action attention matrix can be used to perform confidence regression on multiple candidate action segments to obtain the confidence of the multiple candidate action segments (Figure).

[0332] The specific method in the above S930 can be referred to Figure 7 The embodiments of S722, S724 and S726 in method 700 are not repeated here.

[0333] Optionally, a scale-balanced loss function may be used to supervise the process in S930 .

[0334] The scale-balanced loss function can balance the confidence weights of the candidate action segments of different scales when calculating the loss value.

[0335] For the loss function of scale balancing, please refer to Figure 7 The description of the target loss function in method 700 will not be repeated here.

[0336] For example, the scale-balanced loss function can be used to calculate the loss value of the scale-balanced action confidence regressor based on the confidence of the candidate action segments in the video to be processed, and the parameters of the scale-balanced action confidence regressor can be updated based on the loss value.

[0337] For example, the loss function of scale equalization can be expressed as wbl , which can be calculated by the following formula:

[0338]

[0339] in, represents the IoU between the true value c and the action segment starting at time i and ending at time j, n c is the number of positive samples of the true value c, that is, b i,j Indicates whether the action is a positive sample, ε is the threshold for controlling positive samples, p i,j is the confidence of the method in predicting the action segment with starting time i and ending time j, and N is the total number of samples.

[0340] Specifically, when the IoU between the action clip and the true value c is greater than ε, the action clip is considered a positive sample, otherwise, the action clip is considered a negative sample; N is the sum of the number of positive samples and the number of negative samples; The α in α can adjust the weight between positive and negative samples. For example, when α is 1, if the video contains only one action segment, the scale-balanced loss function l wbl It degenerates into a general logistic regression (logistic loss) function.

[0341] In the scale-balanced action confidence regressor, after predicting the confidence of each candidate action segment, we can use l wblSupervision is performed so that the scale-balanced action confidence regressor can obtain a confidence that is balanced between large-scale actions and small-scale actions.

[0342] It should be noted that the execution order of S920 and S930 is not limited in the embodiment of the present application.

[0343] For example, S920 may be executed first, and then S930; or S930 may be executed first, and then S920; or Figure 10 As shown, S920 and S930 may also be executed in parallel.

[0344] S940 , performing post-processing based on the boundary probability sequence of the video to be processed and the confidence of the candidate action segments in the video to be processed to obtain a target action segment set.

[0345] For example, the scores of the multiple candidate action clips can be determined based on the boundary probability sequence and the confidence of the multiple candidate action clips in the video to be processed; then, the redundant action clips in the action clip set can be removed by the non-maximum suppression method, and the action clips in the action clip set (after removing the redundant action clips) can be sorted to obtain one or more high-quality action clips (i.e., the target action clip set).

[0346] The specific method for determining the target action segment set can be referred to in the prior art and will not be described in detail here.

[0347] Table 1 shows the performance differences of different methods for temporal action detection on the test set ActivityNet1.3 and the test set THUMOS-14.

[0348] Table 1

[0349]

[0350] As can be seen from Table 1, when testing based on the test set ActivityNet1.3, the AUC (the area under average recall vs. average number of proposals curve) after using BSN (boundarysensitive network) to perform the action nomination task is 66.17, the AUC after using BMN (boundarymatching network) to perform the action nomination task is 67.10, and the AUC after using the method in the embodiment of the present application to perform the action nomination task is 68.15; the Ave mAP (the average of all mean average precision on different tIoU) after using BSN to perform the action detection task is 30.03, the Ave mAP after using BMN to perform the action detection task is 33.85, and the Ave mAP after using the method in the embodiment of the present application to perform the action detection task is 33.89.

[0351] When tested based on the test set THUMOS-14, the AR@100 (the average recall of 100 proposals) after using TURN (temporal unit regression network for temporal action proposals) to perform the action nomination task is 31.89, the AR@100 after using BSN to perform the action nomination task is 46.06, the AR@100 after using BMN to perform the action nomination task is 47.72, and the AR@100 after using the method in the embodiment of the present application to perform the action nomination task is 50.51; the mAP@0.5 (the mean average precision of tIoU=0.5) after using TURN to perform the action detection task is 25.6, the mAP@0.5 after using BSN to perform the action detection task is 36.9, the mAP@0.5 after using BMN to perform the action detection task is 38.8, and the mAP@0.5 after using the method in the embodiment of the present application to perform the action detection task is 42.6.

[0352] Figure 13 This is another schematic flowchart of the motion detection method of the present application. Figure 13 The method 1300 includes step 1310, step 1320, step 1330 and step 1340.

[0353] The multi-branch boundary detection in step 1320 can be performed by the multi-branch boundary detector in the above method 900. The multi-branch boundary detector can include a global branch and a local branch. The specific implementation of step 1320 can refer to the above method. Figure 9 The description of S920 in method 900, and the implementation of steps 1310, 1330, and 1340 can refer to the existing technology and will not be repeated here.

[0354] It should be noted that Figure 13 The method 1300 does not limit the execution order of S1320 and S1330.

[0355] For example, S1320 may be executed first, and then S1330; or S1330 may be executed first, and then S1320; or Figure 13 As shown, S1320 and S1330 may also be executed in parallel.

[0356] Table 2 shows the performance differences of different methods for temporal action detection on the test set ActivityNet1.3.

[0357] Table 2

[0358]

[0359] As can be seen from Table 2, when testing based on the test set ActivityNet1.3, the AUC after performing the action nomination task using the traditional method in the existing technology is 66.10, and using Figure 13 Method 1300 in

[13] achieves an AUC of 67.35 after performing the action nomination task.

[0360] Figure 14 This is another schematic flowchart of the motion detection method of the present application. Figure 14 The method 1400 includes step 1410, step 1420, step 1430 and step 1440.

[0361] The scale-balanced action confidence regression in step 1430 can be performed by the scale-balanced action confidence regressor in the above method 900. The specific implementation of step 1430 can refer to the above method. Figure 9 The description of S930 in method 900, and the implementation of steps 1410, 1420, and 1440 can refer to the existing technology and will not be repeated here.

[0362] It should be noted that Figure 14 The method 1400 does not limit the execution order of S1420 and S1430.

[0363] For example, S1420 may be executed first, and then S1430; or S1430 may be executed first, and then S1420; or Figure 14 As shown, S1420 and S1430 may also be executed in parallel.

[0364] Table 3 shows the performance differences of different methods for temporal action detection on the test set ActivityNet1.3.

[0365] Table 3

[0366]

[0367] As can be seen from Table 3, when testing based on the test set ActivityNet1.3, the AUC after performing the action nomination task using the traditional method in the existing technology is 67.10, and using Figure 14 Method 1400 achieves an AUC of 67.90 after performing the action nomination task.

[0368] Figure 15 Schematic diagram of the hardware structure of the motion detection device according to an embodiment of the present application. Figure 15 The motion detection device 4000 shown includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. The memory 4001, the processor 4002, and the communication interface 4003 are connected to each other via the bus 4004.

[0369] Memory 4001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 4001 can store programs. When the program stored in memory 4001 is executed by processor 4002, processor 4002 and communication interface 4003 are used to perform the various steps of the motion detection device of the embodiment of the present application.

[0370] The processor 4002 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the motion detection device of the embodiment of the present application, or to execute the motion detection method of the method embodiment of the present application.

[0371] The processor 4002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the motion detection method of the embodiment of the present application may be completed by hardware integrated logic circuits in the processor 4002 or software instructions.

[0372] The processor 4002 may also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and in combination with its hardware completes the functions required to be executed by the units included in the motion detection device of the embodiment of the present application, or executes the motion detection method of the method embodiment of the present application.

[0373] The communication interface 4003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 4000 and other devices or a communication network. For example, the video to be processed or training data can be obtained through the communication interface 4003.

[0374] The bus 4004 may include a path for transmitting information between various components of the device 4000 (eg, the memory 4001 , the processor 4002 , and the communication interface 4003 ).

[0375] Figure 16 Schematic diagram of the hardware structure of the model training device 5000 of the embodiment of the present application. Similar to the above-mentioned device 4000, Figure 16 The model training device 5000 shown includes a memory 5001, a processor 5002, a communication interface 5003 and a bus 5004. The memory 5001, the processor 5002 and the communication interface 5003 are connected to each other via the bus 5004.

[0376] The memory 5001 can store programs. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 is used to execute the various steps of the training method for training the action detection device of the embodiment of the present application.

[0377] The processor 5002 can adopt a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the training method of the action detection device of the embodiment of the present application.

[0378] The processor 5002 may also be an integrated circuit chip with signal processing capabilities. During the training process, each step of the training method of the motion detection device of the embodiment of the present application may be completed by hardware integrated logic circuits in the processor 5002 or software instructions.

[0379] It should be understood that through Figure 16 The model training device 5000 shown in the figure trains the action detection device, and the trained action detection device can be used to perform the action detection method of the embodiment of the present application. Specifically, the action detection device (for example, the action detection device may include one or more neural networks) can be trained by the device 5000 to obtain Figure 15 The method of the motion detection device is shown.

[0380] Specifically, Figure 16 The device shown can obtain training data and the motion detection device to be trained from the outside world through the communication interface 5003, and then the processor trains the motion detection device to be trained according to the training data.

[0381] Optionally, the above-mentioned training data may include training videos and the true value of the training data, wherein the training video includes one or more action segments, and the true value of the training data may include the start time, end time and action category of each action segment in the training video that has been manually pre-labeled.

[0382] It should be noted that although the above-mentioned apparatus 4000 and apparatus 5000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the apparatus 4000 and apparatus 5000 may also only include the devices necessary to implement the embodiments of the present application, and do not necessarily include Figure 15 and Figure 16 All devices shown in .

[0383] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0384] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0385] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0386] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0387] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0388] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0389] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0390] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0391] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0392] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0393] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0394] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0395] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A motion detection method, characterized in that: include: Obtain a video to be processed, where the video to be processed includes K video frames, where K is a positive integer; Determining a first boundary probability sequence based on the video to be processed, the first boundary probability sequence being used to indicate a first probability that each of the K video frames is a boundary of an action segment, the first probability that each video frame is a boundary of an action segment being determined based on a first video frame set corresponding to each video frame, the first video frame set including video frames that are i frames away from each video frame, where i is an integer ranging from 1 to N, and N is a preset threshold; Determining a second boundary probability sequence based on the video to be processed, the second boundary probability sequence being used to indicate a second probability that each of the K video frames is a boundary of an action segment, the second probability that each video frame is a boundary of an action segment being determined based on a second video frame set corresponding to each video frame, the second video frame set including video frames that are j frames away from each video frame, where j is an integer ranging from 1 to M, and M is an integer greater than N and less than K; A target action segment set is determined according to the first boundary probability sequence and the second boundary probability sequence, where the target action segment set includes one or more target action segments.

2. The method according to claim 1, characterized in that The method further comprises: Determine a plurality of candidate action segments of the video to be processed, each of the plurality of candidate action segments corresponding to a candidate action; Determining an action attention matrix based on the multiple candidate action segments, where the action attention matrix is ​​used to represent a temporal relationship between the multiple candidate actions and / or a semantic relationship between the multiple candidate actions; Performing confidence regression on the multiple candidate action segments using the action attention matrix to obtain confidences of the multiple candidate action segments; The step of determining the target action segment set according to the first boundary probability sequence and the second boundary probability sequence includes: A target action segment set is determined from the multiple candidate action segments according to the confidences of the multiple candidate action segments, the first boundary probability sequence, and the second boundary probability sequence.

3. The method according to claim 2, characterized in that Determining an action attention matrix according to the plurality of candidate action segments includes: Determine an action attention matrix based on the plurality of candidate action segments using a neural network model; The neural network model is trained by the following method: Calculating a loss value of the neural network model based on the confidences of the multiple candidate action segments using a target loss function, wherein the target loss function is capable of balancing the weights of the confidences of the candidate action segments of different scales when calculating the loss value; According to the loss value, the parameters of the neural network model are updated.

4. The method according to claim 3, characterized in that The objective loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value.

5. A motion detection device, characterized in that: include: An acquisition module is used to acquire a video to be processed, where the video to be processed includes K video frames, where K is a positive integer; a first execution module, configured to determine a first boundary probability sequence based on the video to be processed, the first boundary probability sequence being used to indicate a first probability that each of the K video frames is a boundary of an action segment, the first probability that each video frame is a boundary of an action segment being determined based on a first video frame set corresponding to each video frame, the first video frame set including video frames that are i frames away from each video frame, where i is an integer ranging from 1 to N, and N is a preset threshold; The first execution module is configured to determine a second boundary probability sequence based on the video to be processed, the second boundary probability sequence being used to indicate a second probability that each of the K video frames is a boundary of an action segment, the second probability that each video frame is a boundary of an action segment being determined based on a second video frame set corresponding to each video frame, the second video frame set including video frames that are j frames away from each video frame, where j is an integer ranging from 1 to M, and M is an integer greater than N and less than K; The determination module is configured to determine a target action segment set according to the first boundary probability sequence and the second boundary probability sequence, where the target action segment set includes one or more target action segments.

6. The device according to claim 5, characterized in that The apparatus further includes a second execution module, the second execution module being configured to: Determine a plurality of candidate action segments of the video to be processed, each of the plurality of candidate action segments corresponding to a candidate action; Determining an action attention matrix based on the multiple candidate action segments, where the action attention matrix is ​​used to represent a temporal relationship between the multiple candidate actions and / or a semantic relationship between the multiple candidate actions; Performing confidence regression on the multiple candidate action segments using the action attention matrix to obtain confidences of the multiple candidate action segments; The determining module is specifically configured to: A target action segment set is determined from the multiple candidate action segments according to the confidences of the multiple candidate action segments, the first boundary probability sequence, and the second boundary probability sequence.

7. The device according to claim 6, characterized in that The second execution module is specifically configured to: Determine an action attention matrix based on the plurality of candidate action segments using a neural network model; The neural network model is trained by the following method: Calculating a loss value of the neural network model based on the confidences of the multiple candidate action segments using a target loss function, wherein the target loss function is capable of balancing the weights of the confidences of the candidate action segments of different scales when calculating the loss value; According to the loss value, the parameters of the neural network model are updated.

8. The device according to claim 7, characterized in that The objective loss function can make the confidence weights of the candidate action segments of different scales the same when calculating the loss value.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes for execution by a device, wherein the program codes include codes for executing the method according to any one of claims 1 to 4.

10. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 4.

Images