Sequential action nomination method and device

Abstract

The present application discloses a method and device for nominating sequential actions. The method includes: obtaining a video to be processed, dividing the video to be processed into multiple first video segments; determining the atomic action probability corresponding to each first video segment based on a pre-trained atomic action evaluation model to obtain an atomic action probability set; determining an action boundary set from the marked time points corresponding to multiple first video segments based on the atomic action probability set, and determining multiple candidate action nominations based on the action boundary set; extracting candidate nomination features corresponding to each candidate action nomination, and determining the confidence of each candidate action nomination based on the candidate nomination features; performing non-maximum suppression processing on multiple candidate action nominations based on the confidence of multiple candidate action nominations to obtain a target action nomination. The present application solves the technical problem in the related art that the process of obtaining action nominations when performing sequential action detection on videos is relatively complicated and the recall rate is not high.

Description

Technical Field

[0001] The present application relates to the technical field of sequential action detection, and in particular to a sequential action nomination method and device. Background Art

[0002] Temporal action detection has two main tasks: one is to automatically identify the action category in the video; the other is to locate the duration of the action in the video. Usually, the temporal action detection algorithm can be used to extract the highlight moments in sports videos, or to process some instant application tasks of video recommendation, or in the field of video intelligent monitoring.

[0003] Early methods of temporal action detection were inspired by the development paradigm of two-dimensional image detection. The process of this typical paradigm can be divided into two stages: generating action candidate nominations and classifying action candidate nominations. Since candidate nominations need to be obtained first in the whole process, how to obtain high-quality action candidate nominations has become the difficulty of temporal action detection.

[0004] A large number of experiments in articles have shown that high-quality nominations should meet the following conditions: a high overlap in nomination time and a high recall rate; while complying with the above conditions, the number of nominations is small in order to reduce computational overhead and increase generation speed. However, although the relevant atomic evaluation model can guarantee good performance in the nomination task, the relationship between atomic actions is not considered during the entire training phase, resulting in a more complicated action nomination process based on the atomic evaluation model and an inability to achieve a high recall rate.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a method and device for nominating sequential actions, so as to at least solve the technical problems in the related art that the process of obtaining action nominations when performing sequential action detection on a video is relatively complicated and the recall rate is not high.

[0007] According to one aspect of an embodiment of the present application, a method for nominating a temporal action is provided, comprising: obtaining a video to be processed, dividing the video to be processed into a plurality of first video segments, wherein each first video segment has a corresponding marked time point; determining the atomic action probability corresponding to each first video segment based on a pre-trained atomic action evaluation model, and obtaining an atomic action probability set, wherein the atomic action evaluation model is a Transformer model, and the atomic actions include: action start, action in progress, and action end; determining an action boundary set from the marked time points corresponding to the plurality of first video segments based on the atomic action probability set, and determining a plurality of candidate action nominations based on the action boundary set; extracting a candidate nomination feature corresponding to each candidate action nomination, and determining a confidence of each candidate action nomination based on the candidate nomination feature; performing non-maximum suppression processing on the plurality of candidate action nominations based on the confidences of the plurality of candidate action nominations, and obtaining a target action nomination.

[0008] Optionally, dividing the video to be processed into a plurality of first video segments includes: dividing the video to be processed into a plurality of first video segments by an equal-interval sampling method, wherein any two adjacent first video segments have overlapping video frame images.

[0009] Optionally, the atomic action probability corresponding to each first video clip is determined based on a pre-trained atomic action evaluation model to obtain an atomic action probability set, including: inputting multiple first video clips into the atomic action evaluation model in sequence to obtain the atomic action probability corresponding to each first video clip, the atomic action probability including: action start probability, action ongoing probability and action end probability; taking the atomic action probabilities corresponding to multiple first video clips as an atomic action probability set, the atomic action probability set including: action start probability set, action ongoing probability set and action end probability set.

[0010] Optionally, the training process of the atomic action evaluation model includes: obtaining training sample videos and annotation information, the annotation information being used to annotate action instances in the training sample videos; dividing the training sample videos into multiple second video segments, and predicting the atomic action probability corresponding to each second video segment based on the atomic action evaluation model to be trained; constructing a target loss function based on the atomic action probability and annotation information corresponding to each second video segment; and based on the target loss function, adjusting the model parameters of the atomic action evaluation model to be trained by back-propagation gradient descent.

[0011] Optionally, a target loss function is constructed based on the atomic action probability and annotation information corresponding to each second video clip, including: constructing a first loss function based on the action start probability and annotation information corresponding to each second video clip; constructing a second loss function based on the action ongoing probability and annotation information corresponding to each second video clip; constructing a third loss function based on the action end probability and annotation information corresponding to each second video clip; determining the sequence probability based on the action start probability, action ongoing probability and action end probability corresponding to every three adjacent second video clips, and constructing a fourth loss function based on all sequence probabilities, wherein the sequence probability is used to indicate whether every three adjacent second video clips satisfy the atomic action sequence; constructing a target loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function.

[0012] Optionally, the action boundary set includes an action start boundary set and an action end boundary set, and the action boundary set is determined from the marked time points corresponding to multiple first video clips based on the atomic action probability set, and multiple candidate action nominations are determined based on the action boundary set, including: determining the marked time point of the first video clip corresponding to each action start probability greater than a first preset threshold in the action start probability set as the action start boundary, and obtaining the action start boundary set; determining the marked time point of the first video clip corresponding to each action end probability greater than the first preset threshold in the action end probability set as the action end boundary, and obtaining the action end boundary set; taking one action start boundary and one action end boundary from the action start boundary set and the action end boundary set each time, if the time lengths corresponding to the action start boundary and the action end boundary meet the time length range of the action instance in the video to be processed, determining a candidate action nomination based on the action start boundary and the action end boundary; traversing the action start boundary set and the action end boundary set to obtain multiple candidate action nominations.

[0013] Optionally, extracting candidate nomination features corresponding to each candidate action nomination includes: for each candidate action nomination, dividing the candidate action nomination into an action start area, an action in progress area, and an action end area; collecting a first number of action start features from the action start area, collecting a second number of action in progress features from the action in progress area, and collecting a third number of action end features from the action end area; and splicing the action start features, the action in progress features, and the action end features to obtain the candidate nomination features corresponding to the candidate action nomination.

[0014] Optionally, the confidence of each candidate action nomination is determined based on the candidate nomination features, including: inputting the candidate nomination features corresponding to each candidate action nomination into a two-layer perceptron to obtain the confidence of each candidate action nomination, wherein the confidence is used to indicate the overlap between the candidate action nomination and the action instance in the video to be processed.

[0015] Optionally, non-maximum suppression processing is performed on multiple candidate action nominations based on their confidences to obtain a target action nomination, including: determining a first candidate action nomination with the highest confidence and the remaining second candidate action nominations among the multiple candidate action nominations, determining the first candidate action nomination as the target action nomination, and comparing the overlap between the second candidate action nomination and the first candidate action nomination; if the overlap is less than a second preset threshold, the confidence of the second candidate action nomination remains unchanged; if the overlap is not less than the second preset threshold, fractional Gaussian decay is performed on the confidence of the second candidate action nomination; removing the second candidate action nomination and the first candidate action nomination whose confidence after processing is lower than a third preset threshold from the multiple candidate action nominations, and repeating the above processing flow for the remaining candidate action nominations until the last candidate action nomination remains, and determining that the last candidate action nomination is also the target action nomination.

[0016] According to another aspect of an embodiment of the present application, a temporal action nomination device is also provided, including: an acquisition module, used to acquire a video to be processed, and divide the video to be processed into multiple first video segments, wherein each first video segment has a corresponding marked time point; a probability evaluation module, used to determine the atomic action probability corresponding to each first video segment based on a pre-trained atomic action evaluation model, and obtain an atomic action probability set, wherein the atomic action evaluation model is a Transformer model, and the atomic actions include: action start, action in progress, and action end; a boundary determination module, used to determine an action boundary set from the marked time points corresponding to multiple first video segments based on the atomic action probability set, and determine multiple candidate action nominations based on the action boundary set; a feature extraction module, used to extract candidate nomination features corresponding to each candidate action nomination, and determine the confidence of each candidate action nomination based on the candidate nomination features; a nomination determination module, used to perform non-maximum suppression processing on multiple candidate action nominations based on the confidences of the multiple candidate action nominations, and obtain a target action nomination.

[0017] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned sequential action nomination method.

[0018] According to another aspect of an embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein the above-mentioned sequential action nomination method is executed when the program is running.

[0019] In an embodiment of the present application, firstly, by acquiring a video to be processed, the video to be processed is divided into a plurality of first video segments, and based on a pre-trained atomic action evaluation model, the atomic action probability corresponding to each first video segment is determined to obtain an atomic action probability set; based on the atomic action probability set, an action boundary set is determined from the marked time points corresponding to the plurality of first video segments; based on the action boundary set, a plurality of candidate action nominations are determined, and then the candidate nomination features corresponding to each candidate action nomination are extracted, and the confidence of each candidate action nomination is determined based on the candidate nomination features, and finally, based on the confidence of the plurality of candidate action nominations, a non-maximum suppression process is performed on the plurality of candidate action nominations to obtain a target action nomination, wherein the atomic action evaluation model is a Transformer model. The technical effect of better positioning of sequential actions in sequential action detection is achieved, thereby solving the technical problem in the related art that the process of obtaining action nominations when performing sequential action detection on videos is relatively complicated and the recall rate is not high. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a flowchart of a method for nominating sequential actions according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of an optional Transformer atomic action evaluation model;

[0023] Figure 3 is a schematic diagram of the back-propagation process in the optional fusion rule training phase;

[0024] Figure 4 It is a structural diagram of a sequential action nomination device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Example 1

[0028] According to an embodiment of the present application, a method for nominating sequential actions is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of an optional method for nominating sequential actions according to an embodiment of the present application, such as Figure 1 As shown, the method at least includes steps S101-S105, wherein:

[0030] Step S101: obtaining a video to be processed, and dividing the video to be processed into a plurality of first video segments, wherein each first video segment has a corresponding marked time point.

[0031] In order to better analyze the action nominations in the video, the original video can be segmented after being input, that is, the overlapping parts of the video to be processed can be divided into multiple small video clips. In the embodiment of the present application, the video to be processed can be divided into multiple first video clips by an equally spaced sampling method, wherein any two adjacent first video clips have overlapping video frame images.

[0032] For example, after inputting the original video, the input video can be defined as Among them, l v Indicates the number of frames in the video, x n is the nth frame in X. Define the action instance in X as N g represents the number of action instances in X, t s,n ,t e,n Represents an action instance The start time and end time of the

[0033] By decomposing the input video using a sliding window overlapping equal interval sampling method, multiple first video segments can be obtained. Indicates that, among them, l s represents the number of video clips, l s The setting can be set according to experience, such as l s Can be set to 100. The nth video clip can be used Indicates that refers to the tth n RGB image of the frame.

[0034] Step S102, determining the atomic action probability corresponding to each first video clip based on a pre-trained atomic action evaluation model to obtain an atomic action probability set, wherein the atomic action evaluation model is a Transformer model, and the atomic actions include: action start, action in progress, and action end.

[0035] In an optional embodiment of the present application, by sequentially inputting multiple first video clips into an atomic action evaluation model, the atomic action probability corresponding to each first video clip is obtained, and the atomic action probability includes: action start probability, action ongoing probability and action end probability. The atomic action probabilities corresponding to the multiple first video clips are taken as an atomic action probability set, and the atomic action probability set includes: action start probability set S, action ongoing probability set A and action end probability set E.

[0036] The atomic action evaluation model can choose the Transformer model, which consists of an embedding layer, a transformerEncoder (encoder), an MLP Head (classifier), etc. Figure 2 A schematic diagram of the structure of a Transformer model is shown.

[0037] In the embodiment of the present application, it is necessary to determine the atomic action probabilities corresponding to the plurality of first video clips according to the Transformer model, so it is necessary to train the Transformer model, and the training process can be divided into the following steps:

[0038] Step S1, obtaining a training sample video and annotation information, where the annotation information is used to annotate action instances in the training sample video;

[0039] For example, the obtained training sample video is defined as l K represents the number of frames in the sample video, x n is the nth frame in X, and the action instance in X is defined as N Trepresents the number of action instances in X, and ψ T As annotation information.

[0040] Step S2, dividing the training sample video into a plurality of second video segments, and predicting the atomic action probability corresponding to each second video segment based on the atomic action evaluation model to be trained;

[0041] Specifically, the training sample video in step S1 is divided into a plurality of second video segments, which can be defined as a sequence l W Represents the number of video clips. After inputting the obtained multiple second video clips into the Transformer model in sequence and performing probability output, three atomic action probabilities can be obtained, which are: (action start probability), (action end probability) and (Probability of action in progress).

[0042] The atomic actions corresponding to the second video clips all include the start of the action, the ongoing action, and the end of the action. In order to better extract the logical relationship between the atomic actions, when multiple second video clips pass through the Transformer model to be trained, the atomic action probability matrices corresponding to the multiple second video clips are obtained, which are recorded as: the action start probability matrix, the action ongoing probability matrix, and the action end probability score matrix. The probabilities of the corresponding positions of these three probability score matrices are compared, and the maximum value is taken as the maximum probability atomic action of the second video clip. Among them, if the highest probability is the action start probability, the corresponding value of the probability score matrix at this location is recorded as 0; when the maximum probability value is the moment when the atomic action is ongoing, the corresponding value of the probability score matrix at this location is recorded as 1; when the maximum probability value is the moment when the atomic action ends, the corresponding value of the probability score matrix at this location is recorded as 2. According to the above-obtained action start probability matrix, the values ​​corresponding to the action ongoing probability matrix and the action end probability score matrix will be used to construct the atomic action relationship restriction function, and the number of atomic action errors that do not conform to the logical order will be counted.

[0043] Step S3, constructing a target loss function based on the atomic action probability and annotation information corresponding to each second video clip;

[0044] Optionally, the target loss function can be constructed in the following manner: construct a first loss function based on the action start probability and annotation information corresponding to each second video clip; construct a second loss function based on the action ongoing probability and annotation information corresponding to each second video clip; construct a third loss function based on the action end probability and annotation information corresponding to each second video clip; determine the sequence probability based on the action start probability, action ongoing probability and action end probability corresponding to every three adjacent second video clips, and construct a fourth loss function based on all sequence probabilities, wherein the sequence probability is used to indicate whether every three adjacent second video clips satisfy the atomic action sequence; construct a target loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function.

[0045] Specifically, the loss function of the Transformer atomic action evaluation model in the embodiment of the present application is composed of three classification losses and one penalty loss weighted, wherein the three classification losses are respectively the action ongoing category loss function (i.e. the first loss function), action end category loss function (i.e. the second loss function), action start category loss function (ie the third loss function), penalty loss function εLoss limit (ie the fourth loss function), the specific formula is as follows:

[0046]

[0047] In the above formula, η represents the weight coefficient, and ε is defined as the weight of the action penalty function. In practical applications, it can be set based on experience. In this embodiment, η can be set to 2.

[0048] The first loss function, the second loss function, and the third loss function all use the cross entropy loss function, which can be expressed by the following formula:

[0049]

[0050] Among them, p i Indicates the true label value.

[0051] The fourth loss function can be defined as the action penalty function, which can be expressed as follows:

[0052]

[0053] In order to ensure that the atomic actions in the video have a certain sequential logic, the atomic action relationship restriction function is used to count the number of errors in atomic actions that do not conform to the logical order. This can be achieved through the following formula:

[0054]

[0055] The purpose of the above formula is to determine p i-1 ,p i ,p i+1 In the continuous time window probability, whether it conforms to the order of action start, action in progress, and action end, when it is judged as a continuous repeated mark or conforms to the order of atomic action steps, v(p i-1 ,p i ,p i+1 ) is marked as 0, then it is eliminated; otherwise v(p i-1 ,p i ,p i+1 ) is set to 1.

[0056] Step S4: Based on the target loss function, adjust the model parameters of the atomic action evaluation model to be trained by back-propagation gradient descent.

[0057] Figure 3 This is a schematic diagram of the back propagation process in the training phase. The first video segment S is calculated during model training. n and its true annotation ψ T , and then based on the target loss function obtained in step S3, back-propagation gradient descent is performed to update the parameters of the model to be trained.

[0058] Step S103 : determining an action boundary set from the marked time points corresponding to the plurality of first video clips based on the atomic action probability set, and determining a plurality of candidate action nominations based on the action boundary set.

[0059] In an optional embodiment of the present application, the marked time point of the first video segment corresponding to each action start probability greater than a first preset threshold in the action start probability set is determined as the action start boundary to obtain an action start boundary set, and then the marked time point of the first video segment corresponding to each action end probability greater than the first preset threshold in the action end probability set is determined as the action end boundary to obtain an action end boundary set, and each time an action start boundary and an action end boundary are taken from the action start boundary set and the action end boundary set, if the time lengths corresponding to the action start boundary and the action end boundary meet the time length range of the action instance in the video to be processed, a candidate action nomination is determined based on the action start boundary and the action end boundary, and finally the action start boundary set and the action end boundary set are traversed to obtain multiple candidate action nominations.

[0060] Specifically, the action start probability set S and the action end probability set E obtained in step S102 are firstly thresholded from the action start probability set S, and the action start probability set S is defined as When the action starts at the marked time point of the first video clip in the set S If it is greater than the first preset threshold, it can be classified into the action boundary action start boundary set, and the action start boundary set can be used Indicates that B S The number of points contained in is defined as N S ; Similarly, we can get the end boundary set B E .

[0061] Starting from the action boundary set B S and action end boundary set B E Take out the action start boundary t s and the action end boundary t e Combined into a candidate action nomination φ, when the candidate action nomination The length of time [t s ,t e ] satisfies the time length range of the action instance in the video to be processed, that is, the candidate nomination length d = t e -t s ∈[d min ,d max ], where d min ,d max are the shortest time and the longest time of the action instance in the video to be processed, respectively. Then the candidate action nomination φ is classified into the candidate action set ψ p , where ψ p Contains multiple candidate action nominations.

[0062] Step S104: extracting candidate nomination features corresponding to each candidate action nomination, and determining the confidence of each candidate action nomination based on the candidate nomination features.

[0063] Optionally, the candidate nomination features corresponding to each candidate action nomination can be extracted in the following manner: for each candidate action nomination, the candidate action nomination is divided into an action start area, an action in progress area, and an action end area; a first number of action start features are collected from the action start area, a second number of action in progress features are collected from the action in progress area, and a third number of action end features are collected from the action end area; the action start features, the action in progress features, and the action end features are concatenated to obtain the candidate nomination features corresponding to the candidate action nomination.

[0064] Specifically, for the candidate action set ψ obtained in step S103 p, for each candidate action nomination, a corresponding nomination feature CNF (Conjunctive normal form) feature is constructed. In the embodiment of the present application, a start area, an action in progress area, and an end area are constructed for each candidate nomination, and 8 points are sampled for the start area and the end area of ​​the candidate nomination, and 16 points are sampled for the action in progress area of ​​the candidate nomination. The sampled features of the three areas are spliced ​​to obtain the candidate nomination feature, and the nomination feature is recorded as in, The nomination features obtained by sampling the region where the candidate nomination action is in progress, represents the nomination features obtained by sampling the candidate nomination start area, Indicates the nomination features obtained by sampling the candidate nomination end area.

[0065] Optionally, the confidence of each candidate nominated feature can be determined by the following method: the candidate nominated feature corresponding to each candidate action nomination is Input into the two-layer perceptron to obtain the confidence of each candidate action nomination. The confidence of the candidate action nomination can be defined as p conf , which is used to represent the overlap between the candidate action nomination and the action instance in the video to be processed. The generated candidate nomination features (CNF features) are evaluated by a two-layer perceptron, and the evaluation result can be defined as In the expression The action starts t s and the action ends e probability.

[0066] Step S105 , performing non-maximum suppression processing on the multiple candidate action nominations based on the confidences of the multiple candidate action nominations to obtain a target action nomination.

[0067] Since multiple nominations with different overlapping times are generated during the nomination process, redundant nominations can be suppressed by adopting a non-maximum suppression algorithm.

[0068] In an optional embodiment of the present application, a first candidate action nomination with the highest confidence and the remaining second candidate action nominations among multiple candidate action nominations are determined, the first candidate action nomination is determined to be the target action nomination, and the overlap between the second candidate action nomination and the first candidate action nomination is compared. If the overlap is less than a second preset threshold, the confidence of the second candidate action nomination remains unchanged; if the overlap is not less than the second preset threshold, the confidence of the second candidate action nomination is subjected to fractional Gaussian decay; the second candidate action nomination and the first candidate action nomination whose processed confidence is lower than the third preset threshold are removed from the multiple candidate action nominations, and the above processing flow is repeated for the remaining candidate action nominations until the last candidate action nomination remains, and the last candidate action nomination is determined to be the target action nomination as well.

[0069] Specifically, the Soft-NMS (Non-maximum suppression) algorithm is used to process multiple candidate action nominations to obtain high-quality nominations. The Soft-NMS algorithm considers the degree of overlap between the confidence score and the bounding box when performing non-maximum suppression. It mainly suppresses redundant results through the confidence score attenuation function to ensure the output of high-quality structures. The processing process of the Soft-NMS algorithm is:

[0070] First, all candidate action nominations are sorted according to the confidence score, and the first candidate action nomination with the highest confidence score is determined. Nominate the target action, then calculate the first candidate action nomination with the highest confidence score Calculate the overlap with the remaining second candidate action nominations. If the obtained overlap is less than the second threshold, the confidence score of the second candidate action nomination remains unchanged; if the obtained overlap is not less than the second preset threshold, the confidence of the second candidate action nomination is subjected to fractional Gaussian decay, and the second candidate action nominations and the first candidate action nominations whose confidence scores are lower than the third preset threshold are removed from all candidate action nominations. In this way, the remaining candidate action nominations are iterated, and the candidate action nominations with confidence scores lower than the third preset threshold are removed after each iteration until the last candidate action nomination remains and is determined to be the target action nomination. After the iteration, one or more target action nominations can be obtained. The Gaussian decay function in the above Soft-NMS algorithm can be expressed as:

[0071]

[0072] in, Indicates the candidate action nomination with the highest confidence score The overlap degree with the remaining second candidate action nominations is calculated, ω represents the Gaussian function parameter setting, γ represents the second preset threshold, and the values ​​of ω, γ and the third preset threshold can be set according to experience, and no restrictions are imposed here.

[0073] After passing the non-maximum suppression Soft-NMS algorithm, we can get the final nomination

[0074] By using the Transformer model to decompose the input video clips, and then inputting the decomposed multiple first video clips into the pre-trained atomic action evaluation model, the atomic action probability corresponding to each first video and the corresponding atomic action probability set are determined, and then the action boundary set is determined from the marked time points corresponding to the multiple first video clips according to the atomic action probability set, and multiple candidate action nominations are determined from the action boundary set to form a candidate action nomination set, and boundary-sensitive nomination features are constructed for the candidate action nomination set and the boundary-sensitive nomination features are evaluated to obtain the final confidence score of each candidate action nomination in the candidate action nomination set and containing the action instance, so as to select high-confidence candidate nominations, and finally remove redundant nominations through the non-maximum suppression algorithm to obtain high-quality nominations. The above steps improve the end-to-end video classification effect, simplify the process of obtaining action nominations during temporal action detection, and ensure the output of high-quality nominations.

[0075] Example 2

[0076] According to an embodiment of the present application, a time sequence action nomination device for implementing the above time sequence action nomination method is also provided. Figure 4 As shown, the device at least includes an acquisition module 41, a probability assessment module 42, a boundary determination module 43, a feature extraction module 44 and a nomination determination module 45, wherein:

[0077] The acquisition module 41 is used to acquire the video to be processed and divide the video to be processed into a plurality of first video segments, wherein each first video segment has a corresponding marked time point.

[0078] In order to better analyze the action nominations in the video, after the acquisition module obtains the original video, it can be segmented, that is, the overlapping parts of the video to be processed can be divided into multiple small video clips. In the embodiment of the present application, the video to be processed can be divided into multiple first video clips by an equally spaced sampling method, wherein any two adjacent first video clips have overlapping video frame images.

[0079] The probability evaluation module 42 is used to determine the atomic action probability corresponding to each first video clip based on a pre-trained atomic action evaluation model to obtain an atomic action probability set, wherein the atomic action evaluation model is a Transformer model, and the atomic actions include: action start, action in progress and action end.

[0080] Optionally, multiple first video clips are input into the atomic action probability evaluation module in sequence to obtain the atomic action probability corresponding to each first video clip, and the atomic action probability includes: action start probability, action ongoing probability and action end probability; the atomic action probabilities corresponding to the multiple first video clips are taken as an atomic action probability set, and the atomic action probability set includes: action start probability set, action ongoing probability set and action end probability set.

[0081] In the embodiment of the present application, it is necessary to determine the atomic action probabilities corresponding to multiple first video clips based on the atomic action probability evaluation module. Therefore, the atomic action probability evaluation module needs to be trained, and the training process can be divided into the following steps: obtaining training sample videos and annotation information, and the annotation information is used to annotate action instances in the training sample videos; dividing the training sample videos into multiple second video clips, and predicting the atomic action probability corresponding to each second video clip based on the atomic action evaluation model to be trained; constructing a target loss function based on the atomic action probability and annotation information corresponding to each second video clip; based on the target loss function, adjusting the model parameters of the atomic action evaluation model to be trained by back-propagation gradient descent.

[0082] Among them, the target loss in the atomic action evaluation module can be obtained in the following way: construct a first loss function based on the action start probability and annotation information corresponding to each second video clip; construct a second loss function based on the action ongoing probability and annotation information corresponding to each second video clip; construct a third loss function based on the action end probability and annotation information corresponding to each second video clip; determine the sequence probability based on the action start probability, action ongoing probability and action end probability corresponding to every three adjacent second video clips, and construct a fourth loss function based on all sequence probabilities, wherein the sequence probability is used to indicate whether every three adjacent second video clips satisfy the atomic action sequence; construct a target loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function.

[0083] The boundary determination module 43 is used to determine an action boundary set from the marked time points corresponding to the multiple first video clips based on the atomic action probability set, and determine multiple candidate action nominations based on the action boundary set, wherein the action boundary set includes an action start boundary set and an action end boundary set.

[0084] Optionally, the action boundary set and multiple candidate action nominations can be determined in the following manner: determine the marked time point of the first video segment corresponding to each action start probability greater than a first preset threshold in the action start probability set as the action start boundary, and obtain the action start boundary set; determine the marked time point of the first video segment corresponding to each action end probability greater than the first preset threshold in the action end probability set as the action end boundary, and obtain the action end boundary set; each time, take an action start boundary and an action end boundary from the action start boundary set and the action end boundary set, if the time length corresponding to the action start boundary and the action end boundary meets the time length range of the action instance in the video to be processed, determine a candidate action nomination based on the action start boundary and the action end boundary; traverse the action start boundary set and the action end boundary set to obtain multiple candidate action nominations.

[0085] The feature extraction module 44 is used to extract the candidate nomination features corresponding to each candidate action nomination, and determine the confidence of each candidate action nomination based on the candidate nomination features.

[0086] Optionally, the candidate nominated features corresponding to each candidate action can be extracted in the following manner: for each candidate action nomination, the candidate action nomination is divided into an action start area, an action in progress area and an action end area; a first number of action start features are collected from the action start area, a second number of action in progress features are collected from the action in progress area, and a third number of action end features are collected from the action end area; the action start features, the action in progress features and the action end features are concatenated to obtain the candidate nominated features corresponding to the candidate action nomination.

[0087] Optionally, the confidence of each candidate action nomination can be determined by: inputting the candidate nomination features corresponding to each candidate action nomination into a two-layer perceptron to obtain the confidence of each candidate action nomination, wherein the confidence is used to represent the overlap between the candidate action nomination and the action instance in the video to be processed.

[0088] The nomination determination module 45 is used to perform non-maximum suppression processing on multiple candidate action nominations based on the confidence levels of the multiple candidate action nominations to obtain a target action nomination.

[0089] In an optional embodiment of the present application, a first candidate action nomination with the highest confidence and the remaining second candidate action nominations among multiple candidate action nominations are determined, the first candidate action nomination is determined to be the target action nomination, and the overlap between the second candidate action nomination and the first candidate action nomination is compared. If the overlap is less than a second preset threshold, the confidence of the second candidate action nomination remains unchanged; if the overlap is not less than the second preset threshold, the confidence of the second candidate action nomination is subjected to fractional Gaussian decay; the second candidate action nomination and the first candidate action nomination whose processed confidence is lower than the third preset threshold are removed from the multiple candidate action nominations, and the above processing flow is repeated for the remaining candidate action nominations until the last candidate action nomination remains, and the last candidate action nomination is determined to be the target action nomination as well.

[0090] It should be noted that each module in the time sequence action nomination device in the embodiment of the present application corresponds one by one to each implementation step of the time sequence action nomination method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1, and will not be repeated here.

[0091] Example 3

[0092] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the sequential action nomination method in Example 1.

[0093] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following steps:

[0094] Step S1, obtaining a video to be processed, and dividing the video to be processed into a plurality of first video segments, wherein each first video segment has a corresponding marked time point;

[0095] Step S2: determining the atomic action probability corresponding to each first video clip based on the pre-trained atomic action evaluation model to obtain an atomic action probability set, wherein the atomic action evaluation model is a Transformer model, and the atomic actions include: action start, action in progress, and action end;

[0096] Step S3, determining an action boundary set from the marked time points corresponding to the plurality of first video clips based on the atomic action probability set, and determining a plurality of candidate action nominations based on the action boundary set;

[0097] Step S4: extracting candidate nomination features corresponding to each candidate action nomination, and determining the confidence of each candidate action nomination based on the candidate nomination features;

[0098] Step S5: performing non-maximum suppression processing on the multiple candidate action nominations based on the confidences of the multiple candidate action nominations to obtain a target action nomination.

[0099] According to an embodiment of the present application, a processor is further provided, which is used to run a program, wherein the timing action nomination method in embodiment 1 is executed when the program is running.

[0100] Optionally, the following steps are performed when the program is running:

[0101] Obtain a video to be processed, and divide the video to be processed into a plurality of first video segments, wherein each first video segment has a corresponding marked time point; determine the atomic action probability corresponding to each first video segment based on a pre-trained atomic action evaluation model, and obtain an atomic action probability set, wherein the atomic action evaluation model is a Transformer model, and the atomic actions include: action start, action in progress, and action end; determine an action boundary set from the marked time points corresponding to the plurality of first video segments based on the atomic action probability set, and determine a plurality of candidate action nominations based on the action boundary set; extract the candidate nomination feature corresponding to each candidate action nomination, and determine the confidence of each candidate action nomination based on the candidate nomination feature; perform non-maximum suppression processing on the plurality of candidate action nominations based on the confidence of the plurality of candidate action nominations, and obtain a target action nomination.

[0102] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0103] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0106] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0108] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for nominating sequential actions, characterized in that: include: Acquire a video to be processed, and divide the video to be processed into a plurality of first video segments, wherein each of the first video segments has a corresponding marked time point; Determine the atomic action probability corresponding to each of the first video clips based on a pre-trained atomic action evaluation model to obtain an atomic action probability set, wherein the atomic action evaluation model is a Transformer model, and the atomic actions include: action start, action in progress, and action end; Determining an action boundary set from the marked time points corresponding to the plurality of first video clips based on the atomic action probability set, and determining a plurality of candidate action nominations based on the action boundary set; Extracting a candidate nomination feature corresponding to each of the candidate action nominations, and determining a confidence level of each of the candidate action nominations based on the candidate nomination features; Performing non-maximum suppression processing on the multiple candidate action nominations based on the confidences of the multiple candidate action nominations to obtain a target action nomination; Wherein, determining the atomic action probability corresponding to each of the first video clips based on the pre-trained atomic action evaluation model to obtain an atomic action probability set includes: inputting the multiple first video clips into the atomic action evaluation model in sequence to obtain the atomic action probability corresponding to each of the first video clips, the atomic action probability including: an action start probability, an action ongoing probability and an action end probability; using the atomic action probabilities corresponding to the multiple first video clips as the atomic action probability set, the atomic action probability set including: an action start probability set, an action ongoing probability set and an action end probability set; Wherein, the training process of the atomic action evaluation model includes: obtaining a training sample video and annotation information, wherein the annotation information is used to annotate the action instances in the training sample video; dividing the training sample video into a plurality of second video segments, and predicting the atomic action probability corresponding to each second video segment based on the atomic action evaluation model to be trained; constructing a first loss function based on the action start probability and the annotation information corresponding to each second video segment; constructing a second loss function based on the action ongoing probability and the annotation information corresponding to each second video segment; constructing a third loss function based on the action end probability and the annotation information corresponding to each second video segment; determining a sequence probability based on the action start probability, the action ongoing probability and the action end probability corresponding to every three adjacent second video segments, and constructing a fourth loss function based on all the sequence probabilities, wherein the sequence probability is used to indicate whether every three adjacent second video segments satisfy the atomic action sequence; constructing a target loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function; based on the target loss function, adjusting the model parameters of the atomic action evaluation model to be trained by back-propagation gradient descent; The fourth loss function is defined as an action penalty function, which is expressed by the following formula: In the formula, v(p i-1 ,p i ,p i+1 ) represents the atomic action relation restriction function, p i-1 、p i 、p i+1 Represent the true annotation values ​​in the i-1th, i, and i+1th time windows respectively. i-1 、p i 、p i+1 When it is judged as a continuous repeated mark or conforms to the sequence of atomic action steps, v(p i-1 ,p i ,p i+1 ) is calibrated to 0, otherwise v(p i-1 ,p i ,p i+1 ) is marked as 1.

2. The method according to claim 1, characterized in that Dividing the video to be processed into a plurality of first video segments, including: The video to be processed is divided into a plurality of the first video segments by an equal-interval sampling method, wherein any two adjacent first video segments have overlapping video frame images.

3. The method according to claim 1, characterized in that The action boundary set includes an action start boundary set and an action end boundary set, determining the action boundary set from the marked time points corresponding to the plurality of first video clips based on the atomic action probability set, and determining a plurality of candidate action nominations based on the action boundary set, including: Determine the marked time point of the first video segment corresponding to each action start probability greater than a first preset threshold in the action start probability set as an action start boundary, and obtain the action start boundary set; Determine the marked time point of the first video segment corresponding to each action end probability greater than the first preset threshold in the action end probability set as an action end boundary, and obtain the action end boundary set; Each time, one action start boundary and one action end boundary are respectively selected from the action start boundary set and the action end boundary set, and if the time lengths corresponding to the action start boundary and the action end boundary meet the time length range of the action instance in the video to be processed, a candidate action nomination is determined based on the action start boundary and the action end boundary; The action start boundary set and the action end boundary set are traversed to obtain the multiple candidate action nominations.

4. The method according to claim 1, characterized in that: Extracting candidate nomination features corresponding to each candidate action nomination includes: For each of the candidate action nominations, the candidate action nomination is divided into an action start area, an action ongoing area, and an action end area; respectively collecting a first number of action start features from the action start area, collecting a second number of action in-progress features from the action in-progress area, and collecting a third number of action end features from the action end area; The action start feature, the action ongoing feature and the action end feature are concatenated to obtain the candidate nominated feature corresponding to the candidate action nomination.

5. The method according to claim 4, characterized in that Determining the confidence of each candidate action nomination based on the candidate nomination features includes: The candidate nomination feature corresponding to each candidate action nomination is input into a two-layer perceptron to obtain the confidence of each candidate action nomination, wherein the confidence is used to indicate the overlap between the candidate action nomination and the action instance in the video to be processed.

6. The method according to claim 1, characterized in that The method further comprises: performing non-maximum suppression processing on the plurality of candidate action nominations based on the confidences of the plurality of candidate action nominations to obtain a target action nomination, including: Determine a first candidate action nomination with the highest confidence among the multiple candidate action nominations and the remaining second candidate action nominations, determine the first candidate action nomination as the target action nomination, and compare the overlap between the second candidate action nomination and the first candidate action nomination, if the overlap is less than a second preset threshold, the confidence of the second candidate action nomination remains unchanged, if the overlap is not less than the second preset threshold, perform fractional Gaussian decay on the confidence of the second candidate action nomination; The second candidate action nomination and the first candidate action nomination whose confidence after processing is lower than the third preset threshold are removed from the multiple candidate action nominations, and the above processing flow is repeated for the remaining candidate action nominations until the last candidate action nomination remains, and it is determined that the last candidate action nomination is also the target action nomination.

7. A time sequence action nomination device, characterized in that: include: An acquisition module, used for acquiring a video to be processed, and dividing the video to be processed into a plurality of first video segments, wherein each of the first video segments has a corresponding marked time point; A probability evaluation module, for determining the atomic action probability corresponding to each of the first video clips based on a pre-trained atomic action evaluation model, and obtaining an atomic action probability set, comprising: inputting the multiple first video clips into the atomic action evaluation model in sequence, and obtaining the atomic action probability corresponding to each of the first video clips, the atomic action probability including: action start probability, action ongoing probability and action end probability; using the atomic action probabilities corresponding to the multiple first video clips as the atomic action probability set, the atomic action probability set including: action start probability set, action ongoing probability set and action end probability set; wherein the atomic action evaluation model is a Transformer model, the atomic actions include: action start, action ongoing and action end, and the training process of the atomic action evaluation model includes: obtaining training sample videos and annotation information, the annotation information is used to annotate action instances in the training sample videos; dividing the training sample videos into multiple second video clips, and predicting each second video clip based on the atomic action evaluation model to be trained. The atomic action probability corresponding to the second video clip; constructing a first loss function based on the action start probability and the annotation information corresponding to each second video clip; constructing a second loss function based on the action ongoing probability and the annotation information corresponding to each second video clip; constructing a third loss function based on the action end probability and the annotation information corresponding to each second video clip; determining the sequence probability based on the action start probability, the action ongoing probability and the action end probability corresponding to every three adjacent second video clips, and constructing a fourth loss function based on all the sequence probabilities, wherein the sequence probability is used to indicate whether every three adjacent second video clips satisfy the atomic action sequence; constructing a target loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function; based on the target loss function, adjusting the model parameters of the atomic action evaluation model to be trained by back-propagation gradient descent; the fourth loss function is defined as an action penalty function, which is expressed by the following formula: In the formula, v(p i-1 ,p i ,p i+1 ) represents the atomic action relation restriction function, p i-1 、p i 、p i+1 Represent the true annotation values ​​in the i-1th, i, and i+1th time windows respectively. i-1 、p i 、p i+1 When it is judged as a continuous repeated mark or conforms to the sequence of atomic action steps, v(p i-1 ,p i ,p i+1 ) is calibrated to 0, otherwise v(p i-1 ,p i ,p i+1 ) is marked as 1; A boundary determination module, configured to determine an action boundary set from the marked time points corresponding to the plurality of first video clips based on the atomic action probability set, and determine a plurality of candidate action nominations based on the action boundary set; A feature extraction module, used to extract candidate nomination features corresponding to each candidate action nomination, and determine the confidence of each candidate action nomination based on the candidate nomination features; The nomination determination module is used to perform non-maximum suppression processing on the multiple candidate action nominations based on the confidence levels of the multiple candidate action nominations to obtain a target action nomination.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the sequential action nomination method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the time sequence action nomination method according to any one of claims 1 to 6 through the computer program.

Images