Action boundary detection model, model training method, device and storage medium

Abstract

The present invention discloses an action boundary detection model, model training method, device, and storage medium. The action boundary detection model includes: a temporal action nomination network for predicting action candidate regions based on video features; a temporal unidirectional feature processing module for extracting temporal unidirectional features corresponding to the action candidate regions based on video features; a linear interpolation sampling module for intercepting features to be identified from the temporal unidirectional features based on the action candidate regions; and an action detection network for determining the action category, actual start time, and actual end time corresponding to the action segment based on the action candidate regions and the features to be identified. The present invention improves the efficiency and accuracy of detecting action categories and action boundaries in videos.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to an action boundary detection model, a model training method, a device and a storage medium. Background Art

[0002] With video social networking becoming increasingly popular in today's society, video content analysis has gradually become a major focus in the field of artificial intelligence (AI). Currently, most video analysis focuses on understanding the content of pre-edited videos, while research on identifying relevant scenes in unedited videos is relatively new. For example, action segment detection in unedited videos involves detecting human movements, as well as the time and location of their occurrence, within a video.

[0003] Current methods for detecting action clips involve sampling video clips at different time scales. Within the resulting multi-scale sampled clips, a single-scale sliding window is used to sample the clips. A deep residual network is then used for binary classification. The network output and voting methods are used to determine whether the multi-scale clips contain action. Frame-level action recognition methods are then used to output the action category for each video frame. Based on the results for each frame, a statistical fusion strategy is used to determine the action category and action boundaries. This method uses a sliding window approach, resulting in relatively rough detection results. The larger the number of sliding windows, the more computational effort and speed. Furthermore, the subsequent recognition of each frame fails to fully utilize the inter-frame information in the entire video, leading to inaccurate action boundary recognition. Summary of the Invention

[0004] The embodiments of the present invention provide an action boundary detection model, a model training method, a device and a storage medium, aiming to solve the technical problem of inaccurate action boundary recognition in videos using existing action segment detection methods.

[0005] An embodiment of the present invention provides an action boundary detection model, which includes:

[0006] A temporal action nomination network, configured to predict an action candidate region based on video features, wherein the action candidate region is a video segment between a start time and an end time predicted by the temporal action nomination network;

[0007] A temporal unidirectional feature processing module, configured to extract temporal unidirectional features corresponding to the action candidate area based on the video features;

[0008] A linear interpolation sampling module, configured to intercept features to be identified from the temporal unidirectional features based on the action candidate region;

[0009] The action detection network is used to determine the action category, actual start time and actual end time corresponding to the action segment based on the action candidate area and the features to be identified.

[0010] In one embodiment, the temporal action nomination network includes:

[0011] Perimeter-sensitive networks;

[0012] The nomination selection module is connected to the boundary-sensitive network, and the nomination selection module includes matrix multiplication and sorting filtering.

[0013] In one embodiment, the temporal unidirectional feature processing module includes n temporal unidirectional convolutions connected in sequence, each of the temporal unidirectional convolutions has a hole coefficient, and the ratio between the hole coefficients of any two adjacent temporal unidirectional convolutions is the same.

[0014] In addition, the present invention also provides a model training method, which is applied to the action boundary detection model. The model training method includes:

[0015] Obtain video features and annotation information of sample videos;

[0016] Inputting the video features into a temporal action nomination network to obtain an action candidate region, where the action candidate region is a video segment between a start time and an end time predicted by the temporal action nomination network;

[0017] Input the video features into a temporal unidirectional feature processing module, and output the temporal unidirectional features corresponding to the action candidate area;

[0018] Inputting the starting time, the ending time, and the time series unidirectional feature into a linear interpolation sampling module, and outputting a feature to be identified, where the feature to be identified is a local feature intercepted by the linear interpolation sampling module from the time series unidirectional feature according to the starting time and the ending time;

[0019] Inputting the start time, the end time, and the feature to be identified into an action detection network, and outputting the action category, the actual start time, and the actual end time corresponding to the action candidate area;

[0020] The action boundary detection model is jointly trained according to the video features, the annotation information, the action category, the actual start time, and the actual end time to obtain the trained action boundary detection model.

[0021] In one embodiment, the step of inputting the video features into a temporal unidirectional feature processing module and outputting the temporal unidirectional features corresponding to the action candidate regions includes:

[0022] Extracting, from the video features, a first image feature of each video frame corresponding to the action candidate region and a second image feature of a video frame preceding the video frame by the temporal unidirectional feature processing module;

[0023] The second image feature is unidirectionally fused with the first image feature to obtain the temporal unidirectional feature.

[0024] In one embodiment, the step of inputting the start time, the end time, and the feature to be identified into an action detection network and outputting the action category, the actual start time, and the actual end time corresponding to the action candidate region includes:

[0025] Identifying the to-be-identified features through the action detection network to obtain an action category of the action candidate area, a first change amount at the starting time, and a second change amount at the ending time;

[0026] Determining the actual starting time according to the starting time and the first change; and

[0027] The actual end time is determined according to the end time and the second change.

[0028] In one embodiment, the step of jointly training the action boundary detection model based on the video features, the annotation information, the action category, the actual start time, and the actual end time to obtain the trained action boundary detection model includes:

[0029] Determine an action category loss value, a start time loss value, and an end time loss value according to the annotation information, the action category, the actual start time, and the actual end time;

[0030] When the action category loss value is less than a first threshold, the starting moment loss value is less than a second threshold, and the ending moment loss value is less than a third threshold, the training of the action boundary detection model is stopped to obtain the trained action boundary detection model.

[0031] In one embodiment, before the step of obtaining the video features and annotation information of the sample video, the method further includes:

[0032] Extracting an image feature vector of each video frame in the sample video based on a pre-trained classification network;

[0033] The video feature is generated according to the image feature vector of each video frame.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal device, which includes: a memory, a processor, and a model training program stored on the memory and runnable on the processor, and the model training program implements the steps of the above-mentioned model training method when executed by the processor.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium on which a model training program is stored, and when the model training program is executed by a processor, the steps of the above-mentioned model training method are implemented.

[0036] The technical solutions of an action boundary detection model, model training method, device, and storage medium provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0037] The action boundary detection model of the present invention includes a temporal action nomination network for predicting action candidate regions based on video features; a temporal unidirectional feature processing module for extracting temporal unidirectional features corresponding to the action candidate regions based on video features; a linear interpolation sampling module for intercepting the to-be-identified features from the temporal unidirectional features based on the action candidate regions; and an action detection network for determining the action category, actual start time, and actual end time corresponding to the action segment based on the action candidate regions and the to-be-identified features. The present invention uses the action boundary detection model to detect action categories and action boundaries in videos, thereby improving the efficiency and accuracy of detecting action categories and action boundaries in videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the structure of the model training device of the present invention;

[0039] Figure 2 Schematic diagram of the structure of the temporal action nomination network;

[0040] Figure 3 It is a structural diagram of the time series unidirectional feature processing module;

[0041] Figure 4 This is a flow chart of an embodiment of the model training method of the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention. DETAILED DESCRIPTION

[0043] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0044] like Figure 1 As shown, the present invention proposes an embodiment of an action boundary detection model, which is specifically an EPTAL network (efficient and precise temporal action localization network), which includes: a temporal action nomination network 110, a temporal unidirectional feature processing module 120, a linear interpolation sampling module 130 and an action detection network 140; the temporal action nomination network 110 and the temporal unidirectional feature processing network 120 are connected to the linear interpolation sampling module 130, and the temporal action nomination network 110 and the linear interpolation sampling module 130 are connected to the action detection network 140.

[0045] The temporal action nomination network 110 is used to predict action candidate regions based on video features. The video features are the input of the temporal action nomination network 110, and the video features are a sequence of feature vectors composed of image features of each video frame in the video. The action candidate region is a video segment between the start time and the end time predicted by the temporal action nomination network 110. For example, the temporal action nomination network 110 predicts that the start time of the athlete's running segment from the video based on the video features is 1 minute 20 seconds, and the end time is 1 minute 30 seconds. The athlete's running segment from 1 minute 20 seconds to 1 minute 30 seconds is the predicted action candidate region. 1 minute 20 seconds and 1 minute 30 seconds are the start time and end time of the action candidate region, respectively. Then, the output of the temporal action nomination network 110 is the start time and end time of the action candidate region. There are multiple action candidate regions.

[0046] The temporal unidirectional feature processing module 120 is used to extract the temporal unidirectional features corresponding to the action candidate area based on the video features. The temporal unidirectional features are obtained by unidirectional fusion of the image features of each video frame in the action candidate area and the image features of the video frame before each video frame. Due to the particularity of the human visual perception system, humans observe temporal information by inferring the current events through the previous temporal sequence information and the information currently seen. For example, when humans watch TV, they can infer the events happening in the current frame by seeing the current frame and the frame information in the previous memory. Similarly, in the temporal detection field, tasks such as the occurrence of an action or a certain scene, such as the shooting action of a football player, will have a sequential association of running with the ball first, then raising the leg, and finally kicking the ball. If played in reverse, it is not a shooting action, but may be a dance or other action. Therefore, by utilizing this temporal sequential relationship, the boundaries of the action can be better determined. Based on this, the temporal unidirectional feature can be understood as a feature with a larger receptive field and more consistent with human visual observation obtained by globally fusing the image features of a local single video frame. Then, the temporal unidirectional feature processing module 120 obtains the temporal unidirectional feature corresponding to the action candidate area.

[0047] The linear interpolation sampling module 130 is used to extract features to be identified from the temporal unidirectional features based on the action candidate regions. It should be understood that the start and end times of the action candidate regions and the temporal unidirectional features corresponding to the action candidate regions are input to the linear interpolation sampling module 130 as input data. The linear interpolation sampling module 130 selects features corresponding to the start and end times from the temporal unidirectional features based on the start and end times.

[0048] Since the length of the time period formed by the start and end times of the action candidate regions output by the temporal action nomination network 110 is inconsistent, and the number of video frames in the video frame sequences corresponding to different time periods is different, if the detection of action boundaries in the video is achieved through the batch operation of the system's GPU, then the action candidate regions with inconsistent time periods cannot be processed in parallel based on the video frame sequences with different frame numbers, which will reduce the efficiency of action boundary detection. Based on this, the linear interpolation sampling module 130 samples the features of n video frames corresponding to the start and end times in a proportional manner from the temporal unidirectional features, that is, the features to be identified are intercepted, and the features to be identified are local features in the temporal unidirectional features. For example, the starting time and ending time of the action candidate area A correspond to the 1st and 3rd seconds, and the starting time and ending time of feature A corresponding to the starting time and ending time are selected from the temporal unidirectional features. The starting time and ending time also correspond to the 1st and 3rd seconds, and then n frames of video frames corresponding to the starting time and ending time are sampled proportionally, and then the features of n frames of video frames are extracted from the temporal unidirectional features; the starting time and ending time of the action candidate area B correspond to the 4th and 9th seconds, and the starting time and ending time of feature B corresponding to the starting time and ending time are selected from the temporal unidirectional features. The starting time and ending time also correspond to the 4th and 9th seconds, and then n frames of video frames corresponding to the starting time and ending time are sampled proportionally, and then the features of n frames of video frames are extracted from the temporal unidirectional features. Furthermore, in order to realize the detection of action boundaries in the video through the batch operation of the GPU of the system, the features to be identified in the same format must be intercepted from feature A and feature B in the same format, such as the features to be identified from 0-1s, which are feature A to be identified and feature B to be identified, respectively. Feature A to be identified and feature B to be identified both include the features of n frames of video frames. At this time, the starting time and ending time of feature A to be identified and feature B to be identified have nothing to do with the starting time and ending time of the above-mentioned action candidate area A and action candidate area B. It can be understood that feature A to be identified and feature B to be identified are the temporal unidirectional features corresponding to the new video clip, and the starting time of the two is 0s and the ending time is 1s. That is, the format of the features to be identified output by the linear interpolation sampling module 130 is consistent, that is, they are all features of n frames of video frames, that is, the format of the features to be identified input to the features to be identified 140 is consistent. In this way, the detection of action boundaries in the video can be realized through the batch operation of the GPU, thereby improving detection efficiency.

[0049] The action detection network 140 is used to determine the action category, actual start time, and actual end time corresponding to the action segment based on the action candidate region and the features to be identified. Specifically, the action detection network 140 includes two fully connected layers, denoted as the first fully connected layer and the second fully connected layer. The first fully connected layer is used to determine the action category of the action candidate region based on the features to be identified, and the second fully connected layer outputs the change in the start time and the change in the end time based on the features to be identified. It can be understood that since the time periods of the features to be identified input to the features to be identified 140 are consistent, assuming that one of the features to be identified is feature A, the start time of the action candidate region A corresponding to feature A is 1s and the end time is 3s. The start time of feature A is 0 and the end time is 1s. After processing the feature to be identified by the second fully connected layer, the new start time output is 0.3s and the new end time is 0.8s. The change in the start time is 0.3s, and the end time is 1-0.8=0.2s, that is, 0.2s. Therefore, the actual start time of action candidate region A is 1.2s, and the actual end time is 2.8s. If the action category of action candidate region A output by the first fully connected layer is running, the output of the action detection network 140 includes: the action category of action candidate region A is running, the actual start time of action candidate region A is 1.2s, and the actual end time is 2.8s, thereby achieving accurate detection and recognition of action segment boundaries in the video.

[0050] Optional, such as Figure 2 As shown, the temporal action nomination network includes a boundary-sensitive network 111 and a nomination selection module 112, and the boundary-sensitive network 111 is connected to the nomination selection module 112. It is understandable that the temporal action nomination network 110 is also called a proposal network, and the nomination selection module 112 is also called a proposal selection module. The boundary-sensitive network 111 can be a BMN (Boundary-Matching Network) network or other boundary-sensitive networks. Because some boundary-sensitive networks such as the BMN network use peak filtering action start and end points during nomination, the number of nominations is greatly reduced and the number of proposals generated for each video is inconsistent, which in turn causes the network to be unable to perform batch processing. The present invention adds an improved nomination selection module 112 on the basis of the boundary-sensitive network 111, that is, the nomination selection module 112 includes matrix multiplication and sorting filtering. The improved nomination selection module 112 can not only quickly calculate all nominations, but also enable the network to perform batch processing, thereby improving the accuracy of the network.

[0051] Optional, such as Figure 3As shown, the temporal unidirectional feature processing module 120 includes n temporal unidirectional convolutions connected in sequence, each temporal unidirectional convolution has a dilation coefficient, and the ratio between the dilation coefficients of any two adjacent temporal unidirectional convolutions is the same. Among them, the temporal unidirectional convolution is expressed as SUConv, and the dilation coefficient is called dilation in English, which is expressed as D. After setting the dilation coefficient for each temporal unidirectional convolution, the temporal unidirectional feature includes the image features of each video frame and the image features of the video frame before each video frame at intervals of D frames. For example, D=2, 1s-3s includes 10 frames of video frames, where the current video frame is the 7th frame. It can be understood that the temporal unidirectional feature includes the image features of the 1st frame, the image features of the 4th frame, and the image features of the 7th frame.

[0052] Existing convolution operators are implemented in the spatial domain. Currently, both two-dimensional and one-dimensional convolutions use the center of the feature in the spatial or temporal domain as the sampling point, with the convolution calculation area surrounding or surrounding the center. For one-dimensional convolution, this results in the receptive field of the current video frame containing both preceding and following information during the one-dimensional convolution calculation. In particular, as the receptive field grows, deeper features contain more and more following information. This can easily lead to the incorporation of too much unnecessary information while increasing the receptive field, resulting in inaccurate boundaries in the final prediction result.

[0053] Based on the above-mentioned deficiencies, in order to increase the receptive field without introducing too much unnecessary information and without introducing a huge amount of calculation, the present invention proposes a temporal unidirectional feature processing module 120 including n temporal unidirectional convolutions. The specific calculation method of the temporal unidirectional convolution is to use the rightmost point of the convolution as the anchor point when performing the convolution calculation, and the final result of the convolution is the result value of the anchor point.

[0054] For example, the video feature is F, F = {f1,f2,...,f j},f∈R D , if a 1x3 temporal unidirectional convolution is used to convolve F, assuming that the convolution parameter S is {w j-2 ,w j-1 ,w j},w j ∈R DX1 , then for the current feature f j , the result after the convolution operation is In order to avoid introducing a huge amount of calculation, the hole coefficient is added to the newly proposed convolution operator S, that is, for the 1x3 convolution, when D=1, the calculation method is as shown in the above example. If D=2, for the current feature f j For example, the result after the convolution operation is The computational complexity can be reduced by setting a dilation coefficient for temporal unidirectional convolution.

[0055] Based on the above-mentioned action boundary detection model, the present invention provides an embodiment of a model training method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than here. Figure 4 As shown, the model training method of the present invention comprises the following steps:

[0056] Step S210: Obtain video features and annotation information of the sample video.

[0057] In this embodiment, the sample video is the source video for model training. After obtaining the sample video, video features are extracted from the sample video, and different video segments in the sample video are annotated to obtain annotation information. The annotation information and video features are used for subsequent training of the action boundary detection model. The annotation information includes the annotation category, annotation start time, and annotation end time corresponding to the action segment in the sample video. For example, if 1s-3s is dancing, the corresponding annotation category of the action segment is dancing, the annotation start time is the 1s, and the annotation end time is the 3s.

[0058] Specifically, before step S210, the following steps are also included:

[0059] Extracting an image feature vector of each video frame in the sample video based on a pre-trained classification network;

[0060] The video feature is generated according to the image feature vector of each video frame.

[0061] It should be understood that the classification network is pre-trained, such as a two-stream network (TSN, Temporal Segment Networks), which is used to extract video features. A sample video is input into the classification network, which extracts the image feature vector of each video frame in the sample video. The image feature vectors of each video frame are then combined into a feature vector sequence and output, thereby obtaining video features.

[0062] Step S220: Input the video features into a temporal action nomination network to obtain action candidate regions.

[0063] The temporal action nomination network, or proposal network, is pre-selected. Failure to process or improperly processing the proposal network can have two negative effects: too many proposal networks will lead to excessive feature extraction, reducing computational efficiency; and too few effective proposal networks will hinder learning the action detection network. Therefore, multiple proposal networks must be prepared in advance. Each proposal network is then scored based on its output, resulting in a confidence score. The network with the highest confidence score is then selected as the final proposal network. For example, the network with the highest confidence score is selected as the final proposal network.

[0064] Specifically, video features are fed into the temporal action nomination network to generate action candidate regions. These regions are the video segments between the start and end times predicted by the temporal action nomination network. In other words, the action candidate regions encompass the predicted start and end times of the action. The start and end times are based on the entire duration of the sample video. For example, if the sample video is 20 seconds long, the predicted start time of the dance action is the first second, and the end time is the third second.

[0065] Step S230: inputting the video features into a temporal unidirectional feature processing module, and outputting the temporal unidirectional features corresponding to the action candidate region.

[0066] While video features are input into the temporal action nomination network, they are also input into the temporal unidirectional feature processing module. Due to the particularity of the human visual perception system, humans observe temporal information by inferring the current event through previous temporal sequence information and the information currently seen. Based on this, the temporal unidirectional feature processing module processes the video features to obtain temporal unidirectional features corresponding to the action candidate area. Temporal unidirectional features include features at the end of the action and before the end of the action, and do not include information that has not yet occurred after the end of the action. In other words, the temporal unidirectional features are a fusion of features at the end of the action and before the end of the action, and the feature information in the temporal unidirectional features is unidirectional.

[0067] Step S240: input the starting time, the ending time and the time series unidirectional feature into a linear interpolation sampling module, and output the feature to be identified.

[0068] The feature to be identified is a local feature intercepted by the linear interpolation sampling module from the time series unidirectional feature according to the start time and the end time, that is, the time series unidirectional feature includes the feature to be identified, and the feature to be identified is also unidirectional.

[0069] Since the start and end times of the action candidate regions output by the temporal action nomination network are inconsistent in length, if model training is implemented through the system's GPU batch operation, it is impossible to parallelize the action candidate regions with inconsistent time periods, which will reduce the efficiency of model training.

[0070] Specifically, after the start time, end time and time series unidirectional features are input into the linear interpolation sampling module, the linear interpolation sampling module selects the features corresponding to the start time and end time from the time series unidirectional features, and then intercepts the features to be identified from the features corresponding to the selected start time and end time.

[0071] For example, the start time and end time of the action candidate area A correspond to the 1st second and the 3rd second, and the start time and end time of the feature A corresponding to the start time and the end time selected from the temporal unidirectional features also correspond to the 1st second and the 3rd second; the start time and end time of the action candidate area B correspond to the 4th second and the 6th second, and the start time and end time of the feature B corresponding to the start time and the end time selected from the temporal unidirectional features also correspond to the 4th second and the 6th second; furthermore, in order to realize model training through the batch operation of the system's GPU, that is, the features to be identified in the same format must be intercepted from features A and B in the same format, such as the features to be identified of 0-1s, which are features A to be identified and features B to be identified, respectively. At this time, the start time and end time of features A to be identified and features B to be identified have nothing to do with the start time and end time of the above-mentioned action candidate area A and action candidate area B. It can be understood that feature A and feature B are time-series unidirectional features corresponding to the new video clip, starting at the 0th second and ending at the 1st second. This means that the time segments of the different features to be identified output by the linear interpolation sampling module are consistent, and thus the time segments of the features to be identified input to the action detection network are consistent. This allows model training to be implemented using GPU batch operations, improving training efficiency.

[0072] Step S250: inputting the start time, the end time and the features to be identified into an action detection network, and outputting the action category, actual start time and actual end time corresponding to the action candidate area.

[0073] After obtaining the start time, end time, and features to be identified, they are input into the action detection network. The action detection network then outputs the action category, actual start time, and actual end time corresponding to the action candidate region. The action category is the final identified category, the actual start time is the final identified start time, and the actual end time is the final identified end time. The actual start time differs from the start time output by the temporal action nomination network, and the actual end time differs from the end time output by the temporal action nomination network.

[0074] Step S260: jointly training the action boundary detection model according to the video features, the annotation information, the action category, the actual start time, and the actual end time to obtain the trained action boundary detection model.

[0075] After obtaining the action category, actual start time, and actual end time corresponding to the action candidate area, the annotation information, the action category, actual start time, and actual end time corresponding to the action candidate area are output to the corresponding loss function, and the loss value corresponding to the action category, the loss value corresponding to the actual start time, and the loss value corresponding to the actual end time are calculated respectively. If the loss value corresponding to the action category, the loss value corresponding to the actual start time, and the loss value corresponding to the actual end time are all greater than or equal to the corresponding threshold, the action boundary detection model continues to be trained using video features; if the loss value corresponding to the action category, the loss value corresponding to the actual start time, and the loss value corresponding to the actual end time are all less than the corresponding threshold, the training of the action boundary detection model is stopped to obtain the trained action boundary detection model. Among them, there are two types of loss functions, one for calculating the loss of the action category, and the other for calculating the loss of the start and end times. For example, the loss of the action category is calculated using Cross entropy loss, and the loss of the start and end times is calculated using smoothL1 loss.

[0076] Alternatively, the action boundary detection model is iteratively trained according to a preset number of training times. If the iterative training reaches the preset number of training times, it is considered that the error of the action boundary detection model is within an allowable range, that is, the error is very small, and the training of the action boundary detection model is stopped to obtain the trained action boundary detection model.

[0077] According to the above technical solution, this embodiment not only improves the training speed of the action boundary detection model, but also improves the detection efficiency and accuracy of action categories and action boundaries in the video.

[0078] Optionally, step S230 includes the following steps:

[0079] Extracting, from the video features, a first image feature of each video frame corresponding to the action candidate region and a second image feature of a video frame preceding the video frame by the temporal unidirectional feature processing module;

[0080] The second image feature is unidirectionally fused with the first image feature to obtain the temporal unidirectional feature.

[0081] Specifically, after the video features are input into the temporal unidirectional feature processing module, the module extracts the first image features of each video frame corresponding to the action candidate region, as well as the second image features of the previous video frame, from the video features. The module then unidirectionally fuses the second image features with the first image features in the order of the video frames to obtain the temporal unidirectional features. For example, if the hole coefficient D = 1, 1s-3s includes 10 video frames, of which the current video frame is the 7th, the second image features of frames 1-6 are unidirectionally fused with the first image features of frame 7 in the order of the video frames to obtain the temporal unidirectional features. For another example, if the hole coefficient D = 2, 1s-3s includes 10 video frames, of which the current video frame is the 7th, the second image features of frame 1, frame 4, and frame 7 are unidirectionally fused in the order of the video frames to obtain the temporal unidirectional features. The characteristic information in the temporal unidirectional features is sequential. Obtaining the temporal unidirectional features helps reduce the computational complexity during model training and improves training efficiency.

[0082] Optionally, step S250 includes the following steps:

[0083] Identifying the to-be-identified features through the action detection network to obtain an action category of the action candidate area, a first change amount at the starting time, and a second change amount at the ending time;

[0084] Determining the actual starting time according to the starting time and the first change; and

[0085] The actual end time is determined according to the end time and the second change.

[0086] Specifically, after obtaining the feature to be identified, the feature to be identified also corresponds to a start time and an end time. At the same time, the feature to be identified and the start time and end time corresponding to the action candidate area are input into the action detection network. The action detection network identifies the feature to be identified through the first fully connected layer and obtains the action category corresponding to the action candidate area. The action detection network identifies the feature to be identified through the second fully connected layer, outputs the new start time and new end time corresponding to the feature to be identified, and then subtracts the new start time corresponding to the feature to be identified from the start time corresponding to the feature to be identified to obtain a first change; and subtracts the new end time corresponding to the feature to be identified from the end time corresponding to the feature to be identified to obtain a second change. Then, the actual start time of the action candidate area is: the sum of the start time corresponding to the action candidate area and the first change, and the actual end time of the action candidate area is: the difference between the end time corresponding to the action candidate area and the second change, thereby achieving accurate detection and recognition of the boundaries of action segments in the video.

[0087] Assume that one of the features to be identified is feature A, and the corresponding action candidate region A has a start time of 1s and an end time of 3s. If feature A starts at 0 and ends at 1s, the new start time corresponding to feature A is 0.3s and the new end time is 0.8s. The change in start time is 0.3-0 = 0.3, which is 0.3s, and the end time is 1-0.8 = 0.2s, which is 0.2s. Therefore, the actual start time of action candidate region A is 1.2s, and the actual end time is 2.8s. If the action category of action candidate region A is running, the output of the action detection network includes: the action category of action candidate region A is running, the actual start time of action candidate region A is 1.2s, and the actual end time is 2.8s, achieving accurate detection and recognition of action segment boundaries in the video.

[0088] Optionally, step S280 includes the following steps:

[0089] Determine an action category loss value, a start time loss value, and an end time loss value according to the annotation information, the action category, the actual start time, and the actual end time;

[0090] When the action category loss value is less than a first threshold, the starting moment loss value is less than a second threshold, and the ending moment loss value is less than a third threshold, the training of the action boundary detection model is stopped to obtain the trained action boundary detection model.

[0091] Specifically, the annotation information includes the annotation category, annotation start time, and annotation end time corresponding to the action segment in the sample video. The first threshold, the second threshold, and the third threshold are all preset. After obtaining the action category, actual start time, and actual end time corresponding to the action candidate area, the annotation information and the action category, the actual start time, and the actual end time are input into the loss function to calculate the action category loss value between the annotation category and the action category, the start time loss value between the annotation start time and the actual start time, and the end time loss value between the annotation end time and the actual end time. If the action category loss value is greater than or equal to the first threshold, the start time loss value is greater than or equal to the second threshold, and the end time loss value is greater than or equal to the third threshold, it is considered that the error of the action boundary detection model exceeds the allowable range, that is, the error is large, and the training of the action boundary detection model continues. If the action category loss value is less than the first threshold, the start time loss value is less than the second threshold, and the end time loss value is less than the third threshold, it is considered that the error of the action boundary detection model is within the allowable range, that is, the error is very small, and the training of the action boundary detection model is stopped to obtain the trained action boundary detection model, that is, the training of the action boundary detection model is completed.

[0092] Optionally, the present invention also provides a terminal device, comprising: a memory, a processor, and a model training program stored in the memory and executable on the processor, wherein the model training program implements the steps of the above-mentioned model training method when executed by the processor.

[0093] Optionally, the present invention also provides a storage medium on which a model training program is stored, and when the model training program is executed by a processor, the steps of the above-mentioned model training method are implemented.

[0094] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0095] It should be noted that Figure 5 This is a structural diagram of the hardware operating environment of the terminal device.

[0096] As a way to implement this, Figure 5 As shown, the embodiment of the present invention relates to a terminal device, which includes: a processor 1001, such as a CPU, a memory 1002, and a communication bus 1003. The communication bus 1003 is used to implement connection and communication between these components.

[0097] The memory 1002 may be a high-speed RAX memory or a stable memory (non-volatile RAXory), such as a disk memory. Figure 5 As shown, the memory 1002 as a storage medium may include a model training program; and the processor 1001 may be used to call the model training program stored in the memory 1002 and perform the following operations:

[0098] Obtain video features and annotation information of sample videos;

[0099] Inputting the video features into a temporal action nomination network to obtain an action candidate region, where the action candidate region is a video segment between a start time and an end time predicted by the temporal action nomination network;

[0100] Input the video features into a temporal unidirectional feature processing module, and output the temporal unidirectional features corresponding to the action candidate area;

[0101] Inputting the starting time, the ending time, and the time series unidirectional feature into a linear interpolation sampling module, and outputting a feature to be identified, where the feature to be identified is a local feature intercepted by the linear interpolation sampling module from the time series unidirectional feature according to the starting time and the ending time;

[0102] Inputting the start time, the end time, and the feature to be identified into an action detection network, and outputting the action category, the actual start time, and the actual end time corresponding to the action candidate area;

[0103] The action boundary detection model is jointly trained according to the video features, the annotation information, the action category, the actual start time, and the actual end time to obtain the trained action boundary detection model.

[0104] Furthermore, the processor 1001 may be configured to call the model training program stored in the memory 1002 and perform the following operations:

[0105] Extracting, from the video features, a first image feature of each video frame corresponding to the action candidate region and a second image feature of a video frame preceding the video frame by the temporal unidirectional feature processing module;

[0106] The second image feature is unidirectionally fused with the first image feature to obtain the temporal unidirectional feature.

[0107] Furthermore, the processor 1001 may be configured to call the model training program stored in the memory 1002 and perform the following operations:

[0108] Identifying the to-be-identified features through the action detection network to obtain an action category of the action candidate area, a first change amount at the starting time, and a second change amount at the ending time;

[0109] Determining the actual starting time according to the starting time and the first change; and

[0110] The actual end time is determined according to the end time and the second change.

[0111] Furthermore, the processor 1001 may be configured to call the model training program stored in the memory 1002 and perform the following operations:

[0112] Determine an action category loss value, a start time loss value, and an end time loss value according to the annotation information, the action category, the actual start time, and the actual end time;

[0113] When the action category loss value is less than a first threshold, the starting moment loss value is less than a second threshold, and the ending moment loss value is less than a third threshold, the training of the action boundary detection model is stopped to obtain the trained action boundary detection model.

[0114] Furthermore, the processor 1001 may be configured to call the model training program stored in the memory 1002 and perform the following operations:

[0115] Extracting an image feature vector of each video frame in the sample video based on a pre-trained classification network;

[0116] The video feature is generated according to the image feature vector of each video frame.

[0117] According to the above technical solution, this embodiment improves the training speed and accuracy of the action boundary detection model.

[0118] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0123] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0124] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A detection system for action boundaries, characterized in that: An action boundary detection model is included, wherein the action boundary detection model includes: A temporal action nomination network, configured to predict an action candidate region based on video features, wherein the action candidate region is a video segment between a start time and an end time predicted by the temporal action nomination network; A temporal unidirectional feature processing module, configured to extract temporal unidirectional features corresponding to the action candidate area based on the video features; a linear interpolation sampling module, configured to intercept, based on the action candidate region, features to be identified in the temporal unidirectional features, wherein the features to be identified are local features intercepted by the linear interpolation sampling module from the temporal unidirectional features according to the start time and the end time; An action detection network is used to determine the action category, actual start time, and actual end time corresponding to the action segment based on the action candidate area and the features to be identified; The temporal action nomination network includes: Perimeter-sensitive networks; The nomination selection module is connected to the boundary-sensitive network, and the nomination selection module includes matrix multiplication and sorting filtering.

2. The system for detecting action boundaries according to claim 1, wherein: The temporal unidirectional feature processing module includes n temporal unidirectional convolutions connected in sequence, each of the temporal unidirectional convolutions has a hole coefficient, and the ratio between the hole coefficients of any two adjacent temporal unidirectional convolutions is the same.

3. A model training method, characterized in that: Applied to the training of the action boundary detection model according to any one of claims 1 to 2, the model training method comprising: Obtain video features and annotation information of sample videos; Inputting the video features into a temporal action nomination network to obtain an action candidate region, where the action candidate region is a video segment between a start time and an end time predicted by the temporal action nomination network; Input the video features into a temporal unidirectional feature processing module, and output the temporal unidirectional features corresponding to the action candidate area; Inputting the starting time, the ending time, and the time series unidirectional feature into a linear interpolation sampling module, and outputting a feature to be identified, where the feature to be identified is a local feature intercepted by the linear interpolation sampling module from the time series unidirectional feature according to the starting time and the ending time; Inputting the start time, the end time, and the feature to be identified into an action detection network, and outputting the action category, the actual start time, and the actual end time corresponding to the action candidate area; The action boundary detection model is jointly trained according to the video features, the annotation information, the action category, the actual start time, and the actual end time to obtain the trained action boundary detection model.

4. The model training method according to claim 3, wherein: The step of inputting the video features into a temporal unidirectional feature processing module and outputting the temporal unidirectional features corresponding to the action candidate regions comprises: Extracting, from the video features, a first image feature of each video frame corresponding to the action candidate region and a second image feature of a video frame preceding the video frame by the temporal unidirectional feature processing module; The second image feature is unidirectionally fused with the first image feature to obtain the temporal unidirectional feature.

5. The model training method according to claim 3, wherein: The step of inputting the start time, the end time, and the features to be identified into the action detection network and outputting the action category, the actual start time, and the actual end time corresponding to the action candidate area comprises: Identifying the to-be-identified features through the action detection network to obtain an action category of the action candidate area, a first change amount at the starting time, and a second change amount at the ending time; Determining the actual starting time according to the starting time and the first change; and The actual end time is determined according to the end time and the second change.

6. The model training method according to claim 3, wherein: The step of jointly training the action boundary detection model according to the video features, the annotation information, the action category, the actual start time, and the actual end time to obtain the trained action boundary detection model comprises: Determine an action category loss value, a start time loss value, and an end time loss value according to the annotation information, the action category, the actual start time, and the actual end time; When the action category loss value is less than a first threshold, the starting moment loss value is less than a second threshold, and the ending moment loss value is less than a third threshold, the training of the action boundary detection model is stopped to obtain the trained action boundary detection model.

7. The model training method according to claim 3, wherein: Before the step of obtaining the video features and annotation information of the sample video, the method further includes: Extracting an image feature vector of each video frame in the sample video based on a pre-trained classification network; The video feature is generated according to the image feature vector of each video frame.

8. A terminal device, characterized in that: The terminal device includes: a memory, a processor, and a model training program stored in the memory and executable on the processor. When the model training program is executed by the processor, the steps of the model training method as described in any one of claims 3 to 7 are implemented.

9. A storage medium, characterized in that: A model training program is stored thereon, and when the model training program is executed by a processor, the steps of the model training method described in any one of claims 3 to 7 are implemented.

Images