Action segmentation model processing method, device, computer equipment and storage medium

Abstract

The present application relates to a processing method, device, computer equipment and storage medium for an action segmentation model, and specifically to computer vision technology of artificial intelligence, including: performing feature enhancement processing on a feature vector sequence of a sample video to obtain an enhanced feature sequence; dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into multiple logarithmic context levels, obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predicting the boundary classification result corresponding to each video frame according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence; predicting the action classification result corresponding to each video frame according to the enhanced feature sequence; and continuing training after updating the model parameters based on the action category label, boundary classification result and action classification result of each video frame until a trained action segmentation model is obtained. The scheme provided in the embodiment of the present application can improve the accuracy of action classification of video frames near the action boundary.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and storage medium for processing an action segmentation model. Background Art

[0002] Video understanding and recognition is one of the basic tasks of computer vision. With the rapid development of smart terminals and Internet technology, the number of videos uploaded to various platforms has also increased rapidly, making the understanding of video content receive more and more attention in recent years.

[0003] Video action segmentation is the process of segmenting videos containing multiple actions by identifying different action categories in the video. Video action segmentation is a video understanding task with high practical application value. For example, accurate action segmentation helps public opinion monitoring, advertising placement, and many other video understanding-related tasks.

[0004] However, when identifying different action categories in videos, since the action boundary is located in the transition stage between different actions, the frames of adjacent actions at the boundary are very similar. Most of the existing action segmentation methods are insensitive to action boundaries, resulting in low action classification accuracy for video frames near the action boundary. Summary of the invention

[0005] Based on this, it is necessary to provide a processing method, device, computer equipment and storage medium for an action segmentation model that can improve the accuracy of classifying video frame actions near action boundaries in a video in order to address the above technical problems.

[0006] A method for processing an action segmentation model, comprising:

[0007] Obtaining a sample video and action category labels of video frames in the sample video;

[0008] Performing feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence;

[0009] Dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into a plurality of logarithmic context levels, obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predicting the boundary classification result corresponding to each of the video frames according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence;

[0010] Predicting the action classification result corresponding to each of the video frames according to the enhanced feature sequence;

[0011] After updating the model parameters based on the action category labels of each video frame, the boundary classification results and the action classification results, training is continued until a trained action segmentation model is obtained. The action segmentation model is used to predict the action category corresponding to each video frame in the video to be processed, and then obtain video clips segmented by action category.

[0012] In one embodiment, the method further comprises:

[0013] In the feature extraction network, extracting feature vectors from each video frame constituting the sample video;

[0014] A feature vector sequence corresponding to the sample video is obtained according to the feature vector of the video frame.

[0015] In one embodiment, the step of performing feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence includes:

[0016] In a multi-stage feature enhancement network, feature enhancement processing is performed on a previous action classification result corresponding to a previous stage to obtain an enhanced feature sequence corresponding to a current stage; wherein the previous action classification result is obtained by performing action classification on a previous enhanced feature sequence corresponding to the previous stage, and the first enhanced feature sequence is generated based on a feature vector sequence of the sample video.

[0017] In one embodiment, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence are divided into multiple logarithmic context levels, the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into the same level, and the boundary classification result corresponding to each of the video frames is predicted according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence, including:

[0018] Divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence corresponding to the current stage into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict the boundary classification result corresponding to each of the video frames at the current stage according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage;

[0019] The predicting the action classification result corresponding to each of the video frames according to the enhanced feature sequence includes: predicting the action classification result corresponding to each of the video frames at the current stage based on the enhanced feature sequence corresponding to the current stage.

[0020] In one embodiment, the step of performing feature enhancement processing on the previous action classification result corresponding to the previous stage to obtain an enhanced feature sequence corresponding to the current stage includes:

[0021] Inputting the feature vector sequence into the multi-stage feature enhancement network, wherein the multi-stage feature enhancement network includes a feature generation sub-network and a plurality of feature refinement sub-networks;

[0022] In the feature generation subnetwork, feature enhancement processing is performed on the feature vector sequence to obtain an enhanced feature sequence corresponding to the sample video in the feature generation stage;

[0023] The predicting, based on the enhanced feature sequence corresponding to the current stage, the action classification result corresponding to each of the video frames at the current stage includes:

[0024] Action classification is performed according to the enhanced feature sequence corresponding to the feature generation stage to obtain the action classification result corresponding to the sample video in the feature generation stage.

[0025] In one embodiment, the method further comprises:

[0026] In the first feature refinement subnetwork, feature enhancement processing is performed on the action classification results corresponding to the feature generation stage, and after obtaining the enhanced feature sequence corresponding to the sample video at the first feature refinement stage, the enhanced feature sequence corresponding to the sample video at the current feature refinement stage is iteratively obtained according to the action classification results corresponding to the previous feature refinement stage, until the enhanced feature sequence corresponding to the last feature refinement stage is obtained and the iteration is stopped;

[0027] Action classification is performed according to the enhanced feature sequence corresponding to the current feature refinement stage to obtain the action classification result corresponding to the sample video at the current feature refinement stage.

[0028] In one embodiment, predicting the action classification result corresponding to each of the video frames at the current stage based on the enhanced feature sequence corresponding to the current stage includes:

[0029] The enhanced feature vectors corresponding to each video frame in the enhanced feature sequence corresponding to the current stage are subjected to action classification by a linear classifier to obtain the action classification results corresponding to each video frame at the current stage.

[0030] In one embodiment, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence corresponding to the current stage are divided into multiple logarithmic context levels, and the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into the same level, including:

[0031] For each video frame in the sample video, according to the relative direction and distance to the adjacent frames, each enhanced feature vector in the enhanced feature sequence corresponding to the current stage and in the logarithmic context is divided into a plurality of logarithmic context levels;

[0032] For each logarithmic context level, the average vector of the enhanced feature vectors divided into the same level is calculated, and the sequence formed by the average vectors of each logarithmic context level is used as the logarithmic context feature sequence corresponding to each of the video frames at the current stage.

[0033] In one embodiment, predicting the boundary classification result corresponding to each of the video frames at the current stage according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage includes:

[0034] Determining the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage;

[0035] Obtaining a logarithmic context attention score corresponding to each of the video frames according to the similarity;

[0036] Through a linear classifier, the boundary classification result corresponding to each of the video frames at the current stage is predicted according to the logarithmic context attention score.

[0037] In one embodiment, determining the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage includes:

[0038] Performing a linear transformation on the enhanced feature sequence corresponding to the current stage to obtain a feature transformation sequence;

[0039] Performing a linear transformation on the logarithmic context feature sequence to obtain a context transformation feature sequence;

[0040] The similarity between each feature transformation vector in the feature transformation sequence and the corresponding context feature transformation vector in the context transformation feature sequence is obtained through a dot product operation.

[0041] In one embodiment, the updating of the model parameters according to the action category label of each video frame, the boundary classification result and the action classification result and then continuing the training until a trained action segmentation model is obtained includes:

[0042] Determining a corresponding boundary category label according to the action category label of each of the video frames;

[0043] Constructing a total loss function according to the boundary classification result and the boundary category label, the action classification result and the action category label;

[0044] After updating the model parameters according to the total loss function, training is continued until a trained action segmentation model is obtained.

[0045] In one embodiment, constructing a total loss function according to the boundary classification result and the boundary category label, the action classification result and the action category label includes:

[0046] Constructing a boundary classification loss function for all stages based on the boundary classification results corresponding to each stage and the boundary category labels;

[0047] Constructing the action classification loss function of all stages based on the action classification results corresponding to each stage and the action category labels;

[0048] The boundary classification loss function of all stages and the action classification loss function of all stages are summed to obtain a total loss function.

[0049] In one embodiment, the action segmentation model includes a feature extraction network for extracting features from the sample video, a multi-stage feature enhancement network for performing feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence, a boundary classification network for dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into multiple logarithmic context levels, obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predicting the boundary classification results corresponding to each of the video frames according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence, and an action classification network for predicting the action classification results corresponding to each of the video frames according to the enhanced feature sequence.

[0050] In one embodiment, the method further comprises:

[0051] Inputting the video to be processed into the trained action segmentation model;

[0052] The feature vector sequence corresponding to the video to be processed is obtained through the feature extraction network in the action segmentation model. The feature vector sequence of the sample video is subjected to feature enhancement processing through the multi-stage feature enhancement network in the action segmentation model to obtain the enhanced feature sequence corresponding to the final stage. The action classification network in the action segmentation model predicts according to the enhanced feature sequence corresponding to the final stage to obtain the action category corresponding to each video frame in the video to be processed.

[0053] In one embodiment, the method further comprises:

[0054] Obtaining the action category corresponding to each of the video frames in the video to be processed;

[0055] The video to be processed is divided into a plurality of video segments according to video frames corresponding to the same action category.

[0056] In one embodiment, the method further comprises:

[0057] The boundary classification network in the action segmentation model is used to predict the boundary category corresponding to each video frame in the video to be processed according to the enhanced feature sequence corresponding to the final stage.

[0058] A video processing method, comprising:

[0059] Get the video to be processed;

[0060] Inputting the video to be processed into a trained action segmentation model;

[0061] Obtaining a feature vector sequence corresponding to the video to be processed through the action segmentation model, performing feature enhancement processing on the feature vector sequence of the video to be processed to obtain an enhanced feature sequence, and performing prediction based on the enhanced feature sequence to obtain the action category corresponding to each video frame in the video to be processed;

[0062] Among them, the action segmentation model is obtained by model training based on the sample video and the action category labels of each video frame in the sample video, boundary classification results and action classification results; the boundary classification result is to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence of the sample video into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict it according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence of the sample video; the action classification result is predicted based on the enhanced feature sequence of the sample video; the enhanced feature sequence of the sample video is obtained by feature enhancement processing of the feature vector sequence of the sample video.

[0063] A processing device for an action segmentation model, comprising:

[0064] An acquisition module, used to acquire a sample video and action category labels of video frames in the sample video;

[0065] A feature enhancement module, used for performing feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence;

[0066] A boundary classification module, used to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict the boundary classification result corresponding to each of the video frames according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence;

[0067] An action classification module, used for predicting the action classification result corresponding to each of the video frames according to the enhanced feature sequence;

[0068] A parameter updating module is used to update the model parameters based on the action category label of each video frame, the boundary classification result and the action classification result, and then continue training until a trained action segmentation model is obtained. The action segmentation model is used to predict the action category corresponding to each video frame in the video to be processed, and then obtain video clips segmented by action category.

[0069] A video processing device, comprising:

[0070] An acquisition module is used to acquire the video to be processed;

[0071] An input module, used for inputting the video to be processed into a trained action segmentation model;

[0072] An action segmentation module is used to obtain a feature vector sequence corresponding to the video to be processed through the action segmentation model, perform feature enhancement processing on the feature vector sequence of the video to be processed to obtain an enhanced feature sequence, and perform prediction based on the enhanced feature sequence to obtain an action category corresponding to each video frame in the video to be processed;

[0073] Among them, the action segmentation model is obtained by model training based on the sample video and the action category labels of each video frame in the sample video, boundary classification results and action classification results; the boundary classification result is to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence of the sample video into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict it according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence of the sample video; the action classification result is predicted based on the enhanced feature sequence of the sample video; the enhanced feature sequence of the sample video is obtained by feature enhancement processing of the feature vector sequence of the sample video.

[0074] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements at least one of the above-mentioned action segmentation model processing method and the above-mentioned video processing method when executing the computer program.

[0075] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements at least one of the above-mentioned action segmentation model processing method and the above-mentioned video processing method.

[0076] A computer program, the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium, the processor of a computer device reading the computer instructions from the computer-readable storage medium, the processor executing the computer instructions, so that the computer device executes at least one of the steps of the above-mentioned action segmentation model processing method and the above-mentioned video processing method.

[0077] The processing method, device, computer equipment and storage medium of the above-mentioned action segmentation model, when training the action segmentation model, predict the boundary category of each video frame in the sample video to assist the action classification of the video frame, and train the frame-level boundary prediction and frame-level action classification together, so that the action segmentation model can learn more discriminative features of the video frame at the boundary, thereby improving the accuracy of action classification of video frames near the action boundary.

[0078] In addition, when predicting the boundary category of a video frame, since the boundary category is irrelevant to the specific action category, it is difficult to directly predict whether the video frame is a boundary frame based on the enhanced features of the video frame. Instead, the prediction is made by using the similarity between each video frame and its adjacent continuous frames. Specifically, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence are divided into multiple logarithmic context levels, and the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into different levels. In this way, the context information of each video frame in the logarithmic space can be captured, and boundary classification is performed according to the similarity between the enhanced feature vector of each video frame and its logarithmic context feature, which can greatly improve the accuracy of boundary classification. Moreover, by using the context information in the logarithmic space, the context information receptive field range of each video frame can be greatly increased without increasing the computational complexity of calculating the similarity between each video frame and its nearby video frames, thereby overcoming the problem of difficulty in balancing the context range and the similarity computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is an application environment diagram of a method for processing an action segmentation model in one embodiment;

[0080] Figure 2 is a flowchart of a method for processing an action segmentation model in one embodiment;

[0081] Figure 3 A schematic diagram of boundary classification of a video in one embodiment;

[0082] Figure 4 is a schematic diagram of a logarithmic context in one embodiment;

[0083] Figure 5 A schematic diagram of a process of performing boundary prediction at each stage in an embodiment;

[0084] Figure 6 A schematic diagram of a process of performing boundary prediction based on logarithmic context similarity in one embodiment;

[0085] Figure 7 A schematic diagram of performing boundary classification based on logarithmic context in one embodiment;

[0086] Figure 8 A schematic diagram of the structural framework of an action segmentation model in one embodiment;

[0087] Fig. 9 Schematic diagram of the overall framework of an action segmentation model in one embodiment;

[0088] Fig.10 is a flowchart of a method for processing an action segmentation model in a specific embodiment;

[0089] Fig.11 is a schematic flow chart of a video processing method in one embodiment;

[0090] Fig.12 is a structural block diagram of a processing device for an action segmentation model in one embodiment;

[0091] Fig.13 is a structural block diagram of a video processing device in one embodiment;

[0092] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0094] The processing method of the image segmentation model and the image processing method provided in the embodiments of the present application involve artificial intelligence (AI) technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0095] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0096] The processing method and video processing method of the action segmentation model provided in the embodiment of the present application mainly relate to the computer vision technology (Computer Vision, CV) of artificial intelligence. Computer vision is a science that studies how to make a machine "see". Further, it refers to the machine vision such as using a camera and a computer to replace the human eye to identify, track and measure the target, and further do graphic processing to make the computer processing become an image more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning and map construction, etc., and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0097] The processing method of the action segmentation model and the video processing method provided in the embodiments of the present application mainly relate to video understanding in the field of computer vision technology. For example, after the action segmentation model capable of identifying the action category in the video is trained through sample videos, the video to be processed is input into the trained action segmentation model, and the action category of each video frame in the video to be processed is identified through the action segmentation model. Furthermore, the video to be processed can be segmented into video clips corresponding to different actions according to the action category of each video frame.

[0098] The processing method of the action segmentation model provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The server 104 can be used to obtain a video from the terminal 102 as a sample video, and train the action segmentation model through the sample video. Specifically, the server 104 can obtain the sample video and the action category label of the video frame in the sample video; perform feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence; divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict the boundary classification result corresponding to each video frame according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence; predict the action classification result corresponding to each video frame according to the enhanced feature sequence; update the model parameters based on the action category label, boundary classification result and action classification result of each video frame and continue training until a trained action segmentation model is obtained.

[0099] The processing method of the above-mentioned action segmentation model, when training the action segmentation model, predicts the boundary category of each video frame in the sample video to assist the action classification of the video frame, and trains the frame-level boundary prediction and frame-level action classification together, so that the action segmentation model can learn more discriminative features of the video frame at the boundary, thereby improving the accuracy of action classification of video frames near the action boundary.

[0100] In addition, when predicting the boundary category of a video frame, since the boundary category is irrelevant to the specific action category, it is difficult to directly predict whether the video frame is a boundary frame based on the enhanced features of the video frame. Instead, the prediction is made by using the similarity between each video frame and its adjacent continuous frames. Specifically, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence are divided into multiple logarithmic context levels, and the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into different levels. In this way, the context information of each video frame in the logarithmic space can be captured, and boundary classification is performed according to the similarity between the enhanced feature vector of each video frame and its logarithmic context feature, which can greatly improve the accuracy of boundary classification. Moreover, by using the context information in the logarithmic space, the context information receptive field range of each video frame can be greatly increased without increasing the computational complexity of calculating the similarity between each video frame and its nearby video frames, thereby overcoming the problem of difficulty in balancing the context range and the similarity computational complexity.

[0101] The video processing method provided in this application can also be applied to Figure 1In the application environment shown. For example, the server 104 can obtain the video to be processed, input the video to be processed into a trained action segmentation model; obtain the feature vector sequence corresponding to the video to be processed through the action segmentation model, perform feature enhancement processing on the feature vector sequence of the video to be processed, obtain an enhanced feature sequence, and perform prediction based on the enhanced feature sequence to obtain the action category corresponding to each video frame in the video to be processed. The server 104 can also perform prediction based on the enhanced feature sequence to obtain the boundary category corresponding to each video frame in the video to be processed. The server 104 can also return the video clips obtained after segmenting the video to be processed according to the action category to the terminal 102. Figure 1 The input long video is a dance video showing three different dance moves. The server 104 can predict the dance move category corresponding to each video frame through the trained move segmentation model, and then segment the long video according to the dance move category to obtain three video clips. In some scenarios, users who need to learn the dance can divide the dance moves and learn them in sequence, avoiding the need to watch the long video repeatedly. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented as an independent server or a server cluster composed of multiple servers.

[0102] The processing method of the action segmentation model provided in the embodiment of the present application may be executed by the processing device of the action segmentation model provided in the embodiment of the present application, or a computer device integrated with the processing device of the action segmentation model, wherein the processing device of the action segmentation model may be implemented in hardware or software. The video processing method provided in the embodiment of the present application may be executed by the video processing device provided in the embodiment of the present application, or a computer device integrated with the video processing device, wherein the video processing device may be implemented in hardware or software. The computer device may be Figure 1 The terminal 102 or server 104 shown in .

[0103] Action segmentation refers to the process of segmenting the long video according to the action category after identifying the action category of each video frame in the long video. The processing method of the action segmentation model and the video processing method provided in this application can be applied to video surveillance, video public opinion monitoring, advertising, robot navigation and many other scenarios related to video understanding. For example, in a video surveillance scenario, the video can be analyzed to find out the video clips that are not in compliance with the law, so as to prompt the user to edit some clips in the video or remove the video from the shelves. For example, in the scene of a football match video, the action segmentation model can be applied to analyze the football match video, and the wonderful video clips such as "goal" and "penalty card" that appear less frequently in the long video can be segmented to obtain the essence of the football match video. For another example, in a human-machine collaboration scenario, the action segmentation model can be applied to analyze the human action behavior, identify the human action and behavior, and feed it back to the robot, so that the human-machine collaboration is smoother.

[0104] In one embodiment, Figure 2 As shown, a method for processing an action segmentation model is provided, and the method is applied to Figure 1 Computer equipment in Figure 1 The terminal or server in the example is used to illustrate, including the following steps:

[0105] Step 202: Obtain a sample video and action category labels of video frames in the sample video.

[0106] The sample video is used to train the action segmentation model, and the action category label of the video frame is the real action category corresponding to the video frame. The action category is the action classification of each video frame in the video. For example, the actions in a long video about squeezing orange juice can be divided into four categories, namely cutting oranges, squeezing orange juice, taking a cup and pouring orange juice. The action category label of the video frame can be obtained by manual annotation. In an embodiment of the present application, the action segmentation model can be learned through sample videos, so as to learn the ability to classify the actions of each video frame in the video.

[0107] Specifically, the computer device can download sample videos from the network, and the computer device can also obtain sample videos transmitted by other computer devices, such as the above Figure 1 In the example, the server 104 obtains the video transmitted by the terminal 102, and the computer device can also obtain the sample video generated on the local computer, for example, by performing a series of video editing or clipping on the existing video to generate a new sample video. In addition, when the computer device obtains the sample video, it also obtains the corresponding action category label. For example, the sample video includes T video frames, and the corresponding action category label can be a vector with a length of T, which is c = [c1, c2, c3, ..., cT ].

[0108] Step 204: perform feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence.

[0109] The feature vector is a representation of the deep features of the video frame obtained by abstractly extracting and combining the image features of the video frame. The image features include the color features, texture features, shape features, etc. of the video frame. The feature vector sequence is a sequence composed of the feature vectors of each video frame in the sample video. The feature vector sequence is the initial feature representation of the sample video. In order to enhance the feature representation of the sample video and obtain the connection between each video frame in the sequential order, the computer device can perform feature enhancement processing on the feature vector sequence to obtain an enhanced feature sequence that can characterize the temporal relationship between each video frame.

[0110] For example, the feature vector sequence X of the sample video is [x1, x2, x3, ..., x T ]∈R d×T , where T is the length of the feature vector sequence, feature vector sequence X = [x1, x2, x3, ..., x T ]∈R d×T The tth eigenvector x in t represents the feature vector corresponding to the tth frame in the sample video, and d is the dimension of the feature vector corresponding to the video frame. Perform feature enhancement on the feature vector sequence to obtain Z = [z1, z2, z3, ..., z T ]∈R d′×T , where d' is the dimension of the enhanced feature vector.

[0111] In one embodiment, the computer device can extract feature vectors from each video frame constituting the sample video in a feature extraction network; and obtain a feature vector sequence corresponding to the sample video based on the feature vectors of the video frames.

[0112] Specifically, the action segmentation model may include a feature extraction network for extracting a feature vector sequence of a video. The feature extraction network may be a pre-trained convolutional neural network or a convolutional neural network trained together with other networks in the action segmentation model.

[0113] In one embodiment, the feature extraction network is a 2D convolutional neural network, and the computer device extracts a feature vector from each video frame of the sample video through the feature extraction network, and obtains a feature vector sequence according to the feature vector corresponding to each video frame.

[0114] In one embodiment, the feature extraction network is a 3D convolutional neural network, such as I3D, C3D, etc. The computer device extracts a feature from several video frames before and after the current frame through the feature extraction network as the feature vector of the current frame, and obtains a feature vector sequence according to the feature vector corresponding to each video frame. For example, the sample video has a total of 8 frames, and the computer device can extract the first feature according to the 1st-3rd frames as the feature vector of the 1st frame, extract the second feature according to the 2nd-4th frames as the feature vector of the 2nd frame, extract the third feature according to the 3rd-5th frames as the feature vector of the 3rd frame, and so on. Optionally, if the length of the generated feature vector sequence is to be kept consistent with the length of the sample video, the feature vector of each frame can be generated after 0 is added. For example, the computer device extracts the first feature vector according to 0 and the 1st-2nd frames, extracts the second feature vector according to the 1st-3rd frames, and so on, and extracts the 8th feature vector according to the 7th-8th frames and 0.

[0115] After obtaining the feature vector sequence corresponding to the sample video, the computer device can perform feature enhancement processing on the feature vector sequence to obtain an enhanced feature sequence. The computer device can use the temporal dependency between video frames to enhance the feature representation of each video frame in the feature vector sequence. For example, a temporal convolutional neural network can be used to build a feature enhancement network to enhance the feature representation of each video frame, such as MS-TCN++ (Multi-Stage Temporal Convolutional Network), Spatial CNN (Spatial Convolutional Neural Network), Bi-LSTM (Bi-directional Long Short-Term Memory), and the like.

[0116] In step 206, the enhanced feature vectors in the logarithmic context of the enhanced feature sequence are divided into multiple logarithmic context levels, and the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into the same level, and the boundary classification result corresponding to each video frame is predicted according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence.

[0117] Boundary classification is to predict whether the current frame is a boundary frame. Boundary frames are video frames that transition between different action categories, such as the start frame or end frame of consecutive actions of the same category. The video frames between the start frame and the end frame are called intermediate frames. The boundary classification result is to classify the video frame into one of these three categories. Traditional action segmentation methods treat all video frames equally when classifying actions in video frames. They do not distinguish between boundary frames between two different action categories. Video frames near the boundary are very similar, resulting in low classification accuracy for video frames at the boundary.

[0118] like Figure 3 FIG. 1 is a schematic diagram of a video boundary classification in one embodiment. Figure 3 ,Video frames of four consecutive actions in the same video. Each row represents an action. The first and last columns show the start and end frames of the four actions respectively. The middle two columns show the middle frames of the actions. In each action, the middle frames and boundary frames are different. However, the start frame of each action is very similar to the end frame of the previous action, and the end frame of each action is very similar to the start frame of the next action, making it difficult to classify.

[0119] Based on this, when training the action segmentation model in the embodiment of the present application, the boundary category of each video frame in the sample video is predicted, that is, whether the video frame is the starting frame, middle frame or ending frame of a certain action category is predicted to assist in the action classification of the video frame. The frame-level boundary prediction and frame-level action classification are trained together. Adding the boundary classification task can force the action segmentation model to pay more attention to the frames around the boundary, learn the more discriminative features of the video frame at the boundary, and improve the accuracy of action classification of video frames near the action boundary.

[0120] Since the boundary type is irrelevant to the specific action category, the start frame of each action is very similar to the end frame of the previous action, and the end frame of each action is very similar to the start frame of the next action, it is difficult to directly predict whether the video frame is a boundary frame based on the enhanced feature vector of the video frame. Figure 3 , considering that if a video frame is similar to its subsequent frame but different from the previous frame, it may be the start frame of a new action; if it is different from the subsequent frame but similar to the previous frame, it should be the end frame; if it is similar to adjacent frames in both directions, it may be in the middle of the action and is an intermediate frame. Therefore, when predicting the boundary category of a video frame, the similarity between the enhanced feature vector of the video frame and its adjacent continuous frames is used instead of the enhanced feature vector of the video frame itself as the basis for boundary classification.

[0121] For example, for the enhanced feature sequence Z = [z1, z2, z3, ..., z T ]∈R d′×T , where zt is the enhanced feature vector corresponding to the t-th frame. Assuming that the context range is the two frames before and after, the computer device can calculate the similarity between the t-th frame and the adjacent frames based on the enhanced feature vectors corresponding to the t-2th frame, the t-1th frame, the t+1th frame, the t+2th frame and the enhanced feature vector corresponding to the t-th frame when predicting the action category of the t-th frame, and predict the boundary category of the t-th frame based on the similarity.

[0122] In the present application, the inventors further realized that when calculating the similarity between the enhanced feature vector of a video frame and its adjacent continuous frames, if the above example is adopted and only the enhanced feature vector of each video frame is compared with the enhanced feature vectors of several adjacent frames, the local receptive field of each video frame will be limited and the context information cannot be fully explored. If the number of selected adjacent frames is large and the similarity between the video frame and these selected adjacent frames is calculated, the large range will not only introduce a lot of computational complexity, but the large range context is more likely to include noise frames, and boundary prediction is also more likely to be affected by noise frames.

[0123] To this end, a computer device divides the enhanced feature vectors in the logarithmic context of the enhanced feature sequence into multiple logarithmic context levels, and obtains the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into different levels. In this way, the context information of each video frame can be captured in the logarithmic space, and boundary classification is performed according to the similarity between the enhanced feature vector of each video frame and its logarithmic context feature, which can greatly improve the accuracy of boundary classification. At the same time, the use of context information in the logarithmic space can greatly increase the receptive field range of the context information of each video frame without increasing the computational complexity of calculating the similarity between each video frame and its nearby video frames, thereby overcoming the problem of difficulty in balancing the context range and the similarity computational complexity.

[0124] The logarithmic context is the context information obtained by dividing the context of the video frame according to the logarithmic distance from the video frame. The range of the logarithmic context is determined by the hyperparameters set during model training. The hyperparameters include N, d1, and f, where N is the number of levels in each direction, d1 is the number of video frames in the closest level in the front and back directions, and f is the growth factor of the number of frames in the level as the level increases. For each video frame, the number of frames included in the logarithmic context is the same, which is 2d1f N-1video frames, but since the position of each video frame is different, the corresponding video frames in the logarithmic context are different. The logarithmic context level refers to dividing the features in the logarithmic context into multiple levels. When dividing, the number of video frames contained in each level increases exponentially with the increase of the level. In other words, the range of the logarithmic context increases exponentially with the linear increase of the level, and more context information can be obtained. In this way, for the current frame, greater importance can be assigned to adjacent frames and less importance can be assigned to video frames that are farther away, thereby reducing the impact of noise frames or abnormal frames.

[0125] Specifically, for the tth frame, the enhanced feature sequence Z = [z1, z2, z3, ..., z T ]∈R d′×T The tth enhanced feature vector z in t is the enhanced feature vector corresponding to the t-th frame. The index of the current t-th frame is set to 0, and the indexes of the forward adjacent frames are 1, 2, 3, ..., d1f N-1 , the indexes of the backward adjacent frames are -1, -2, -3, ..., -d1f N-1 . The d1 neighboring frames of the t-th frame are divided into the first level of the forward direction, that is, the 1st frame to the d1th frame of the t-th frame are divided into the first level of the forward direction, and the d1+1th frame to the d1fth frame of the t-th frame are divided into the second level of the forward direction; similarly, for n from 3 to N, the d1fth frame is divided into the second level of the forward direction. n-2 +1 frame to d1f n-1 Frames are divided into the nth level in the forward direction; it can be seen that the farther away from the current frame, the more frames there are in the level, and the indexes of the end frames in the N levels are equidistant in logarithmic space, that is:

[0126] log(d1f n )-log(d1f n-1 )=log(f).

[0127] Similarly, backward d1f N-1 The adjacent frames are also divided into N upper and lower logarithmic text levels. The farther away from the current frame, the more frames there are in the level, and the indexes of the reverse end frames of each level are equidistant in the logarithmic space.

[0128] For example, assuming that d1 takes the value of 2, f takes the value of 2, and N takes the value of 4, for the current frame z3: the 1st frame to the 2nd frame of the frame, that is, the two forward frames, that is, z4 to z5, are divided into the first forward level, the 3rd frame to the 4th frame of the frame, that is, the two forward frames, that is, z6 to z7, are divided into the second forward level; the 5th frame to the 8th frame of the frame, that is, the four forward frames, that is, z8 to z11, are divided into the third forward level; the 9th frame to the 16th frame of the frame, that is, the eight forward frames, that is, z12 to z19, are divided into the fourth forward level; the indexes of the end frames in each level are 2, 4, 8, and 16, respectively, which are equidistant in the logarithmic space.

[0129] Similarly, the 1st to 2nd frames backward of the frame, that is, the two backward frames, that is, z1 to z2 are divided into the 1st backward level, the 3rd to 4th frames backward of the frame are divided into the 2nd backward level, the 5th to 8th frames backward of the frame are divided into the 3rd backward level, and the 9th to 16th frames backward of the frame are divided into the 4th backward level. However, since there are only two frames z1 and z2 backward of the current frame z3, the frames of the 2nd backward level, the 3rd backward level and the 4th backward level can be padded with 0.

[0130] like Figure 4 FIG. 1 is a schematic diagram of a logarithmic context in one embodiment. Figure 4 , the enhanced feature sequence includes enhanced feature vectors corresponding to 25 video frames, and the enhanced feature vectors in the logarithmic context of the current frame in the enhanced feature sequence are divided into multiple logarithmic context levels. Taking N=3, d1=2, f=2, and the current frame as the 11th frame as an example, the 8 forward and backward frames are divided into 3 different levels respectively. Figure 4 , the index of the current frame is marked as 0. In the logarithmic space, the index of the end frame of the level in each direction is equidistant. As the number of levels increases, the number of video frames included in the logarithmic context increases exponentially.

[0131] Next, the computer device obtains the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors classified into the same level, and predicts the boundary classification result corresponding to each video frame according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence.

[0132] The logarithmic context feature sequence is a sequence of logarithmic context features corresponding to each video frame in the sample video, and the logarithmic context feature corresponding to each video frame is a feature obtained by encoding the enhanced feature vector in each level of the logarithmic context of the video frame.

[0133] Specifically, the computer device can calculate the average vector based on the enhanced feature vectors divided into the same level, and obtain the context features corresponding to the current frame based on the average vector corresponding to each level, so as to obtain the context features corresponding to each video frame in the sample video. The similarity between the logarithmic context feature sequence and the enhanced feature sequence includes the similarity between the logarithmic context features and the enhanced feature vectors corresponding to each video frame in the sample video. In this way, based on the similarity between the frame features and the context of each video frame, the boundary classification results corresponding to each video frame can be obtained. The obtained boundary classification results can be expressed by B=[b1, b2, b3, ..., b T ]∈R 3×T , the tth boundary classification vector b in the boundary classification result t is the probability that the predicted t-th frame belongs to the start frame, middle frame, or end frame.

[0134] For example, at the tth frame, according to the logarithmic context construction method, according to the input hyperparameters, the logarithmic context of the frame is obtained, and for each logarithmic context level, the average value of the enhanced feature vector falling into the level is calculated to obtain the average vector of the 2N levels in the logarithmic context The value range of n includes -N, ..., -1, 1, ..., N. The sequence composed of these 2N average vectors is used as the context feature of the t-th frame. By obtaining the enhanced feature vector z of the t-th frame t Respectively with 2N average vectors The similarity between the constructed context features can be used to classify the boundary category of the t-th frame and obtain the boundary classification result corresponding to the t-th frame. Figure 4 , the computer device can calculate the average vector of the enhanced feature vectors in the forward first level based on the forward first frame and the second frame, calculate the average vector of the enhanced feature vectors in the forward second level based on the forward third frame to the fourth frame, and so on.

[0135] Step 208: predicting the action classification result corresponding to each video frame according to the enhanced feature sequence.

[0136] Specifically, after obtaining the enhanced feature vector corresponding to each video frame in the sample video, the computer device performs action classification on each video frame based on the enhanced feature vector corresponding to each video frame to obtain the action classification result corresponding to each video frame. The computer device can apply the linear classifier of action classification to the enhanced feature vector corresponding to each video frame to perform frame-by-frame action classification, and the obtained action classification result Where C is the number of action categories, and the t-th action classification vector y in the action classification result t is the probability of belonging to each action category in the predicted t-th frame.

[0137] Step 210 , after updating the model parameters based on the action category label, boundary classification result and action classification result of each video frame, the training is continued until a trained action segmentation model is obtained.

[0138] In the process of training the model, the action classification task and the boundary classification task are jointly trained, so that the action segmentation model can learn more discriminative features of different categories of video frames near the boundary through the training process, so as to improve the action classification accuracy of the frames around the boundary, thereby improving the performance of the action segmentation task. After the computer device processes the sample video according to the aforementioned steps through the action segmentation model to obtain the boundary classification results and action classification results corresponding to each video frame in the sample video, it updates the model parameters based on the action category label, boundary classification results and action classification results of each video frame and continues training until the training stop condition is met, and a trained action segmentation model is obtained.

[0139] In one embodiment, when the computer device obtains the action category label of the sample video, it can also obtain the boundary category label of the manually annotated sample video.

[0140] In one embodiment, the computer device can determine the corresponding boundary category label based on the action category label of each video frame, rather than directly obtaining the boundary category label of the sample video. Specifically, if a video frame is similar to its subsequent frame but different from the previous frame, it may be the start frame of a new action. If it is different from the subsequent frame but similar to the previous frame, it should be the end frame. If it is similar to adjacent frames in both directions, it may be in the middle of the action and is an intermediate frame. Therefore, the computer device can determine the boundary category label corresponding to the current frame by obtaining whether the category labels between the current frame and the previous and next frames are the same based on the action category label corresponding to the sample video.

[0141] For example, the boundary category is defined as 1 for the start frame, 2 for the middle frame, and 3 for the end frame. Assume that the sample video includes 9 video frames, and the action categories are divided into 4 categories, and the action category labels are [4 4 3 3 3 1 1 2 2]. Among them, the first frame is obviously the start frame, and the boundary category label is 1. The action category of the first frame is 4, and the action category of the second frame is 4. The second frame has the same action category as the previous frame, i.e., the first frame, but different from the action category of the next frame, i.e., the third frame, so the second frame is the end frame, and the boundary category label is 3. The action category of the third frame is 3, which is different from the action category of the previous frame, i.e., the second frame, but the same as the action category of the next frame, i.e., the fourth frame, so the third frame is the start frame, and the boundary category label is 1. The action category of the fourth frame is 3, which is the same as the action category of the previous and next frames, so the fourth frame is the middle frame, and the boundary category label is 2. By analogy, the corresponding boundary category label sequence can be obtained from the sequence of action category labels corresponding to each video frame in the sample video, that is, [1 3 1 23 1 3 1 3].

[0142] In one embodiment, after determining the corresponding boundary category label based on the action category label of each video frame, the computer device can construct a total loss function based on the boundary classification results and boundary category labels, the action classification results and action category labels, and continue training after updating the model parameters based on the total loss function until a trained action segmentation model is obtained.

[0143] In one embodiment, the computer device can construct a first cross entropy loss based on the action classification result and the action category label, construct a second cross entropy loss based on the boundary classification result and the boundary category label, and use the first cross entropy loss and the second cross entropy loss to construct a total loss function and then train simultaneously to update the model parameters of the two networks.

[0144] In the processing method of the above-mentioned action segmentation model, when training the action segmentation model, the boundary category of each video frame in the sample video is predicted to assist the action classification of the video frame. The frame-level boundary prediction and frame-level action classification are trained together, so that the action segmentation model can learn more discriminative features of the video frame at the boundary, thereby improving the accuracy of action classification of video frames near the action boundary.

[0145] In addition, when predicting the boundary category of a video frame, since the boundary category is irrelevant to the specific action category, it is difficult to directly predict whether the video frame is a boundary frame based on the enhanced features of the video frame. Instead, the prediction is made by using the similarity between each video frame and its adjacent continuous frames. Specifically, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence are divided into multiple logarithmic context levels, and the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into each level. In this way, the context information of each video frame in the logarithmic space is captured, and boundary classification is performed according to the similarity between the enhanced feature vector of each video frame and its logarithmic context feature, which can greatly improve the accuracy of boundary classification. Moreover, by using the context information in the logarithmic space, the context information receptive field range of each video frame can be greatly increased without increasing the computational complexity of calculating the similarity between each video frame and its nearby video frames, thereby overcoming the problem of difficulty in balancing the context range and the similarity computational complexity.

[0146] In one embodiment, in order to mine the features of the temporal relationship between video frames in a video, the computer device may use a multi-stage temporal convolutional neural network to enhance the feature vector sequence.

[0147] Specifically, step 204 performs feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence, including: in a multi-stage feature enhancement network, performing feature enhancement processing on the previous action classification result corresponding to the previous stage to obtain an enhanced feature sequence corresponding to the current stage; wherein the previous action classification result is obtained by performing action classification on the previous enhanced feature sequence corresponding to the previous stage, and the first enhanced feature sequence is generated based on the feature vector sequence of the sample video.

[0148] The multi-stage feature enhancement network includes feature enhancement sub-networks of multiple stages, and the feature enhancement sub-networks of multiple stages are cascaded to form a multi-stage feature enhancement network. The computer device inputs the feature vector sequence into the multi-stage feature enhancement network, and enhances the frame features through the sub-networks of each stage in turn. After the first sub-network generates the first enhanced feature sequence according to the feature vector sequence of the sample video, it generates the first action classification result according to the first enhanced frame sequence. The second sub-network performs feature enhancement processing according to the first action classification result. By analogy, the sub-network of the current stage performs feature enhancement processing on the previous action classification result to obtain the enhanced feature sequence corresponding to the current stage. The previous action classification result is obtained by performing action classification on the previous enhanced feature sequence corresponding to the previous stage. This is repeated multiple times to obtain the enhanced feature sequence output by the sub-network of the final stage.

[0149] Specifically, the computer device converts the feature vector sequence X = [x1, x2, x3, ..., x T ]∈R d×TInput the multi-stage feature enhancement network to perform feature enhancement and obtain the enhanced feature sequence corresponding to each stage Where S is the number of stages in the multi-stage feature enhancement network, that is, the number of sub-networks. The sub-network corresponding to the first stage converts the feature vector sequence X = [x1, x2, x3, ..., x T ]∈R d×T As input, and output the first enhanced feature sequence Z 0 and X have the same length, and the computer device generates an enhanced feature sequence Z according to the output of the first stage. 0 Predict the action category and get the previous action classification result: Where C is the number of action categories. Similarly, the sth stage will be based on the action classification results of the previous stage Generate enhanced feature sequence The enhanced feature sequence Z output by each stage s They will be used for action classification and boundary classification in the current stage to obtain the action classification results Boundary classification results

[0150] In the multi-stage feature enhancement network, the network structure of the sub-network in each stage is the same. Each sub-network contains multiple double dilated convolution layers, each of which has two dilated temporal convolutions with different dilation factors. For one of the dilated convolutions, the dilation factor starts from a small value and increases exponentially with the number of layers. For the other dilated convolution, the dilation factor starts from a large value and decreases exponentially with the number of layers. In this way, both high-level and low-level layers have large local temporal receptive fields, which can capture long-term dependencies and multi-scale temporal information between video frames in the last layer of each stage.

[0151] In this embodiment, by utilizing each sub-network in the multi-stage feature enhancement network to sequentially perform feature enhancement processing on the input feature vector sequence, the association between the video frames in the sample video in terms of chronological sequence can be mined to obtain a more accurate video feature representation.

[0152] Furthermore, the enhanced feature sequence output by each stage of the multi-stage feature enhancement network is used for boundary classification. Figure 5 FIG. 1 is a flow chart of performing boundary prediction at each stage in an embodiment. Figure 5In step 206, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence are divided into multiple logarithmic context levels, and the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into the same level, and the boundary classification result corresponding to each video frame is predicted according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence, which specifically includes:

[0153] Step 502: Divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence corresponding to the current stage into a plurality of logarithmic context levels.

[0154] Specifically, the computer determines the logarithmic context range corresponding to each video frame according to the special hierarchical structure and hyperparameters described above and the enhanced feature sequence corresponding to the current stage, and divides the logarithmic context range of each video frame into multiple logarithmic context levels.

[0155] Step 504 , obtaining a logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors classified into the same level.

[0156] Among them, the logarithmic context feature sequence corresponding to the sample video includes the context features corresponding to each video frame, and the context features corresponding to each video frame include the average vectors corresponding to 2N levels. The average vector in each level is obtained by average pooling based on the enhanced feature vectors classified into the level.

[0157] Step 506 , predicting the boundary classification result corresponding to each video frame at the current stage according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage.

[0158] Among them, the enhanced feature sequence corresponding to the current stage includes an enhanced feature vector corresponding to each video frame. For the current frame, the computer device obtains the context feature corresponding to the current frame from the logarithmic context feature sequence, that is, including the average vector corresponding to 2N levels, obtains the enhanced feature vector corresponding to the current frame from the enhanced feature sequence, and calculates the similarity between each average vector and the enhanced feature vector of the current frame respectively, thereby obtaining the similarity sequence of the current frame, and predicts the boundary classification result corresponding to the current frame based on the similarity sequence. In this way, the computer device can obtain the boundary classification result corresponding to each video frame in the sample video.

[0159] In one embodiment, step 502, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence corresponding to the current stage are divided into multiple logarithmic context levels, including: for each video frame in the sample video, according to the relative direction and distance with the adjacent frames, each enhanced feature vector in the logarithmic context in the enhanced feature sequence corresponding to the current stage is divided into multiple logarithmic context levels; step 504, according to the enhanced feature vectors divided into the same level, the logarithmic context feature sequence corresponding to the sample video is obtained, including: for each logarithmic context level, the average vector of the enhanced feature vectors divided into the same level is calculated, and the sequence formed by the average vector of each logarithmic context level is used as the logarithmic context feature sequence corresponding to each video frame in the current stage.

[0160] Specifically, the enhanced feature sequence generated in the sth stage is In the sth stage, for the tth frame, according to the logarithmic context construction method described above, according to Z s Get the logarithmic context of the frame, a total of 2d1f N-1 video frames, divide them into 2N logarithmic context levels, and for each logarithmic context level, calculate the average value of the enhanced feature vectors falling into that level, and obtain the average vector of the 2N levels in the logarithmic context. The value range of n includes -N, ..., -1,, 1, ..., N. The sequence composed of these 2N average vectors is used as the context feature corresponding to the t-th frame in the s-th stage. The enhanced feature vector z corresponding to the t-th frame in the s-th stage is obtained respectively. t The similarity between each average vector in the context features corresponding to the s-th stage can obtain the similarity sequence corresponding to the t-th frame, thereby classifying the boundary category of the t-th frame and obtaining the boundary classification result corresponding to the t-th frame in the s-th stage.

[0161] In this embodiment, after the logarithmic context is divided into multiple logarithmic context levels, since the similarity only needs to be calculated once for each logarithmic context level instead of calculating the similarity once for each video frame in each level, that is, as the context range increases exponentially, the calculation amount of the similarity only increases linearly, so that the context information receptive field range of each video frame can be greatly increased without increasing the calculation amount of calculating the similarity between each video frame and its nearby video frames, thereby overcoming the problem of difficulty in balancing the context range and the similarity calculation amount.

[0162] like Figure 6 FIG. 1 is a flow chart of performing boundary prediction based on logarithmic context similarity in one embodiment. Figure 6Step 506, predicting the boundary classification result corresponding to each video frame at the current stage according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage, specifically includes:

[0163] Step 602: Determine the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage.

[0164] Specifically, the computer device can encode the enhanced feature vectors in each level into a key through average pooling. Average pooling can reduce the impact of noise frames or abnormal frames, and use the enhanced feature vector corresponding to the current frame at the current stage as the query to calculate the similarity between the query and each key.

[0165] In one embodiment, step 602 determines the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage, including: performing a linear transformation on the enhanced feature sequence corresponding to the current stage to obtain a feature transformation sequence; performing a linear transformation on the logarithmic context feature sequence to obtain a context transformation feature sequence; and obtaining, through a dot multiplication operation, the similarity between each feature transformation vector in the feature transformation sequence and the corresponding context feature transformation vector in the context transformation feature sequence.

[0166] In this embodiment, in order to better learn the importance of adjacent frames to the current frame, the computer device linearly transforms the enhanced feature vector corresponding to each video frame at the current stage to obtain the corresponding feature transformation vector, and uses the transformed feature transformation vector as the query of each video frame, and the feature transformation vectors corresponding to all video frames constitute a feature transformation sequence. Similarly, the computer device linearly transforms the 2N context features corresponding to each video frame at the current stage to obtain 2N transformed context transformation features, and uses the transformed context transformation features as keys, and the context transformation features corresponding to all video frames constitute a context transformation feature sequence.

[0167] For example, the computer device corresponds to the z of the current frame at the current stage t Apply a linear transformation: Where W q is the projection matrix, is the feature transformation vector, which is used as the query. At the same time, the computer device performs 2N context features respectively. n = -N, ..., -1, 1, ..., N Apply another linear transformation: Where W k is the projection matrix, Transform the nth context feature and transform all context features As a key. q and W kare all model parameters that need to be learned for model training. The computer device can calculate the similarity between the query and each key respectively through a dot product operation, thereby obtaining the similarity between the current frame and the video frame in each logarithmic context level.

[0168] Step 604, obtaining the logarithmic context attention score corresponding to each video frame according to the similarity.

[0169] When predicting whether the current frame is a boundary frame, the closer the video frame is to the current frame in the logarithmic context, the greater the influence on the prediction result, and the farther the video frame is from the current frame, the smaller the influence on the prediction result. The logarithmic context is divided into different levels through the embodiment of the present application. In each logarithmic context level, the closer the level is to the current frame, the fewer video frames it contains, and the farther the level is from the current frame, the more video frames it contains. Since the similarity is calculated based on the average vector of multiple enhanced feature vectors in the level, the more video frames there are in the level, the weaker the importance of the video frames in the level. Through this special hierarchical structure, greater importance is assigned to video frames that are closer and less importance is assigned to video frames that are farther away.

[0170] For example, if the first forward level of the current frame includes one video frame, the computer device will directly calculate the similarity between the enhanced feature vector of the frame and the enhanced feature vector of the current frame, and the importance is 1; if the farthest forward level of the current frame includes 8 frames, the computer device needs to first calculate the average vector based on the enhanced feature vectors of these 8 video frames, and then use the average vector to calculate the similarity with the enhanced feature vector of the current frame, then the relative importance of each video frame in this level is 1 / 8.

[0171] The computer device obtains the similarity between the enhanced feature vector corresponding to each video frame at the current stage and each average vector, applies a normalization operation to the 2N similarities corresponding to each video frame, and obtains the logarithmic context attention score of each adjacent frame in the 2N context levels to the current frame at the current stage.

[0172] For example, according to the enhanced feature vector z t Get the feature transformation vector According to contextual features Get context change features Then, we calculate and each After calculating the similarity between the two frames, we apply normalization to all similarities and obtain the attention score a corresponding to the current frame at each logarithmic context level. t , is a vector of length 2N.

[0173] Step 606: Use a linear classifier to predict the boundary classification result corresponding to each video frame at the current stage according to the logarithmic context attention score.

[0174] Specifically, the computer device uses a weight W a ∈R 3×d′ The linear classifier converts the context attention score a of the tth frame t Divided into three categories, the output of the t-th frame corresponds to the score b of the three boundary categories t =W a a t , b t It is a vector of dimension 3, indicating the probability that the tth frame is the start frame, middle frame, and end frame of an action.

[0175] like Figure 7 FIG. 1 is a schematic diagram of performing boundary classification based on logarithmic context in one embodiment. Figure 7 , the hyperparameter d1 used to determine the logarithmic context takes a value of 2, f takes a value of 2, and N takes a value of 3. The logarithmic context of the current frame in the enhanced feature sequence Z is divided into 6 levels, namely forward level 1, forward level 2 and forward level 3, backward level 1, backward level 2 and backward level 3. The enhanced feature vector corresponding to the current frame is Wq. Average pooling is performed on the enhanced feature vectors divided into 6 logarithmic context levels to obtain the average vector Wk in each level. The enhanced feature vector Wq of the current frame is linearly projected to obtain the transformed features. The 6 average vectors are linearly projected to obtain the transformed features. The similarities between the transformed features of the current frame and the 6 transformed features are calculated by dot multiplication operations. Finally, the 6 similarities are normalized and predicted using the boundary classifier to obtain the probabilities that the current frame corresponds to the start frame, the middle frame and the end frame.

[0176] Further, the enhanced feature sequence outputted by each stage of the multi-stage feature enhancement network is used for action classification. Step 208, predicting the action classification result corresponding to each video frame according to the enhanced feature sequence, specifically includes: predicting the action classification result corresponding to each video frame at the current stage based on the enhanced feature sequence corresponding to the current stage.

[0177] The computer device generates an enhanced feature sequence corresponding to the sth stage. After that, the enhanced feature sequence Z output by each stage s All will be used for action classification in the current stage to obtain the action classification results

[0178] In one embodiment, based on the enhanced feature sequence corresponding to the current stage, the action classification result corresponding to each video frame at the current stage is predicted, including: performing action classification on the enhanced feature vectors corresponding to each video frame in the enhanced feature sequence corresponding to the current stage through a linear classifier to obtain the action classification result corresponding to each video frame at the current stage.

[0179] The computer device can use a linear classifier to perform action classification on each enhanced feature vector in each stage of the enhanced feature sequence to obtain the action classification result. Where C is the number of action categories, S is the total number of stages, is the probability that the t-th frame predicted in the s-th stage belongs to each action category.

[0180] In this embodiment, since the boundary classification is trained together with the boundary classification during the training process, the enhanced feature sequence output by the multi-stage feature enhancement network can learn more discriminative features of the video frame at the boundary, thereby improving the accuracy of action classification based on the enhanced feature sequence.

[0181] like Figure 8 As shown, in one embodiment, the action segmentation model includes a feature extraction network for extracting features from a sample video, a multi-stage feature enhancement network for performing feature enhancement processing on a feature vector sequence of the sample video to obtain an enhanced feature sequence, a boundary classification network for dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into multiple logarithmic context levels, obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predicting the boundary classification result corresponding to each video frame according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence, and an action classification network for predicting the action classification result corresponding to each video frame according to the enhanced feature sequence.

[0182] In one embodiment, the multi-stage feature enhancement network includes a feature generation subnetwork and multiple feature refinement subnetworks. The computer device sequentially performs feature enhancement processing on the feature vector sequence of the sample video through the feature generation subnetwork and the multiple feature refinement subnetworks.

[0183] In one embodiment, feature enhancement processing is performed on a previous action classification result corresponding to a previous stage to obtain an enhanced feature sequence corresponding to a current stage, including: inputting a feature vector sequence into a multi-stage feature enhancement network; in a feature generation subnetwork, feature enhancement processing is performed on the feature vector sequence to obtain an enhanced feature sequence corresponding to the sample video at the feature generation stage; based on the enhanced feature sequence corresponding to the current stage, the action classification result corresponding to each video frame at the current stage is predicted, including: performing action classification according to the enhanced feature sequence corresponding to the feature generation stage to obtain the action classification result corresponding to the sample video at the feature generation stage.

[0184] Among them, the stage feature enhancement network includes a feature generation subnetwork and multiple feature refinement subnetworks, for example, S feature refinement subnetworks. The feature generation subnetwork generates an enhanced feature sequence corresponding to the feature generation stage, and each feature refinement subnetwork refines the enhanced feature sequence generated in the previous stage.

[0185] For example, in the feature generation stage, the feature vector sequence X of the sample video is [x1, x2, x3, ..., x T ]∈R d×T As the input of the feature generation sub-network, the enhanced sequence is output through the feature generation sub-network Z 0 has the same length as X, Z 0 The z corresponding to the tth frame in t Enhances x in X by fusing long-term dependencies t , the enhanced feature sequence output by the feature generation subnetwork It will be used for action classification to obtain the initial action classification result, that is, the action classification result corresponding to the feature generation stage

[0186] Furthermore, after generating the action classification result corresponding to the feature generation stage, the action classification result corresponding to the feature generation stage is used as input to the feature refinement subnetwork of the feature refinement stage to continue the feature enhancement process.

[0187] Specifically, the above method also includes: in the first feature refinement subnetwork, performing feature enhancement processing on the action classification results corresponding to the feature generation stage, after obtaining the enhanced feature sequence corresponding to the sample video in the first feature refinement stage, iteratively obtaining the enhanced feature sequence corresponding to the sample video in the current feature refinement stage according to the action classification results corresponding to the previous feature refinement stage, until the iteration is stopped when the enhanced feature sequence corresponding to the last feature refinement stage is obtained; performing action classification according to the enhanced feature sequence corresponding to the current feature refinement stage, and obtaining the action classification result corresponding to the sample video in the current feature refinement stage.

[0188] For example, the first feature refinement subnetwork converts the action classification results corresponding to the feature generation stage As input, output is the enhanced feature sequence corresponding to the first feature refinement stage Similarly, the sth (s=1,2,…,S)th feature refinement stage will be based on the action classification results of the previous feature refinement stage Generate enhanced feature sequence The enhanced feature sequence Z output at each feature refinement stage s They will be used for action classification and boundary classification in the current feature refinement stage to obtain the action classification results Boundary classification results The enhanced feature sequence output by each feature refinement stage uses the same classification network for action classification and the same boundary classification network for boundary classification.

[0189] In one embodiment, a total loss function is constructed based on the boundary classification result and the boundary category label, the action classification result and the action category label, including:

[0190] Based on the boundary classification results and boundary category labels corresponding to each stage, the boundary classification loss function of all stages is constructed; based on the action classification results and action category labels corresponding to each stage, the action classification loss function of all stages is constructed; the boundary classification loss function of all stages and the action classification loss function of all stages are summed to obtain the total loss function.

[0191] Specifically, each feature refinement stage will output the action classification results and boundary classification results corresponding to each video frame in the sample video. In order to improve the generalization of the model, each feature refinement stage is constrained. The computer equipment can count the classification loss and boundary loss of all stages, so that the output of each stage is more discriminative.

[0192] The computer device can use cross entropy loss as the classification loss, and use truncated mean squared error (TMSE) loss to reduce over-segmentation errors.

[0193] For example, the cross entropy loss corresponding to the sample video at the sth stage is:

[0194]

[0195] Among them, the sample video has a total of T video frames, t is the tth video frame x in the sample video t , c t is x t The corresponding action category label, yes The c t elements, represents the model's prediction x t Belongs to c t The probability of a class.

[0196] The truncated mean square error loss corresponding to the sample video at the sth stage is:

[0197]

[0198]

[0199] in, Represents the predicted x t The probability of belonging to the cth category in the sth stage, there are a total of C action categories, Represents the predicted previous frame x t-1 The probability of belonging to the cth class in the sth stage, θ is the preset threshold.

[0200] The computer device sums the classification loss and truncated mean square error loss of all video frames in all stages to construct the action classification loss function for all stages:

[0201] Among them, λ is a preset value.

[0202] When constructing the boundary classification loss, the computer device needs to first calculate the action category label c = [c1, c2, c3, ..., c T ] obtain the corresponding boundary category label e = [e1, e2, e3, ..., e T ], where e t ∈{0,1,2}, representing the corresponding start frame, middle frame and end frame respectively.

[0203] For the sth stage, the boundary classification loss for all frames is:

[0204]

[0205] Among them, e t is x t The corresponding boundary category label, Represents the predicted x t The boundary classification result corresponding to the sth stage is a 3D vector, which indicates the probability that the tth frame is the starting frame, the middle frame, and the ending frame of an action.

[0206] The computer device sums the boundary classification losses of all video frames in all stages and constructs the boundary classification loss function for all stages:

[0207]

[0208] Finally, the boundary classification loss function of all stages is summed with the action classification loss function of all stages to obtain the total loss function:

[0209] L=L cls +βL bp .

[0210] like Fig. 9 FIG. 1 is a schematic diagram of the overall framework of an action segmentation model in an embodiment, referring to FIG. Fig. 9 In the training phase, the input of the action segmentation model is the feature vector sequence X corresponding to the sample video, and the output is the action classification result Y and boundary classification result B corresponding to each video in the sample video. The feature vector sequence X corresponding to the sample video is input into the feature generation subnetwork of the multi-stage feature enhancement network. The feature vector sequence X is subjected to feature enhancement processing by the feature generation subnetwork to obtain the enhanced feature sequence Z0 corresponding to the sample video in the feature generation phase. Z0 is input into the action classification network to obtain the action classification result Y0 corresponding to each video frame in the sample video in the feature generation phase. Z0 is input into the boundary classification network based on logarithmic context attention to obtain the boundary classification result B0 corresponding to each video frame in the sample video in the feature generation phase. Then the action classification result Y0 is input into the first feature refinement subnetwork and input into the feature refinement subnetwork. Output the enhanced feature sequence Z1 corresponding to the first feature refinement stage, input Z1 into the action classification network, obtain the action classification result Y1 corresponding to each video frame in the sample video at the first feature refinement stage, input Z0 into the boundary classification network, obtain the boundary classification result B1 corresponding to each video frame in the sample video at the first feature refinement stage, and so on, until the enhanced feature sequence corresponding to the final feature refinement stage is obtained, and then the iteration is stopped. Finally, action classification and boundary classification are performed according to the enhanced feature sequence corresponding to the final feature refinement stage to obtain the final action classification result and boundary classification result of each video frame in the sample video. The computer device constructs an action classification loss according to the action classification result of each video frame output from all stages, and constructs a boundary classification loss according to the boundary classification result of each video frame output from all stages.

[0211] For the action segmentation model verified by the test set, the computer device can use the action classification network in the action segmentation model to identify the action category of each video frame in the long video.

[0212] In one embodiment, the method further includes: inputting the video to be processed into a trained action segmentation model; obtaining a feature vector sequence corresponding to the video to be processed through a feature extraction network in the action segmentation model; performing feature enhancement processing on the feature vector sequence of the sample video through a multi-stage feature enhancement network in the action segmentation model to obtain an enhanced feature sequence corresponding to the final stage; and performing prediction based on the enhanced feature sequence corresponding to the final stage through an action classification network in the action segmentation model to obtain the action category corresponding to each video frame in the video to be processed.

[0213] In this embodiment, after the action segmentation model is trained, in a scenario where boundary classification is not required, the computer device can only use the feature extraction network, the multi-stage feature enhancement network and the action classification network to perform action segmentation on the input video to be processed to obtain the action category corresponding to each video frame.

[0214] In one embodiment, the method further includes: predicting, using a boundary classification network in the action segmentation model, according to an enhanced feature sequence corresponding to the final stage, to obtain boundary categories corresponding to each video frame in the video to be processed.

[0215] In this embodiment, in other scenarios, the computer device may also use a boundary classification network to predict the boundary category corresponding to each video frame in the video to be processed.

[0216] In one embodiment, the method further includes: obtaining an action category corresponding to each video frame in the video to be processed; and dividing the video to be processed into a plurality of video segments according to video frames corresponding to the same action category.

[0217] like Fig.10 FIG. 1 is a flow chart of a method for processing an action segmentation model in a specific embodiment. Fig.10 , including the following steps:

[0218] Step 1002: Obtain a sample video and action category labels of video frames in the sample video.

[0219] Step 1004: input the sample video and action category labels into the action segmentation model for training.

[0220] Step 1006: extract feature vectors from each video frame of the sample video in the feature extraction network of the action segmentation model.

[0221] Step 1008: Obtain a feature vector sequence corresponding to the sample video according to the feature vector of the video frame.

[0222] Step 1010, input the feature vector sequence into the multi-stage feature enhancement network of the action segmentation model.

[0223] Step 1012, in the feature generation subnetwork of the multi-stage feature enhancement network, feature enhancement is performed on the feature vector sequence to obtain an enhanced feature sequence corresponding to the feature generation stage.

[0224] Step 1014, in the action classification network of the action segmentation model, action classification is performed according to the enhanced feature sequence corresponding to the feature generation stage, and the action classification result corresponding to the sample video in the feature generation stage is obtained.

[0225] Step 1016, in the boundary classification network of the action segmentation model, the enhanced feature vectors in the enhanced feature sequence that are in the logarithmic context of the current frame are divided into multiple logarithmic context levels according to the direction and distance from the current frame; for each logarithmic context level, the average vector of the enhanced feature vectors divided into the same level is calculated to obtain the logarithmic context features corresponding to the current frame in the feature generation stage; the similarity between the feature vector of the current frame and each logarithmic context feature in the feature vector sequence is calculated, and the logarithmic context attention score corresponding to the current frame is obtained according to the similarity, and the boundary classification result corresponding to the current frame in the feature generation stage is predicted according to the logarithmic context attention score.

[0226] Step 1018, in the first feature refinement subnetwork of the multi-stage feature enhancement network, feature enhancement is performed according to the action classification result corresponding to the feature generation stage to obtain an enhanced feature sequence corresponding to the first feature refinement stage.

[0227] Step 1020, in the action classification network of the action segmentation model, action classification is performed according to the enhanced feature sequence corresponding to the first feature refinement stage, and the action classification result corresponding to the sample video in the first feature refinement stage is obtained.

[0228] Step 1022, in the boundary classification network of the action segmentation model, the enhanced feature vectors in the logarithmic context of the current frame in the enhanced feature sequence corresponding to the first feature refinement stage are divided into multiple logarithmic context levels according to the direction and distance from the current frame; for each logarithmic context level, the average vector of the enhanced feature vectors divided into the same level is calculated to obtain the logarithmic context features corresponding to the current frame in the first feature refinement stage; the similarity between the enhanced feature vector of the current frame in the enhanced feature sequence and each logarithmic context feature is calculated, and the logarithmic context attention score corresponding to the current frame is obtained according to the similarity, and the boundary classification result corresponding to the current frame in the first feature refinement stage is predicted according to the logarithmic context attention score.

[0229] Step 1024, iteratively obtain the enhanced feature sequence corresponding to the sample video at the current feature refinement stage according to the action classification result corresponding to the previous feature refinement stage, and stop the iteration when the enhanced feature sequence corresponding to the last feature refinement stage is obtained.

[0230] Step 1026: Determine the corresponding boundary category label according to the action category label of each video frame.

[0231] Step 1028, constructing the boundary classification loss function of all stages based on the boundary classification results and boundary category labels corresponding to each stage.

[0232] Step 1030, constructing the action classification loss function of all stages based on the action classification results and action category labels corresponding to each stage.

[0233] Step 1032, summing the boundary classification loss functions of all stages and the action classification loss functions of all stages to obtain a total loss function.

[0234] Step 1034, continue training after updating the model parameters according to the total loss function until a trained action segmentation model is obtained.

[0235] Step 1036, input the video to be processed into the trained action segmentation model.

[0236] Step 1038: Obtain the action category corresponding to each video frame in the video to be processed through the trained action segmentation model.

[0237] Step 1040 , dividing the video to be processed into a plurality of video segments according to video frames corresponding to the same action category.

[0238] In the processing method of the above action segmentation model, the similarity-based boundary classifier predicts the boundary type according to the similarity scores between consecutive frames, forcing the model to learn more discriminative features to improve the action classification accuracy of frames around the boundary, thereby improving the performance of the action segmentation task. By setting a special logarithmic context attention, the adjacent frames of each frame in the logarithmic context are divided into different levels to capture structured context information and greatly increase the local temporal receptive field range without increasing the similarity calculation amount. Extensive experiments on benchmark datasets show that the action segmentation method based on local logarithmic context attention has comparable or better performance than other action segmentation methods, indicating the effectiveness and prospects of this method in video action segmentation tasks.

[0239] In one embodiment, Fig.11 As shown, a video processing method is provided, which is applied to Figure 1 Computer equipment in Figure 1 The terminal or server in the example is used to illustrate, including the following steps:

[0240] Step 1102, obtaining the video to be processed.

[0241] Step 1104: input the video to be processed into the trained action segmentation model.

[0242] Step 1106, obtaining a feature vector sequence corresponding to the video to be processed through the action segmentation model, performing feature enhancement processing on the feature vector sequence of the video to be processed to obtain an enhanced feature sequence, and performing prediction based on the enhanced feature sequence to obtain the action category corresponding to each video frame in the video to be processed.

[0243] Among them, the action segmentation model is obtained by model training based on the sample video and the action category labels of each video frame in the sample video, the boundary classification results and the action classification results; the boundary classification result is to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence of the sample video into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict it according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence of the sample video; the action classification result is predicted based on the enhanced feature sequence of the sample video; the enhanced feature sequence of the sample video is obtained by performing feature enhancement processing on the feature vector sequence of the sample video.

[0244] Regarding the training steps of the action segmentation model in the above video processing method, the embodiments provided by the above action segmentation model processing method are provided and will not be described in detail here.

[0245] It should be understood that, although the various steps in the flow chart in the above-described embodiment are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flow chart in the above-described embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0246] Regarding the performance of the action segmentation model provided in the present application, the performance of the action segmentation model of the embodiment of the present application was evaluated on three datasets: the 50Salads dataset, the GTEA dataset, and the Breakfast dataset. The 50Salads dataset contains 50 top-view videos with 17 action classes, each video lasting an average of about 6.4 minutes and containing about 20 action instances. The average effect is reported after five cross-validations. The GTEA dataset contains 28 first-person videos with 11 action classes, including 7 background classes. Each video has an average of about 20 action instances. The average performance is reported after four cross-validations. The Breakfast dataset contains 1712 third-person perspective videos with 48 action categories. Each video contains an average of about 6 action instances. The average performance is reported after four cross-validations.

[0247] On all datasets, each video is represented as a series of visual features. Using I3D frame features as input, the temporal video resolution of all video datasets is fixed to 15fps, and the segmented F1 score on overlap thresholds of 10%, 25%, 50% (F1@{10,25,50}), the measured segmented edit distance (Edit) between the predicted segmentation path and the true segmentation path, and the frame-by-frame accuracy (Acc) are used as evaluation metrics.

[0248] For the multi-stage feature enhancement network, all hyperparameters (such as the number of stages, the number of layers, the number of filters per layer, the dilation factor per layer, the random seed, etc.) remain unchanged, and all refinement stages share parameters. The Adam optimizer is used, and the learning rate is set to 0.0005. The current frame is predicted to be at the beginning, middle, or end of the action based on the similarity between the current frame and the surrounding frames. In this process, the dot product between the frame features with linear projection can be used to calculate the similarity, and the dot product between the frame features without linear projection can be used as their similarity, and the similarity is directly used as input to predict the boundary.

[0249] Table 1 shows the comparison results of the method of the embodiment of the present application and the simplified model without linear projection and / or soft-max on the 50salads dataset. The hyperparameters β, d1, f and N are fixed to 0.15, 1, 2 and 3 respectively. Both linear projection and normalization can improve performance.

[0250] Table 1 The impact of linear projection and normalized nonlinear activation

[0251]

[0252] The weight of the boundary prediction loss is controlled by a hyperparameter β. Table 2 compares the results obtained by training the models with different β values. β = 0.25 or 0.35 generally achieves the best results.

[0253] Table 2 Impact of boundary prediction loss weights

[0254]

[0255] The setting of logarithmic context for boundary classification includes three hyperparameters: the number of frames d1 in the innermost layer, the growth rate f, and the number of levels N. The model is trained with different hyperparameters by changing d1, f, and N from 1 to 3 respectively. The results are compared in Table 3. These hyperparameters control the size and granularity of the context receptive field. Larger d1, larger f, and larger N increase the local receptive field by including more adjacent frames, but when the number of levels N is fixed, the levels are coarser because each level contains more frames and the information of these frames is averaged. When d1 and f are too large, some frames in the middle of the action may be incorrectly predicted as boundary candidates because the boundary of the action may be contained in its inner level. When N is too large, the number of frames in the outermost level will be very large, and the effective context information may be submerged after averaging. When the size of the entire local receptive field is fixed, increasing the number of levels helps capture more detailed and finer configurations of the local context. From Table 3, it can be seen that setting d1 = 1 or 2, f = 2, and N = 3 generally achieves better results. On all datasets, we fix d1, f, and N to 1, 2, and 3, respectively.

[0256] Table 3. Effect of hyperparameters

[0257]

[0258] In Table 3, when N is set to 1, there is only one level in both the forward and reverse directions, so f has no effect, so d1 adjacent frames in both directions can be used as context. For each direction, the average of d1 frame features is used as the key. In all performance indicators, 3 levels and a factor of 2 outperform the degradation models with d1=1 and d1=2, respectively. It is also possible to uniformly sample several frames in both directions as key points. In Table 4, 3 frames are sampled in each direction, and the sampling intervals are 1 and 2, respectively. When the interval is 2, the receptive field of the context is similar to that of the logarithmic context with d1=1, f=2, and N=3, but the effect of the logarithmic context is better.

[0259] Table 4 Performance comparison under different context configurations

[0260]

[0261]

[0262] A direct alternative method for predicting action boundaries is to apply the classifier directly to the obtained enhanced frame features. We use "directly" to represent this method. As shown in Table 5, the logarithmic context boundary classification method provided by the embodiment of the present application outperforms this method in all performance indicators, which verifies the effectiveness of predicting action boundaries based on the similarity between local context ranges.

[0263] The method of the present application distinguishes between the start and end boundaries. However, when predicting the boundary, it is also possible to only predict whether the frame is a boundary frame, without further subdividing the boundary type into the start and end. We also compared with this method, which directly performs binary prediction based on the enhanced frame features and is represented by "direct two-class". The local similarity-based boundary prediction module of the present application can also perform two-class prediction between boundary frames and non-boundary frames. We use "two-class" to represent this method. It can be observed from Table 5 that even if predictions are made directly based on features, it is beneficial to distinguish boundary types. The method of the present application outperforms the two-class boundary prediction methods based on frame features and local similarity in all performance indicators, which proves the advantage of distinguishing fine-grained boundary types.

[0264] Table 5 Comparison with different boundary prediction methods

[0265]

[0266] Tables 6, 7, and 8 compare the performance of the method of the present application with other state-of-the-art action segmentation methods on the 50Salads dataset, the GTEA dataset, and the Breakfast dataset, respectively, where MS-TCN++(sh) represents the shared parameters of all refinement stages of MS-TCN++. On all datasets, the method of the present application generally outperforms MS-TCN++(sh) in performance indicators. By predicting action boundaries based on the similarity between local logarithmic contexts, the method of the present application imposes additional constraints on the frame features of the final layer for regularization, which helps to learn more discriminative and boundary-sensitive frame features.

[0267] On all datasets, the method of the present application is far superior to other methods except MS-TCN++. Compared with MS-TCN++ without parameter sharing, the method of the present application can obtain comparable results on the 50Salads dataset and better results on the other two datasets. Since the parameters are not shared in the refinement stage, the total number of parameters of MS-TCN++ increases many times. The feature enhancement backbone of the method of the present application can also use MS-TCN++ to achieve further performance improvement by not sharing parameters.

[0268] Table 6 Comparison with other action segmentation methods on the 50Salads dataset

[0269]

[0270] Table 7 Comparison with other action segmentation methods on the GTEA dataset

[0271]

[0272]

[0273] Table 8 Comparison with other action segmentation methods on the Breakfast dataset

[0274]

[0275] In one embodiment, Fig.12 As shown, a processing device 1200 for an action segmentation model is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an acquisition module 1202, a feature enhancement module 1204, a boundary classification module 1206, an action classification module 1208 and a parameter updating module 1210, wherein:

[0276] An acquisition module 1202 is used to acquire a sample video and an action category label of a video frame in the sample video;

[0277] A feature enhancement module 1204 is used to perform feature enhancement processing on a feature vector sequence of a sample video to obtain an enhanced feature sequence;

[0278] The boundary classification module 1206 is used to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict the boundary classification result corresponding to each video frame according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence;

[0279] An action classification module 1208 is used to predict the action classification result corresponding to each video frame according to the enhanced feature sequence;

[0280] The parameter updating module 1210 is used to update the model parameters based on the action category label, boundary classification result and action classification result of each video frame and then continue training until a trained action segmentation model is obtained.

[0281] In one embodiment, the processing device 1200 of the action segmentation model further includes a feature extraction module for extracting feature vectors from each video frame constituting a sample video in a feature extraction network; and obtaining a feature vector sequence corresponding to the sample video based on the feature vectors of the video frames.

[0282] In one embodiment, the feature enhancement module 1204 is used to perform feature enhancement processing on the previous action classification result corresponding to the previous stage in a multi-stage feature enhancement network to obtain an enhanced feature sequence corresponding to the current stage; wherein the previous action classification result is obtained by performing action classification on the previous enhanced feature sequence corresponding to the previous stage, and the first enhanced feature sequence is generated based on the feature vector sequence of the sample video.

[0283] In one embodiment, the boundary classification module 1206 is used to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence corresponding to the current stage into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict the boundary classification result corresponding to each video frame at the current stage according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage; the action classification module 1208 is used to predict the action classification result corresponding to each video frame at the current stage based on the enhanced feature sequence corresponding to the current stage.

[0284] In one embodiment, the feature enhancement module 1204 is used to input the feature vector sequence into a multi-stage feature enhancement network, which includes a feature generation subnetwork and multiple feature refinement subnetworks; in the feature generation subnetwork, feature enhancement processing is performed on the feature vector sequence to obtain an enhanced feature sequence corresponding to the sample video in the feature generation stage; the action classification module 1208 is used to perform action classification according to the enhanced feature sequence corresponding to the feature generation stage to obtain the action classification result corresponding to the sample video in the feature generation stage.

[0285] In one embodiment, the feature enhancement module 1204 is used to perform feature enhancement processing on the action classification results corresponding to the feature generation stage in the first feature refinement subnetwork, and after obtaining the enhanced feature sequence corresponding to the sample video in the first feature refinement stage, iteratively obtain the enhanced feature sequence corresponding to the sample video in the current feature refinement stage according to the action classification results corresponding to the previous feature refinement stage, and stop the iteration when the enhanced feature sequence corresponding to the last feature refinement stage is obtained; the action classification module 1208 is used to perform action classification according to the enhanced feature sequence corresponding to the current feature refinement stage, and obtain the action classification result corresponding to the sample video in the current feature refinement stage.

[0286] In one embodiment, the action classification module 1208 is used to perform action classification on the enhanced feature vectors corresponding to each video frame in the enhanced feature sequence corresponding to the current stage through a linear classifier to obtain the action classification results corresponding to each video frame at the current stage.

[0287] In one embodiment, the boundary classification module 1206 is used to divide each enhanced feature vector in the logarithmic context in the enhanced feature sequence corresponding to the current stage into multiple logarithmic context levels for each video frame in the sample video according to the relative direction and distance to the adjacent frames; for each logarithmic context level, calculate the average vector of the enhanced feature vectors divided into the same level, and use the sequence formed by the average vectors of each logarithmic context level as the logarithmic context feature sequence corresponding to each video frame at the current stage.

[0288] In one embodiment, the boundary classification module 1206 is used to determine the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage; obtain the logarithmic context attention score corresponding to each video frame based on the similarity; and predict the boundary classification result corresponding to each video frame at the current stage based on the logarithmic context attention score through a linear classifier.

[0289] In one embodiment, the boundary classification module 1206 is used to perform a linear transformation on the enhanced feature sequence corresponding to the current stage to obtain a feature transformation sequence; perform a linear transformation on the logarithmic context feature sequence to obtain a context transformation feature sequence; and obtain the similarity between each feature transformation vector in the feature transformation sequence and the corresponding context feature transformation vector in the context transformation feature sequence through a dot multiplication operation.

[0290] In one embodiment, the parameter updating module 1210 is used to determine the corresponding boundary category label according to the action category label of each video frame; construct a total loss function according to the boundary classification results and boundary category labels, the action classification results and the action category labels; and continue training after updating the model parameters according to the total loss function until a trained action segmentation model is obtained.

[0291] In one embodiment, the parameter updating module 1210 is used to construct the boundary classification loss function of all stages based on the boundary classification results and boundary category labels corresponding to each stage; construct the action classification loss function of all stages based on the action classification results and action category labels corresponding to each stage; sum the boundary classification loss function of all stages and the action classification loss function of all stages to obtain the total loss function.

[0292] In one embodiment, the action segmentation model includes a feature extraction network for extracting features from a sample video, a multi-stage feature enhancement network for performing feature enhancement processing on a feature vector sequence of the sample video to obtain an enhanced feature sequence, a boundary classification network for dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into multiple logarithmic context levels, obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predicting the boundary classification result corresponding to each video frame according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence, and an action classification network for predicting the action classification result corresponding to each video frame according to the enhanced feature sequence.

[0293] In one embodiment, the processing device 1200 of the action segmentation model also includes a processing video classification module, which is used to input the processing video into the trained action segmentation model; through the feature extraction network in the action segmentation model, a feature vector sequence corresponding to the processing video is obtained; through the multi-stage feature enhancement network in the action segmentation model, feature enhancement processing is performed on the feature vector sequence of the sample video to obtain the enhanced feature sequence corresponding to the final stage; through the action classification network in the action segmentation model, prediction is performed according to the enhanced feature sequence corresponding to the final stage to obtain the action category corresponding to each video frame in the processing video.

[0294] In one embodiment, the processing device 1200 of the action segmentation model further includes a video segmentation module for obtaining the action category corresponding to each video frame in the video to be processed; and segmenting the video to be processed into multiple video segments according to the video frames corresponding to the same action category.

[0295] In one embodiment, the module for classifying videos to be processed is further used to predict, through a boundary classification network in the action segmentation model, the enhanced feature sequence corresponding to the final stage, to obtain boundary categories corresponding to each video frame in the video to be processed.

[0296] In one embodiment, Fig.13 As shown, a video processing device 1300 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an acquisition module 1302, an input module 1304 and an action segmentation module 1306, wherein:

[0297] An acquisition module 1302 is used to acquire a video to be processed;

[0298] An input module 1304 is used to input the video to be processed into the trained action segmentation model;

[0299] The action segmentation module 1306 is used to obtain a feature vector sequence corresponding to the video to be processed through the action segmentation model, perform feature enhancement processing on the feature vector sequence of the video to be processed to obtain an enhanced feature sequence, and perform prediction based on the enhanced feature sequence to obtain the action category corresponding to each video frame in the video to be processed;

[0300] Among them, the action segmentation model is obtained by model training based on the sample video and the action category labels of each video frame in the sample video, the boundary classification results and the action classification results; the boundary classification result is to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence of the sample video into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict it according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence of the sample video; the action classification result is predicted based on the enhanced feature sequence of the sample video; the enhanced feature sequence of the sample video is obtained by performing feature enhancement processing on the feature vector sequence of the sample video.

[0301] The processing device 1200 and the video processing device 1300 of the above-mentioned action segmentation model, when training the action segmentation model, predict the boundary category of each video frame in the sample video to assist the action classification of the video frame, and train the frame-level boundary prediction and the frame-level action classification together, so that the action segmentation model can learn more discriminative features of the video frame at the boundary, thereby improving the accuracy of action classification of video frames near the action boundary.

[0302] In addition, when predicting the boundary category of a video frame, since the boundary category is irrelevant to the specific action category, it is difficult to directly predict whether the video frame is a boundary frame based on the enhanced features of the video frame. Instead, the prediction is made by using the similarity between each video frame and its adjacent continuous frames. Specifically, the enhanced feature vectors in the logarithmic context in the enhanced feature sequence are divided into multiple logarithmic context levels, and the logarithmic context feature sequence corresponding to the sample video is obtained according to the enhanced feature vectors divided into different levels. In this way, the context information of each video frame in the logarithmic space can be captured, and boundary classification is performed according to the similarity between the enhanced feature vector of each video frame and its logarithmic context feature, which can greatly improve the accuracy of boundary classification. Moreover, by using the context information in the logarithmic space, the context information receptive field range of each video frame can be greatly increased without increasing the computational complexity of calculating the similarity between each video frame and its nearby video frames, thereby overcoming the problem of difficulty in balancing the context range and the similarity computational complexity.

[0303] For the specific definition of the processing device 1200 of the action segmentation model, please refer to the definition of the processing method of the action segmentation model in the above text. For the specific definition of the video processing device 1300, please refer to the definition of the video processing method in the above text, which will not be repeated here. Each module in the above-mentioned device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0304] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.14 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for processing an action segmentation model and / or a video processing method is implemented.

[0305] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0306] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0307] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0308] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0309] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0310] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0311] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for processing an action segmentation model, comprising: Obtaining a sample video and action category labels of video frames in the sample video; Performing feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence; Dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into a plurality of logarithmic context levels, obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predicting the boundary classification result corresponding to each of the video frames according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence; Predicting the action classification result corresponding to each of the video frames according to the enhanced feature sequence; After updating the model parameters based on the action category labels of each video frame, the boundary classification results and the action classification results, training is continued until a trained action segmentation model is obtained. The action segmentation model is used to predict the action category corresponding to each video frame in the video to be processed, and then obtain video clips segmented by action category.

2. The method according to claim 1, characterized in that The method further comprises: In the feature extraction network, extracting feature vectors from each video frame constituting the sample video; A feature vector sequence corresponding to the sample video is obtained according to the feature vector of the video frame.

3. The method according to claim 1, characterized in that The step of performing feature enhancement processing on the feature vector sequence of the sample video to obtain an enhanced feature sequence includes: In a multi-stage feature enhancement network, feature enhancement processing is performed on a previous action classification result corresponding to a previous stage to obtain an enhanced feature sequence corresponding to a current stage; wherein the previous action classification result is obtained by performing action classification on a previous enhanced feature sequence corresponding to the previous stage, and the first enhanced feature sequence is generated based on a feature vector sequence of the sample video.

4. The method according to claim 3, characterized in that The method further comprises dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence into a plurality of logarithmic context levels, obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predicting the boundary classification result corresponding to each of the video frames according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence, including: Divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence corresponding to the current stage into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict the boundary classification result corresponding to each of the video frames at the current stage according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage; The predicting the action classification result corresponding to each of the video frames according to the enhanced feature sequence includes: predicting the action classification result corresponding to each of the video frames at the current stage based on the enhanced feature sequence corresponding to the current stage.

5. The method according to claim 4, characterized in that The step of performing feature enhancement processing on the previous action classification result corresponding to the previous stage to obtain an enhanced feature sequence corresponding to the current stage includes: Inputting the feature vector sequence into the multi-stage feature enhancement network, wherein the multi-stage feature enhancement network includes a feature generation sub-network and a plurality of feature refinement sub-networks; In the feature generation subnetwork, feature enhancement processing is performed on the feature vector sequence to obtain an enhanced feature sequence corresponding to the sample video in the feature generation stage; The predicting, based on the enhanced feature sequence corresponding to the current stage, the action classification result corresponding to each of the video frames at the current stage includes: Action classification is performed according to the enhanced feature sequence corresponding to the feature generation stage to obtain the action classification result corresponding to the sample video in the feature generation stage.

6. The method according to claim 5, characterized in that The method further comprises: In the first feature refinement subnetwork, feature enhancement processing is performed on the action classification results corresponding to the feature generation stage, and after obtaining the enhanced feature sequence corresponding to the sample video at the first feature refinement stage, the enhanced feature sequence corresponding to the sample video at the current feature refinement stage is iteratively obtained according to the action classification results corresponding to the previous feature refinement stage, until the enhanced feature sequence corresponding to the last feature refinement stage is obtained and the iteration is stopped; Action classification is performed according to the enhanced feature sequence corresponding to the current feature refinement stage to obtain the action classification result corresponding to the sample video at the current feature refinement stage.

7. The method according to claim 4, characterized in that The predicting, based on the enhanced feature sequence corresponding to the current stage, the action classification result corresponding to each of the video frames at the current stage includes: The enhanced feature vectors corresponding to each video frame in the enhanced feature sequence corresponding to the current stage are subjected to action classification by a linear classifier to obtain the action classification results corresponding to each video frame at the current stage.

8. The method according to claim 4, characterized in that The step of dividing the enhanced feature vectors in the logarithmic context in the enhanced feature sequence corresponding to the current stage into a plurality of logarithmic context levels, and obtaining the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, comprises: For each video frame in the sample video, according to the relative direction and distance to the adjacent frames, each enhanced feature vector in the enhanced feature sequence corresponding to the current stage and in the logarithmic context is divided into a plurality of logarithmic context levels; For each logarithmic context level, the average vector of the enhanced feature vectors divided into the same level is calculated, and the sequence formed by the average vectors of each logarithmic context level is used as the logarithmic context feature sequence corresponding to each of the video frames at the current stage.

9. The method according to claim 4, characterized in that The predicting, based on the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage, the boundary classification result corresponding to each of the video frames at the current stage comprises: Determining the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage; Obtaining a logarithmic context attention score corresponding to each of the video frames according to the similarity; Through a linear classifier, the boundary classification result corresponding to each of the video frames at the current stage is predicted according to the logarithmic context attention score.

10. The method according to claim 9, characterized in that The determining the similarity between the logarithmic context feature sequence and the enhanced feature sequence corresponding to the current stage includes: Performing a linear transformation on the enhanced feature sequence corresponding to the current stage to obtain a feature transformation sequence; Performing a linear transformation on the logarithmic context feature sequence to obtain a context transformation feature sequence; The similarity between each feature transformation vector in the feature transformation sequence and the corresponding context feature transformation vector in the context transformation feature sequence is obtained through a dot product operation.

11. The method according to claim 3, characterized in that The updating of the model parameters according to the action category label of each video frame, the boundary classification result and the action classification result and continuing the training until a trained action segmentation model is obtained comprises: Determining a corresponding boundary category label according to the action category label of each of the video frames; Constructing a total loss function according to the boundary classification result and the boundary category label, the action classification result and the action category label; After updating the model parameters according to the total loss function, training is continued until a trained action segmentation model is obtained.

12. The method according to claim 11, characterized in that The constructing a total loss function according to the boundary classification result and the boundary category label, the action classification result and the action category label comprises: Constructing a boundary classification loss function for all stages based on the boundary classification results corresponding to each stage and the boundary category labels; Constructing the action classification loss function of all stages based on the action classification results corresponding to each stage and the action category labels; The boundary classification loss function of all stages and the action classification loss function of all stages are summed to obtain a total loss function.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: Inputting the video to be processed into the trained action segmentation model; The feature vector sequence corresponding to the video to be processed is obtained through the feature extraction network in the action segmentation model. The feature vector sequence of the sample video is subjected to feature enhancement processing through the multi-stage feature enhancement network in the action segmentation model to obtain the enhanced feature sequence corresponding to the final stage. The action classification network in the action segmentation model predicts according to the enhanced feature sequence corresponding to the final stage to obtain the action category corresponding to each video frame in the video to be processed.

14. The method according to claim 13, characterized in that The method further comprises: Obtaining the action category corresponding to each of the video frames in the video to be processed; The video to be processed is divided into a plurality of video segments according to video frames corresponding to the same action category.

15. A video processing method, comprising: Get the video to be processed; Inputting the video to be processed into a trained action segmentation model; Obtaining a feature vector sequence corresponding to the video to be processed through the action segmentation model, performing feature enhancement processing on the feature vector sequence of the video to be processed to obtain an enhanced feature sequence, and performing prediction based on the enhanced feature sequence to obtain the action category corresponding to each video frame in the video to be processed; Among them, the action segmentation model is obtained by model training based on the sample video and the action category labels of each video frame in the sample video, boundary classification results and action classification results; the boundary classification result is to divide the enhanced feature vectors in the logarithmic context in the enhanced feature sequence of the sample video into multiple logarithmic context levels, obtain the logarithmic context feature sequence corresponding to the sample video according to the enhanced feature vectors divided into the same level, and predict it according to the similarity between the logarithmic context feature sequence and the enhanced feature sequence of the sample video; the action classification result is predicted based on the enhanced feature sequence of the sample video; the enhanced feature sequence of the sample video is obtained by feature enhancement processing of the feature vector sequence of the sample video.

Images