Video processing method and related apparatus

Abstract

Provided in the embodiments of the present application are a video processing method and a related apparatus, which are used for optimizing the matting effect of a video. The method comprises: acquiring at least one key frame among at least one first video frame; inputting the at least one first video frame into a first neural network model, so as to obtain at least one second video frame and the confidence of each second video frame; on the basis of the confidence of the at least one second video frame, screening the at least one first video frame to select at least one third video frame; in response to an operation for at least one frame among the at least one key frame or the at least one third video frame, correcting the at least one frame to obtain at least one fourth video frame; inputting the at least one fourth video frame into the first neural network model for training, so as to obtain a second neural network model; and inputting the at least one first video frame into the second neural network model, and outputting at least one fifth video frame.

Description

Video processing method and related apparatus

[0001] This application claims priority to the Chinese patent application No. 202411375327.7, filed on September 29, 2024, and entitled “Video processing method and related apparatus”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular, to a video processing method and related apparatus. BACKGROUND

[0003] Video background removal technology is commonly used in film and television production. The producer mixes the extracted person or object with a new background to form a new picture or video. When performing video matting, a large number of pictures need to be extracted, and the continuity between video frames needs to be ensured.

[0004] Currently, some video processing software can be used to segment the person or object in the video with a pure color background, including green screen or blue screen background. However, this method has poor matting effect in non-pure color backgrounds. In natural scenes and other non-pure color backgrounds, a deep learning-based matting model is usually used to automatically mat the video.

[0005] However, the effect of the deep learning-based automatic matting model is greatly affected by the training data set. It is difficult for the training data to cover all scenes, resulting in poor matting effect of the model in some scenes. SUMMARY

[0006] The present application provides a video processing method and related apparatus. The method uses a neural network model to perform video matting, filters out video frames with poor matting effect and corrects them. The corrected video frames are used as a training set to train the model, obtain an optimized neural network model, and use the optimized neural network model to perform video matting, so that the output video has better matting effect.

[0007] Therefore, in a first aspect, the present application provides a video processing method. In the method, a key frame is obtained from at least one first video frame. The key frame is a video frame with representative and large amount of information in the video. For example, the key frame can be a video frame with large motion change, a video frame with more information of the main content of the picture, etc.

[0008] The original video (at least one first video frame) is input into a deep learning-based neural network model, which performs matting processing on the original video, i.e., separates the target object from the background in the first video frame to obtain at least one second video frame and the confidence of each second video frame. The first video frame includes R channel values, G channel values and B channel values, and the second video frame includes R channel values, G channel values, B channel values and alpha channel values. The target object includes the main part, the foreground part or the part that needs to be cut out of the video frame, for example, a person, an animal, a building, etc. in the video frame, and the second video frame includes the target object part after matting. One second video frame corresponds to one first video frame. The confidence is used to represent the degree of certainty of the model for the output second video frame. The greater the confidence, the greater the possibility that the model predicts the second video frame accurately.

[0009] The first video frame is screened according to the confidence of the second video frame to obtain a third video frame. Further, the first video frame corresponding to the second video frame with a confidence less than a first threshold value can be taken as the third video frame to obtain a frame with a low degree of certainty of the matting result of the model.

[0010] The frame confirmed by the user to have a problem is further screened from the key frame or the third video frame, and the frame with the problem is corrected to obtain a fourth video frame. The correction operation includes steps of denoising, separating, smoothing, feathering the edge of the target object, etc. The fourth video frame after correction has a more detailed matting effect.

[0011] The fourth video frame is taken as a training set and input into the neural network model for training to obtain an optimized neural network model. The original video (at least one first video frame) is input into the optimized neural network model for video matting processing to separate the target object from the original video to obtain a video after matting processing (at least one fifth video frame) and the confidence of each frame image. The fourth video frame includes the channel values of the frame image.

[0012] In the embodiments of the present application, the key frame is obtained from the first video, the key frame having a representative or important role in the video is screened, the first video frame is screened according to the confidence of the output second video frame to obtain a third video frame, further, the frame confirmed by the user to have a problem is screened from the key frame and the third video frame and corrected, the video frame after correction is taken as a training set and input into the matting model for training to optimize the parameters of the matting model, and an optimized matting model is obtained. The video is processed using the optimized matting model to improve the confidence of the output of the matting model and obtain a more detailed matting video, and the matting effect of the model is optimized. In addition, the video frame after correction is taken as a training set, even if the video scene changes, the video frame after correction in the changed scene can still be taken as a training set, so that the model can cover the current video scene.

[0013] In a possible implementation, the acquiring the at least one key frame in the at least one first video frame comprises: acquiring the at least one key frame in the at least one first video frame according to a type of the target object. For example, in a case where the target object is a person or an animal, or the like, which has a relatively large motion change posture, a key frame can be extracted by using a motion characteristic (for example, a speed or a displacement). In a case where the target object is a building, a vehicle, or other static object, or the like, which has a visual saliency effect, a key frame can be extracted by using a salient feature of the target object.

[0014] In the implementation of the present application, different manners are selected according to the type of the target object to acquire the key frame from the first video frame, so that the key frame can be acquired according to the characteristics and requirements of the target object, and the process of acquiring the key frame is optimized.

[0015] In a possible implementation, the acquiring the at least one key frame in the at least one first video frame according to the type of the target object comprises: in a case where the target object belongs to a first type, inputting the at least one first video frame into a third neural network model to obtain at least one dynamic key point of the at least one first video frame; and taking a first video frame in which a displacement or a speed of the dynamic key point exceeds a first threshold value as a key frame.

[0016] The target detection model is used to acquire the target object in the video, and the third neural network model (a key point detection model) is used to acquire a dynamic key point in the video. In a case where the target object is a person, a human pose estimation model can be used as the key point detection model to identify a dynamic key point (for example, each bone point of the human body) of the human body. In a case where the target object is an animal or other object which has a relatively large motion change posture, a neural network model can be trained by using training data to obtain a key point detection model for the target object, so as to identify a dynamic key point of the target object. The at least one first video frame is input into the third neural network model to obtain at least one dynamic key point of the target object.

[0017] The displacement or the speed of the dynamic key point in the pixel coordinate system is determined by using an optical flow algorithm, and the optical flow algorithm includes a Lucas-Kanade optical flow algorithm, a Horn-Schunck optical flow algorithm, or the like. A video in which the dynamic key point of the target object is detected is input into the optical flow algorithm to obtain a displacement or a speed of the dynamic key point corresponding to a current frame and a previous frame. A first video frame in which the displacement or the speed exceeds a first threshold value is taken as a key frame. The first threshold value is used to determine whether the motion of the first video frame is significant, and can be adjusted according to actual application.

[0018] In the embodiments of the present application, in the case that the target object belongs to the first type, the dynamic key points of the target object are identified by using the neural network model, the dynamic key points are taken as points for measuring the motion characteristics of the target object, and in the case that the speed or displacement of the dynamic key points exceeds the first threshold value, the video frame is taken as a key frame, so that an important action of the target object with a changeable posture, such as a human or an animal, can be better captured, and the frame with a larger action change amplitude is taken as a key frame.

[0019] In a possible implementation, the at least one key frame in the at least one first video frame is obtained according to the type of the target object, including: in the case that the target object belongs to the second type, the at least one first video frame is input into a fourth neural network model to obtain a salient object region of the at least one first video frame; and the first video frame with a feature difference degree of the salient object region exceeding a second threshold value is taken as a key frame.

[0020] In the case that the target object belongs to the second type, a key frame can be extracted from the first video frame by using a content analysis-based manner. The second type includes static objects such as buildings, cups, and flowerpots. The static objects are widely classified, and different static objects can be distinguished by using a salient object detection model.

[0021] The content analysis-based manner for extracting the key frame can include the following steps: inputting the at least one first video frame into a fourth neural network model (a salient object detection model) to obtain a salient object region of the at least one first video frame; obtaining a feature difference degree of the salient object region between different first video frames; and selecting the first video frame with the feature difference degree exceeding a second threshold value as a key frame.

[0022] The at least one first video frame is input into the salient object detection model to obtain a target detection map of the salient object region. The target detection map includes a salient object region corresponding to the target object, and the target detection map can be represented as a mask map, a label map, or an image that only retains the salient object region.

[0023] The feature difference degree of the salient object region between the first video frames is obtained, including: extracting feature points from the salient object region of each first video frame, matching the feature points of adjacent first video frames by using a feature matching algorithm, and calculating the feature difference degree by using a Euclidean distance or other distance calculation method.

[0024] The difference degree of the color histogram of the salient object region between the first video frames is obtained, and the difference degree of the color histogram is used to compare the distribution of different colors in the salient object region. The difference degree of the color histogram can be obtained by the following steps: first, the color histogram of each first video frame is calculated, and different color spaces such as RGB and HSV can be used. Then, the difference of the histograms between adjacent first video frames is calculated using a histogram comparison method. The histogram comparison method includes histogram intersection, Bhattacharyya distance, Manhattan distance, etc.

[0025] The difference degree of the edge feature of the salient object region between the first video frames is obtained. The edge feature is used to detect the structural information of the image, and the difference degree of the edge feature can be obtained by the following steps: first, an edge detection algorithm such as Canny edge detection is applied to each first video frame to obtain the edge feature. Then, the difference degree between the edge features can be calculated using a similarity index or a mean square error.

[0026] The first video frame with a feature difference degree exceeding a threshold is taken as a key frame. The first video frame corresponding to the difference degree of the feature value exceeding the feature value threshold can be taken as the key frame, the first video frame corresponding to the difference degree of the color histogram exceeding the histogram threshold can be taken as the key frame, and the first video frame corresponding to the difference degree of the edge feature exceeding the edge feature threshold can be taken as the key frame, and the specific implementation is not limited here.

[0027] In the embodiments of the present application, the salient object region in the video is obtained by a salient object detection model, and the feature difference degree of the salient object region between different video frames is analyzed, and the video frame with a feature difference degree exceeding a threshold is taken as a key frame, thereby providing a way to obtain key frames from videos for different static objects.

[0028] In a possible implementation, the feature difference degree of the salient object region includes the difference degree of the feature value, the difference degree of the color histogram, or the difference degree of the edge feature.

[0029] In the embodiments of the present application, the difference degree of the feature value can be used to analyze the feature difference of the salient object region at the pixel level, the difference degree of the color histogram can be used to analyze the color feature of the region, and the difference degree of the edge feature can be used to analyze the boundary contour of the region, thereby capturing the structural and shape information of the region. The difference degree includes the difference degree of the feature value, the difference degree of the color histogram, or the difference degree of the edge feature, and the features of the salient object region are compared in different ways.

[0030] In a possible implementation, inputting the at least one first video frame into the first neural network model to obtain the at least one second video frame and the confidence of each second video frame includes: inputting the at least one first video frame into the first neural network model to obtain the at least one second video frame, each second video frame including at least one first pixel point; and calculating the mean of the confidence of all first pixel points in each second video frame to obtain the confidence of each second video frame.

[0031] In the implementation, after inputting the at least one first video frame into the neural network model for matting processing, the second video frame and the confidence of each pixel point in the second video frame are obtained, the sum of the product of the confidence and the weight of all pixel points in the second video frame is calculated to obtain the confidence of the second video frame. By calculating the confidence in the manner of the sum of the product of the confidence and the weight, the weight of an important region (for example, a region where a target object is located) can be increased, so that the important region contributes more to the confidence of the video frame.

[0032] In a possible implementation, inputting the at least one first video frame into the first neural network model to obtain the at least one second video frame and the confidence of each second video frame includes: inputting the at least one first video frame into the first neural network model to obtain the at least one second video frame, each second video frame including at least one first pixel point; and obtaining the confidence of each second video frame according to the confidence and the weight of the at least one first pixel point in each second video frame.

[0033] In the implementation, after inputting the at least one first video frame into the neural network model for matting processing, the second video frame and the confidence of each pixel point in the second video frame are obtained, the sum of the product of the confidence and the weight of all pixel points in the second video frame is calculated to obtain the confidence of the second video frame. By calculating the confidence in the manner of the sum of the product of the confidence and the weight, the weight of an important region (for example, a region where a target object is located) can be increased, so that the important region contributes more to the confidence of the video frame.

[0034] In a possible implementation, the at least one third video frame is selected from the at least one first video frame according to the confidence of the at least one second video frame, including: taking, as the third video frame, the first video frame corresponding to the second video frame whose confidence is less than a third threshold value.

[0035] In the implementation, the first video frame corresponding to the second video frame whose confidence is less than the third threshold value is taken as the third video frame, so that the first video frame corresponding to the matting result with low confidence can be selected, so as to perform fine matting on the selected first video frame in the future.

[0036] In a possible implementation, the first neural network model is an encoder-decoder structure, the output of the decoder includes the R channel value, the G channel value, the B channel value, the alpha channel value and the confidence of the second video frame, and the first neural network model includes a restnet, a mobilenet, an LRASSP or a convGRU network structure.

[0037] The first neural network model is trained based on deep learning, and the model is used for matting operation on a video to separate a target object from a background in a first video frame, and outputs the R channel value, the G channel value, the B channel value, the alpha channel value of the video frame after matting and the confidence of the video frame after matting. The confidence is the confidence degree of the prediction result of the model, that is, the confidence degree of each channel value of the output video frame after matting.

[0038] A feasible structure of the model is to use an encoder-decoder structure. The encoder extracts multi-level features of the first video frame, captures high-dimensional features through layer-by-layer down-sampling, and can use a classical deep learning module such as restnet or mobilenet as the encoder. The decoder recovers the spatial resolution of the image through layer-by-layer up-sampling. The LRASSP layer can be introduced during decoding. The LRASSP layer uniformly divides the input into 1, 4 and 16 parts in terms of width and height, respectively, and performs global mean pooling on each part. Finally, the pooled features are spliced together to obtain the scale information of the feature map in different spaces, thereby enhancing the robustness of the network.

[0039] Meanwhile, the convGRU is used to extract the continuous features between frames, consider the features of the previous video frames, apply convolution operation in space and introduce the gating mechanism. The convGRU can better capture and stabilize the dynamic information of the video, which helps to reduce the matting error caused by the change between frames, make the alpha channel value change smoothly, and stabilize the segmentation effect of the continuous frames of the video.

[0040] In a possible implementation, the pooling layer of the first neural network model includes other spatial pyramid pooling structures such as SPP, ASPP, DSPP and MDSPP.

[0041] In the embodiments of the present application, the pooling layer introduces spatial pyramid pooling structures such as SPP, ASPP, DSPP, MDSPP and LRASSP, which can process the pooling operation of the convolutional neural network with different sizes of input. The input video frame is divided into different levels, each level uses a different size of the pooling window for the pooling operation, and then the pooling results of all levels are spliced together as the feature representation of the network. Different sizes of the pooling level are combined, so that the pooling operation can be performed on the input image of any size, and the information of the image can be better preserved.

[0042] In a second aspect, the present application provides a video processing apparatus, comprising:

[0043] an acquisition unit configured to acquire at least one key frame in at least one first video frame;

[0044] an image separation unit configured to input the at least one first video frame into a first neural network model to obtain at least one second video frame and a confidence degree of each second video frame, the first neural network model being configured to perform image separation on a target object in the at least one first video frame;

[0045] a screening unit configured to screen at least one third video frame from the at least one first video frame according to the confidence degree of the at least one second video frame;

[0046] a correction unit configured to correct at least one frame of the at least one key frame or the at least one third video frame to obtain at least one fourth video frame in response to an operation on the at least one frame;

[0047] a training unit configured to input the at least one fourth video frame into the first neural network model to train the first neural network model to obtain a second neural network model;

[0048] an output unit configured to input the at least one first video frame into the second neural network model to output at least one fifth video frame.

[0049] In some optional embodiments, the acquisition unit is specifically configured to acquire the at least one key frame in the at least one first video frame according to the target object.

[0050] In some optional embodiments, the acquisition unit is further configured to, in a case where the target object belongs to a first type, input the at least one first video frame into a third neural network model to obtain at least one dynamic key point of the at least one first video frame; and take a first video frame in which displacement or speed of the dynamic key point exceeds a first threshold value as the key frame.

[0051] In some optional embodiments, the acquisition unit is further configured to, in a case where the target object belongs to a second type, input the at least one first video frame into a fourth neural network model to obtain a salient object region of the at least one first video frame; and take a first video frame in which feature difference degree of the salient object region exceeds a second threshold value as the key frame.

[0052] In some optional embodiments, the feature difference degree comprises one or more of the following: difference degree of feature value, difference degree of color histogram, and difference degree of edge feature.

[0053] In some optional embodiments, the image separation unit is specifically configured to input the at least one first video frame into the first neural network model to obtain at least one second video frame, each second video frame including at least one first pixel point; and calculate a mean value of the confidence of the at least one first pixel point in each second video frame to obtain the confidence of each second video frame.

[0054] In some optional embodiments, the image separation unit is further configured to input the at least one first video frame into the first neural network model to obtain at least one second video frame, each second video frame including at least one first pixel point; and obtain the confidence of each second video frame according to the confidence and the weight of the at least one first pixel point in each second video frame.

[0055] In some optional embodiments, the screening unit is specifically configured to take the first video frame corresponding to the second video frame with the confidence less than the third threshold value as the third video frame.

[0056] In some optional embodiments, the first neural network model is an encoder-decoder structure, the output of the decoder includes the R channel value, the G channel value, the B channel value, the alpha channel value and the confidence of the second video frame, and the first neural network model includes a restnet, a mobilenet, an LRASSP or a convGRU network structure.

[0057] The video processing apparatus in the second aspect can perform the operations performed by the first aspect or any optional embodiment of the first aspect and achieve the same technical effects, and details are not repeated here.

[0058] In the third aspect, the present application provides a computing device including a processor and a memory, the processor being coupled to the memory, and the memory being configured to store programs or instructions, when the programs or instructions are executed by the processor, causing the computing device to perform the method in the first aspect or any optional embodiment of the first aspect.

[0059] In the fourth aspect, the present application provides a computing device cluster including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute the instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method in the first aspect or any optional embodiment of the first aspect.

[0060] In the fifth aspect, the present application provides a computer program product including instructions, when the instructions are executed by the computing device cluster, the computing device cluster performs the method in the first aspect or any optional embodiment of the first aspect.

[0061] In a sixth aspect, the present application provides a computer-readable storage medium, comprising computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect or any optional implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0062] FIG. 1 is a schematic diagram of an electronic device provided by the present application;

[0063] FIG. 2 is a schematic diagram of an application scenario provided by the present application;

[0064] FIG. 3 is a schematic diagram of a system architecture provided by the present application;

[0065] FIG. 4 is a flowchart of a video processing method provided by the present application;

[0066] FIG. 5 is a schematic diagram of a significant object region provided by the present application;

[0067] FIG. 6 is a schematic diagram of a structure of a video processing model provided by the present application;

[0068] FIG. 7 is a flowchart of a video processing method provided by the present application;

[0069] FIG. 8 is a schematic diagram of a structure of a video processing apparatus provided by the present application;

[0070] FIG. 9 is a schematic diagram of a structure of a computer device provided by the present application;

[0071] FIG. 10 is a schematic diagram of a structure of a computing device cluster provided by the present application;

[0072] FIG. 11 is a schematic diagram of another structure of a computing device cluster provided by the present application. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. As technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0074] The terms "first", "second", and the like in the description, claims, and drawings of the application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of use in either order. Moreover, the terms "include", "have", and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, system, product or apparatus.

[0075] First, the embodiments of the present application relate to a large number of neural network related applications, in order to better understand the scheme of the embodiments of the present application, the following first to the neural network related terms and concepts that may be involved in the embodiments of the present application are introduced.

[0076] (1) Neural network

[0077] The neural network can be composed of neural units, and the neural unit can refer to an operation unit taking x s (i.e. input data) and intercept 1 as input, and the output of the operation unit can be:

[0078] Wherein, s = 1, 2, … n, n is a natural number greater than 1, W s is the weight parameter of x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next layer of convolution layer, and the activation function can be sigmoid function. The neural network is a network formed by connecting a plurality of the above single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neural units.

[0079] (2) Loss function

[0080] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is actually intended to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the target value that is actually intended to be predicted (of course, the parameters of each layer in the deep neural network are usually pre-configured before the first update), for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the deep neural network can predict the target value that is actually intended to be predicted or a value very close to the target value that is actually intended to be predicted. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the loss. The loss function can usually include error square mean square, cross entropy, logarithm, exponential, etc. For example, the error mean square can be used as the loss function, which is defined as Specifically, a specific loss function can be selected according to an actual application scenario.

[0081] (3) Back propagation algorithm

[0082] An algorithm for calculating the gradient of the model parameters according to the loss function and updating the model parameters. The neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal will produce an error loss, and the initial neural network model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0083] In the training phase of the model, the BP algorithm can be used to train the model to obtain the trained model.

[0084] (4) Attention (also known as attention mechanism)

[0085] Attention mechanism can quickly extract important features of sparse data. Attention mechanism occurs between encoder and decoder, or between input sentence and generated sentence. While self-attention mechanism in self-attention model occurs within the encoding matrix, or within the output sequence, which can extract the relationship between words far apart in the same sentence, such as syntactic features (phrase structure). Self-attention mechanism provides an effective modeling method for capturing global context information through QKV. Assuming the input is Q (query), and the context is stored in the form of key-value pair (K, V). Then the attention mechanism is actually a mapping function from query to a series of key-value pairs. The essence of attention function can be described as a mapping from query to a series of (key, value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in the form of (K, V), then attention completes the addressing by calculating the similarity between Q and K. The similarity calculated by Q and K reflects the importance of the extracted V value, that is, the weight, and then the weighted sum is obtained. The final feature value.

[0086] The calculation of attention mainly includes three steps. The first step is to calculate the similarity between query and each key to obtain the weight. The commonly used similarity function is dot product, concatenation, perception, etc. Then the second step is to normalize these weights using a softmax function (on the one hand, it can be normalized to get a probability distribution whose sum of all weight coefficients is 1. On the other hand, the characteristic of softmax function can highlight the weight of important elements); Finally, the weighted sum of the weight and the corresponding key value value is obtained to get the final feature value. The specific calculation formula can be as follows:

[0087] Where d is the dimension of the matrix QK.

[0088] In addition, attention includes self-attention and cross-attention. Self-attention can be understood as a special attention, that is, the input of QKV is consistent. While the input of QKV in cross-attention is inconsistent. Attention uses the similarity (such as inner product) between features as a weight to integrate the queried features as the updated value of the current feature. Self-attention is the attention extracted based on the attention of the feature map itself.

[0089] For convolution, the setting of the convolution kernel limits the size of the receptive field, resulting in the network often needing multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is that its attention is global, and it can obtain the global spatial information of the feature map through simple query and assignment. The special point of self-attention in the query, key, and value (QKV) model is that the input corresponding to QKV is consistent.

[0090] (5) Raw data: Raw data records the original information of the image sensor, which is in an unprocessed and uncompressed format. RAW can be conceptualized as "original image encoding data" or more figuratively as "digital negative".

[0091] (6) R (red), G (green), B (blue)

[0092] Wherein, R represents red, G represents green, and B represents blue. Each image can be represented by color values of the three channels. For example, an RGB image represents an image with three color channels, and an RGGB image represents an image with four color channels, two of which are G.

[0093] Video matting is widely used in video editing, special effect making, augmented reality, etc. Video matting is used to separate specific target objects from video frames, to remove the background in the video frame, and to highlight or extract the target region of interest.

[0094] The embodiment of the present application provides a video processing method, which is used for improving the matting effect of the video. The embodiment of the present application also provides a video processing device, an electronic device, a computer readable storage medium and a computer program product.

[0095] The method provided by the present application can include various deployment modes. For example, the method provided by the present application can be deployed in an electronic device, and a user can directly use the electronic device to process a video, or the method can be deployed in a cloud platform to provide a video processing service for a user terminal. Different deployment modes will be introduced below.

[0096] Deployment mode one, deployed in an electronic device

[0097] The electronic device provided in the embodiments of the present application can specifically include a handheld device, a vehicle-mounted device, a computing device, and the like, and can include an image sensor or be connected with an image sensor. The electronic device can also include a digital camera, a cellular phone, a smart phone, a personal digital assistant (PDA) computer, a tablet computer, a laptop computer, a machine type communication (MTC) terminal, a point of sales (POS), a vehicle-mounted computer, a head-mounted device, a data processing apparatus (such as a bracelet, a smart watch, and the like), a security device, a virtual reality (VR) device, an augmented reality (AR) device, and other electronic devices with an imaging function.

[0098] Taking a digital video camera as an example, the digital video camera is a camera that converts an optical image into a digital signal by using a photoelectric sensor, and includes a single-lens reflex camera, an industrial camera, and a high-speed camera. Unlike a traditional camera that relies on the change of a photosensitive chemical substance on a film to record an image, the sensor of the digital video camera is a light-sensitive charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). Compared with the traditional camera, the digital video camera has the advantages of being more convenient, faster, repeatable, and more timely, because it directly uses an image sensor for photoelectric conversion. With the development of CMOS processing technology, the function of the digital video camera is becoming more and more powerful, and it has almost completely replaced the traditional film camera, and has extremely wide applications in the fields of consumer electronics, human-computer interaction, computer vision, and automatic driving.

[0099] Exemplarily, FIG. 1 is a schematic diagram of an electronic device provided in the present application. The electronic device can include a lens group 110, a sensor 120, and an electrical signal processor 130. The electrical signal processor 130 can include an analog-to-digital (A / D) converter 131 and a digital signal processor 132. The analog-to-digital converter 131 is an analog signal-to-digital signal converter, which is used to convert an analog electrical signal into a digital electrical signal. The sensor 120 can specifically include an image sensor, a multispectral image (MSI) sensor, and the like.

[0100] It should be noted that the electronic device shown in FIG. 1 is not limited to including the above devices, and can further include more or less other devices, such as a battery, a flash, a key, a sensor, etc. The embodiments of the present application are only illustrated by taking the electronic device with the sensor 120 as an example, but the elements installed on the electronic device are not limited thereto.

[0101] The light signal reflected by the object is converged by the lens group 110 and imaged on the sensor 120. The sensor 120 converts the light signal into an analog electric signal. The analog electric signal is converted into a digital electric signal by an analog-to-digital (A / D) converter 131 in the electric signal processor 130, and the digital electric signal is processed by a digital signal processor 132, for example, by a series of complex mathematical algorithm operations, to optimize the digital electric signal, and finally output an image. The electric signal processor 130 can also include an analog signal preprocessor 133 for pre-processing the analog electric signal transmitted by the image sensor and outputting the analog electric signal to the analog-to-digital converter 131.

[0102] The method provided by the present application can be specifically deployed in the electric signal processor 130 of the electronic device, for example, can be specifically deployed in the digital signal processor 132, or can be deployed in other processors of the electronic device. The image or video collected by the sensor 120 is input into the neural network model for automatic matting, the target object in the video is separated from the background part, the video frames and part of the key frames with low confidence are screened out, the screened frames are corrected, the corrected frames are taken as a training set, the neural network model is trained, an optimized neural network model is obtained, and the original video is matting by using the optimized neural network model, so that the matting effect of the model is better.

[0103] Deployment mode two, deployed in a cloud platform

[0104] The method provided by the present application can also be deployed in a cloud platform, and one or more terminals access the platform to provide image or video matting services for the terminals.

[0105] Please refer to FIG. 2, which is an application scenario of the video processing method in the embodiments of the present application. The scenario can include a cloud platform 11 and a terminal 12, and the cloud platform 11 and the terminal 12 can be connected by wire or wirelessly.

[0106] The cloud platform 11 can specifically include a server cluster with storage and processing functions. The method provided in the embodiments of the present application can be deployed in the cloud platform 11, and specifically can receive a video from the terminal 12, filter out key frames in the video, perform background subtraction on the video by using a neural network model, and output the confidence of the video frames. Further, according to the key frames and the confidence, the user filters out the video frames that need to be corrected, and corrects the video frames, adds the corrected video frames to a training set under the current scene, trains the neural network model, obtains an optimized neural network model, performs matting on the original video again, obtains the video after matting, and feeds back the video after matting to the terminal 12.

[0107] The terminal 12 can realize image processing by interacting with the cloud platform 11. The terminal can specifically include a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, a smart car, and the like. The terminal 12 can transmit a video to the cloud platform 11, and the video can include a video captured by the terminal itself, a video input by a user, a video stored locally in the terminal, and the like. For example, the cloud platform can provide services for the user through a client deployed in the terminal or a web page in the terminal, and taking the deployment of the client in the terminal as an example, the user can send a video collected by the terminal to the cloud platform through the client deployed in the terminal. The cloud platform 11 processes the original video by using the method provided in the present application, and feeds back the video after matting to the terminal 12.

[0108] In a possible scenario, the method can also be applied to a scenario of multiple terminals. For example, a user can use another terminal different from the terminal 12 to capture the aforementioned video, transmit the video to the terminal 12, upload the video to the cloud platform 11 by the terminal 12, perform matting processing on the video by the cloud platform 11, and feed back the video after matting to the terminal 12, and the terminal 12 feeds back the video after matting to the terminal that captures the original video.

[0109] The above describes the deployment of the video processing method in the present application, and the system architecture based on which the video processing method in the present application is described below.

[0110] In combination with FIG. 3, the system architecture based on which the embodiments of the present application are described includes a data acquisition module, a data processing module, a model training and inference module, an automatic filtering module, a visualization and user interaction module, and the like.

[0111] Video data is acquired by a data acquisition device, and the data acquisition device includes a high-speed camera, a single-lens reflex camera, an industrial camera, a general camera, and the like. The data acquisition device is described in detail in the foregoing FIG. 1, and will not be described here again.

[0112] The video data obtained from the server, the local storage or the data acquisition device is input into the data processing module for pre-processing, such as calibration of the video to obtain a mask of the video.

[0113] The original video is input into the automatic screening module, and different ways are adopted to obtain the key frames in the video according to different target objects in the video. The target objects include the main part, the foreground part or the part to be cut out of the video frame.

[0114] The original video is input into the neural network model for video matting operation, and the video frame after matting and the confidence of each video frame are output. The automatic screening module screens the video frame according to the confidence of the video frame.

[0115] In the user interaction module, the user confirms whether the key frame and the video frame screened according to the confidence need to be corrected, further screens the video frame to be corrected and corrects the video frame, adds the corrected video frame to the training set of the neural network model, trains the neural network model, and obtains an optimized neural network model. The original video is input into the optimized neural network model for video matting processing, and the video after matting is output.

[0116] Please refer to FIG. 4, which is a flowchart of a video processing method provided by the present application.

[0117] 401. Obtain at least one key frame in at least one first video frame.

[0118] A video is a sequence of images, including a plurality of video frames, and each video frame is a static image. The file format of the video includes MP4, MKV, AVI, etc. The video can be obtained from various sources, such as from a local storage, from a server or from an image acquisition device.

[0119] Obtaining at least one first video frame includes obtaining all video frames in the video or obtaining part of the video frames in the video. The video frames can be extracted at a fixed interval or according to actual needs.

[0120] The first video frame can be an image captured by the electronic device or an image read from the storage.

[0121] For example, if the method provided by the embodiment of the present application is deployed in an electronic device, the first video frame can be captured by the electronic device. The electronic device can refer to the description of the corresponding description of the foregoing FIG. 1, which will not be repeated here.

[0122] For another example, if the method provided by the embodiment of the present application is deployed in a cloud platform, the first video frame can be an image received from a terminal. For example, a user can input an image in the terminal, or use the terminal to capture an image, and send the image to the cloud platform through the terminal.

[0123] Specifically, the first video frame can include an image of at least one channel. For example, it can include a raw image, an RGB image, a YUV image, etc., to use a variety of visible light denoising scenarios.

[0124] Optionally, in the process of obtaining the first video frame, after obtaining the video collected by the image sensor, a pre-processing operation such as video alignment or color space conversion can be performed to obtain a first video frame that conforms to the preset specification, so as to facilitate further processing of the first video frame.

[0125] The key frame is a video frame with representative and large amount of information in the video. For example, the key frame can be a video frame with large motion change, a video frame with more information of the main content of the picture, etc.

[0126] For different video matting tasks, the target object to be cut in the video is different, and different ways can be taken to obtain the key frame. The target object includes a specific target or a foreground object separated from the video frame. For example, the scene of the video includes a person in a running state passing by a building, and the target object can be determined as the person, the building, or a combination of the two. For a person in a motion state in the video, a motion analysis based method can be taken to obtain the key frame. For the building in the video, a content analysis based method can be taken to obtain the key frame.

[0127] In the case where the target object belongs to a first type, a motion analysis based method can be taken to extract the key frame. The first type includes a target object with multiple changeable postures such as a person and an animal. The motion analysis based method for extracting the key frame can be briefly described as follows: first, the dynamic key points in the video are obtained according to a target detection model. Then, the motion information features of the dynamic key points in each frame of image are calculated, and the key frame is selected according to a certain strategy.

[0128] A method for obtaining a key frame from a first video frame will be described in detail below.

[0129] S1, input at least one first video frame into a third neural network model to obtain at least one dynamic key point of the at least one first video frame.

[0130] The dynamic key point includes a key position or a feature point changing over time in a video sequence, and the third neural network model is a target detection model. A suitable target detection model can be used to identify a target object in a video, such as a sliding window detection model, YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Regions with CNN features), and the like. Some target detection models are described below.

[0131] The sliding window detection model scans each pixel of an image through a sliding window, regards each window as a potential target object, and classifies it by using a classifier. First, a fixed-size window is defined, the size of the window is approximately the same as the size of a target object to be detected in each frame of image, the window is placed at the top left corner of the image, and the image in the anchor window is obtained. The image in the anchor window is classified to determine whether it contains a target object. Then, the window is moved to the right or down by a certain step, which can be half of the width or height of the window, or a smaller value, and the image in the new window is classified. Repeat the above steps until the window slides across the entire image to identify the target object in the video frame.

[0132] YOLO uses an end-to-end training method to directly regress the position and class information of an object on the entire image. Each frame of input image is divided into S*S grids. For each grid, B bounding boxes are predicted, each of which includes the confidence of being a target object and the probability of each class. S*S*B target windows are predicted, and the target windows with low confidence are removed by combining the threshold, and the remaining target windows include the target objects identified in the video frame.

[0133] R-CNN extracts image features through a convolutional neural network and uses a support vector machine or a fully connected layer for classification and regression. First, 1000 to 2000 candidate regions of each frame of image are proposed by using the SS algorithm. Each candidate region is wrapped and processed, and the processed image is input into a convolutional neural network for feature extraction to obtain a feature vector of the candidate region. The feature vector is input into the SVM of each class for classification to determine whether it belongs to the current class. The classified candidate regions are classified by linear regression to obtain more accurate target windows, and the target windows include the target objects identified in the video frame.

[0134] After obtaining the target object in each frame of image, key point detection is performed on the target object. Key point detection is used to locate the positions of feature points of the target object, and these feature points have important structural features of the target object and can be used to describe the pose changes of the target object, such as the joint points (elbows, knees, shoulders, etc.) of a person, the facial feature points (eyes, nose, mouth, etc.) of a person, or the key points of an object (such as the wheels and door handles of a car).

[0135] In actual applications, when the target object is a person, the positions of various joints or skeletal points of the human body can be determined by means of a human pose estimation model. When the target object is an animal or other object with multiple changing poses, a large amount of data set of the target object can be used to train a neural network model to obtain a key point detection model of the target object, so as to extract the key points of the animal or other object through the trained key point detection model. The acquisition of the key points of the target object will be introduced in detail below.

[0136] When the target object is a person, the image corresponding to at least one first video frame or the target window obtained in the foregoing step can be input into a human pose estimation model to obtain multiple human pose key points. The human pose estimation model includes OpenPose, DWPose, MoveNet, PoseNet, etc.

[0137] Taking DWPose as an example, DWPose adopts a two-stage distillation strategy and can predict the pose key points of the whole human body. First, the distillation of the first stage learns the basic key point detection by training a guide model (such as OpenPose). Then, in the second stage, the knowledge of this guide model is further transferred to a more complex network structure to achieve higher-precision positioning. The DWPose model adopts an Encoder-Decoder architecture, in which the Encoder is used to extract image features and the Decoder is used to generate a heat map of the key point positions. In the Decoder, a multi-scale feature fusion technology is adopted to fuse features of different scales to improve the adaptability to human bodies of different sizes and poses. In addition, DWPose also introduces context information extraction and attention mechanisms to further enhance the performance of the model.

[0138] In the case that the target object is an animal or an object, a large amount of image data of the target object is acquired, and the image data is preprocessed, the preprocessing steps including calibrating, cropping, scaling, normalizing, etc. of the image, taking the processed data as a data set, and dividing the data set into a training set, a validation set and a test set, training the training set data by means of a key point detection model (for example, YOLO), adjusting the model parameters by means of an optimization algorithm to minimize the loss function, evaluating the model performance on the validation set, performing hyperparameter tuning and model optimization, evaluating the final performance of the model on the test set, checking the accuracy and robustness of the model, to obtain a trained neural network model, inputting the image or original video data corresponding to the target window obtained in the foregoing part into the trained neural network model, and obtaining a plurality of dynamic key points of the animal or object.

[0139] S2, taking the first video frame in which the displacement or speed of the dynamic key point exceeds the first threshold value as a key frame.

[0140] The first video frame in which the displacement or speed of the dynamic key point in the pixel coordinate system exceeds the first threshold value can be taken as a key frame. The pixel coordinate system represents the projection of a three-dimensional space object on an image plane, and is a coordinate system established based on a two-dimensional image. For example, the pixel coordinate system takes the upper left corner or center of the image plane as the origin of the coordinate system, and the position of each pixel is determined by the coordinates on the row (y-axis) and column (x-axis) where it is located, wherein the x-axis is parallel to the horizontal right of the display device, and the y-axis is perpendicular to the x-axis downward.

[0141] In actual application, the displacement or speed of the dynamic key point in the pixel coordinate system can be determined by means of an optical flow algorithm. The optical flow is the instantaneous speed of the pixel motion of a spatial moving object on the imaging plane, and when the time interval is very small (such as between two consecutive frames of a video), it is also equal to the displacement of the target point. The optical flow algorithm is a method of finding the correspondence between the previous frame and the current frame by means of the change of the pixels in the image sequence in the time domain and the correlation between adjacent frames, so as to calculate the motion information of the object between adjacent frames.

[0142] In practical applications, the current first video frame and corresponding dynamic key points, previous first video frames and corresponding dynamic key points, and other parameters are input into an optical flow algorithm to obtain the displacement or speed of the corresponding dynamic key points in the previous frames and the current frame, i.e., optical flow vectors. In the case of obtaining the displacement of the dynamic key points, the components of the displacement between the continuous video frames can be calculated to obtain the speed of the dynamic key points between the video frames. The first video frame with a displacement or speed exceeding a first threshold value is taken as a key frame. The first threshold value is used to determine whether the motion of the first video frame is significant, and can be adjusted according to actual experience. The optical flow algorithm includes Lucas-Kanade optical flow algorithm, Horn-Schunck optical flow algorithm, etc. The Lucas-Kanade optical flow algorithm is suitable for small-range motion estimation, and the Horn-Schunck optical flow algorithm is suitable for global optical flow estimation and is more suitable for large-range motion.

[0143] In practical applications, the maximum speed or maximum displacement of all dynamic key points of the first video frame can be taken as the judgment standard of the first video frame. When the maximum speed or maximum displacement exceeds the first threshold value, the first video frame is determined to be a key frame. The speeds or displacements of all dynamic key points in the first video frame can also be compared with the first threshold value. When the dynamic key points exceeding the first threshold value in the first video frame reach a certain proportion, the first video frame is determined to be a key frame, which is not limited here.

[0144] In summary, in the case that the target object belongs to the first type, the dynamic key points of the target object are recognized by using the neural network model, the dynamic key points are taken as points for measuring the motion characteristics of the target object, and the video frame is taken as a key frame when the speed or displacement of the dynamic key points exceeds the first threshold value. A kind of way can better capture the important actions of the person or the object with the change of the posture, and the frame with a large action change range is taken as a key frame.

[0145] In the case that the target object belongs to the second type, a content analysis based method can be used to extract key frames from the first video frame. The second type includes static objects such as buildings, cups, and flowerpots. The static objects are widely varied, and different static objects can be distinguished by using a salient object detection model.

[0146] The content analysis based method for extracting key frames can be briefly described as follows: performing salient object detection on at least one first video frame to obtain a salient object region of the at least one first video frame; and taking the first video frame with a feature difference of the salient object region exceeding a second threshold value as a key frame.

[0147] A method for obtaining key frames from first video frames will be described in detail below.

[0148] S1, input at least one first video frame into a fourth neural network model to obtain a salient object region of the at least one first video frame.

[0149] The salient object detection model based on a neural network can be used to obtain the salient object region, and the salient object region includes a region where the target object is located.

[0150] The salient object detection model is used to detect the location of the target object, and outputs a mask image or a label image, etc. of the target object. Each frame of image is ANDed with the mask image of the target object to obtain the salient object region corresponding to the target object.

[0151] The mask image is a binary image used to control the region of video processing and highlight the salient object region where the target object is located in the video frame. In salient object detection, each pixel of the mask image can include two values: one indicates that the pixel belongs to the salient object region, and the other indicates that the pixel does not belong to the salient object region. As shown in FIG. 5, in actual application, the pixel value in the salient object region is represented by 1, and the pixel value outside the salient object region is represented by 0 to obtain the mask image corresponding to the salient object region. The pixel value of the mask image is ANDed with the pixel value of the image to be processed, and the output image is an image that only retains the salient object region.

[0152] In addition, the mask image can also be multi-channel, wherein each channel is used to represent a different target object. For example, assuming that the target objects in the first video frame include a car and a tree, a two-channel mask image can be used to represent the two target objects. Channel 1 represents a binary mask of the car region, and when the pixel value of channel 1 is 1, it indicates that the target object corresponding to the pixel is a car. Channel 2 represents a binary mask of the tree region, and when the pixel value of channel 2 is 1, it indicates that the target object corresponding to the pixel is a tree. For example, when the value of channel 1 of a certain pixel is 1 and the value of channel 2 is 0, it can be determined that the target object corresponding to the pixel is a car. When the value of channel 1 of a certain pixel is 0 and the value of channel 2 is 0, it can be determined that the pixel is not in the salient object region and does not correspond to a target object.

[0153] The label image is used to divide the video frame into different semantic regions, and each pixel point in the label image includes the class or label of the target object at the position, such as a road, a building, etc. Further, a unique identifier can be assigned to each object belonging to the same class to distinguish different objects of the same class in the video frame. In actual application, according to the class of each pixel point in the label image, the pixel points where the target object is located are filtered, and the pixel value at the corresponding pixel point is set to 1. ANDing the pixel value with the pixel value of the first video frame can also obtain the salient object region corresponding to the target object.

[0154] The significant object detection model based on neural network includes sparse detection method, dense detection method, etc. The significant object detection model uses neural network, gradually refines the significant features in the video frame through a series of convolutional layers and pooling layers, and outputs the target detection map of the significant object region through the classification and regression module. The target detection map of the significant object region includes the mask map of the target object, the label map or the image only retaining the significant object region. The significant object detection model based on neural network is introduced below.

[0155] The sparse detection method includes SuperCNN, MCDL, ELD and other models. The model takes a small area in the video frame as a calculation unit, independently extracts features and judges the significant value of each small area, and the generated target detection map of the significant region usually has a fuzzy boundary.

[0156] The dense detection method includes models based on Encoder-Decoder structure, models combining superpixel method and encoding-decoding structure, models based on U-Net structure, etc.

[0157] The model based on Encoder-Decoder structure includes UCF model. The image features of the first video frame are extracted through the encoder, and the image information is decoded and reconstructed on this basis. The mask map of the target object is obtained through the softmax classifier. In practical application, time sequence can be added to the model based on Encoder-Decoder structure, so that the neural network is repeatedly trained at different times. The significant region target detection map output by the model at the last time is used as the prior knowledge for the model training at the next time, and the original video frame is sent into the encoder together, which reduces the loss of significant features in the video frame to a certain extent.

[0158] The model combining superpixel method and encoding-decoding structure, for example, Multi-Task DNN. The model first uses a fully convolutional network to capture the semantic information of the target object at different levels in the original video frame, thereby obtaining a preliminary significant region target detection map. Then, a nonlinear regression model based on superpixel segmentation map is used to refine and adjust the boundary of the initial significant region target detection map, thereby obtaining the final significant region target detection map with fine granularity. In practical application, the level set method can be introduced into the model. The level set is iteratively updated in the training process of the model, so as to minimize the energy function value of the model, so as to accurately segment the target object from the background and reduce the boundary blur.

[0159] A model based on the U-Net structure, for example, U2-Net, adopts a two-level nested U structure, so that the model training can be deeper, reaching high resolution, without significantly increasing the memory and computing cost, using RSU (ReSidual U-blocks) at the bottom of the model, which can extract intra-level multi-scale features without reducing the resolution of the feature map, and a structure similar to U-Net at the top, where each level is filled with RSU blocks.

[0160] It should be noted that the above significant object detection method is only a part of the example, and other methods for detecting the region where the target object is located can also be used, which is not limited here.

[0161] S2, analyze the feature difference degree of the significant object region between different first video frames.

[0162] The feature difference degree of the significant object region between different frames is analyzed by the inter-frame difference method. The inter-frame difference method compares the feature difference degree of two or three frames of images in time sequence. If the feature difference degree exceeds the threshold, it indicates that the target object reaches a certain motion amplitude. The feature difference degree includes the difference degree of feature value, the difference degree of color histogram, or the difference degree of edge feature.

[0163] The difference degree of feature value can analyze the feature difference of the significant object region at the pixel level. The difference degree of color histogram can be used to analyze the color features of the region. The difference degree of edge feature can be used to analyze the boundary contour of the region, capture the structure and shape information of the region, and compare the features of the significant object region from different angles.

[0164] The calculation of the feature difference degree is described below.

[0165] The difference degree of the feature value of the significant object region between the first video frames is calculated. Feature points are extracted from the significant object region of each first video frame, and the feature points of adjacent first video frames are matched using a feature matching algorithm. The feature matching algorithm includes brute force matching, FLANN, SIFT, SURF, etc. Then, the difference degree of the feature value between the matched feature points is calculated.

[0166] In practical applications, the Euclidean distance or other distance measurement methods can be used to calculate the feature value difference degree. For example, assuming that the feature value vector of video frame A is [0.1, 0.2, 0.3], and the feature value vector of video frame B is [0.4, 0.5, 0.6], the feature value difference degree between video frame A and video frame B is determined according to the Euclidean distance, and the feature value difference degree is

[0167] The difference degree of the color histogram of the salient object region between the first video frames is calculated. The difference degree of the color histogram is used to compare the distribution of different colors in the salient object region. The difference degree of the color histogram can be calculated by the following steps: first, the color histogram of each first video frame is calculated, and different color spaces such as RGB and HSV can be used. Then, the difference between the histograms of adjacent first video frames is calculated using a histogram comparison method. The histogram comparison method includes histogram intersection, Bhattacharyya distance, Manhattan distance, etc.

[0168] The difference degree of the edge feature of the salient object region between the first video frames is calculated. The edge feature is used to detect the structural information of the image, and the difference degree of the edge feature can be calculated by the following steps: first, an edge detection algorithm such as Canny edge detection is applied to each first video frame to obtain the edge feature. Then, the difference degree between the edge features is calculated. In practical applications, the structural similarity index can be used to calculate the difference degree between the edge features, and the closer the structural similarity index value is to 1, the smaller the difference is. The mean square error can also be used to calculate the difference degree between the edge features, and the smaller the mean square error is, the smaller the difference is. Or other methods that can calculate the difference degree of the feature, which are not limited here.

[0169] S3, selecting the first video frame with a feature difference degree exceeding a threshold value as a key frame.

[0170] When the feature difference degree exceeds the threshold value, it indicates that there is a large difference in the features of the salient object region between adjacent first video frames, and the first video frame can be selected as a key frame. The threshold value can be an empirically determined value.

[0171] In combination with the foregoing steps, the first video frame corresponding to the feature value whose difference degree exceeds the feature value threshold can be selected as a key frame, the first video frame corresponding to the color histogram whose difference degree exceeds the histogram threshold can be selected as a key frame, and the first video frame corresponding to the edge feature whose difference degree exceeds the edge feature threshold can be selected as a key frame, which are not limited here.

[0172] In summary, when the target object belongs to the second type, the salient object region in the video is obtained through the salient object detection model, and the feature difference degree of the salient object region between different video frames is analyzed. The video frame with a feature difference degree exceeding a threshold value is selected as a key frame, providing a way to obtain key frames from videos for different static objects.

[0173] 402, inputting at least one first video frame into a first neural network model to obtain at least one second video frame and a confidence degree of each second video frame, the first neural network model being used for image separation of a target object in the at least one first video frame.

[0174] The first neural network model is obtained based on deep learning training. The model separates the target object and the background in the first video frame through video matting operation. In combination with FIG. 6, a feasible structure of the model is to use an encoder-decoder structure. The encoder extracts multi-level features of the first video frame, captures high-dimensional features through layer-by-layer down-sampling, and can use classic deep learning modules such as restnet and mobilenet as the encoder. The decoder recovers the spatial resolution of the image through layer-by-layer up-sampling. The LRASSP layer can be introduced during decoding. The LRASSP layer divides the input into 1, 4 and 16 parts uniformly in terms of width and height, respectively, and performs global average pooling on each part. Finally, the pooled features are spliced together to obtain the scale information of the feature map in different spaces, thereby enhancing the robustness of the network. Meanwhile, the convGRU is used to extract the continuous features between frames, consider the features of the previous video frames, apply convolution operation in space and introduce gating mechanism. The convGRU can better capture and stabilize the dynamic information of the video, which helps to reduce the matting error caused by the change between frames, make the alpha channel value change smoothly, and stabilize the segmentation effect of the continuous frames of the video.

[0175] In actual application, the pooling layer can introduce the LRASSP structure, and can also introduce other spatial pyramid pooling structures such as SPP, ASPP, DSPP and MDSPP, which are not limited here. The spatial pyramid pooling structure is used for the pooling operation of the convolutional neural network processing different sizes of input, divides the input video frame into different levels, performs pooling operation on each level using different size of pooling window, and then splices the pooling results of all levels together as the feature representation of the network. Combining different sizes of pooling levels, the pooling operation can be performed on the input image of any size, and the image information can be better preserved.

[0176] The first video frame includes R channel value, G channel value and B channel value. The original video (at least one first video frame) is input into the first neural network model, and the R channel value, G channel value, B channel value, alpha channel value and confidence of each second video frame after video matting processing are output. The first video frame input into the model has a corresponding second video frame. The confidence is the confidence degree of the single prediction result of the matting model, which is usually expressed as a probability value. The greater the probability value, the greater the possibility of correct prediction of the model.

[0177] For example, the original video includes 5 frames of images. The original video is input into the neural network model, and the 5 frames of images after matting processing and the confidence of each frame of image are 0.89, 0.22, 0.31, 0.62 and 0.91, respectively. It is indicated that the confidence degree of the prediction result of the first frame and the fifth frame is high, the prediction accuracy is high, and the matting effect is good.

[0178] The process of inputting the first video frame into the neural network model to output the confidence is introduced below.

[0179] The at least one first video frame is input into the neural network model, and the second video frame corresponding to each first video frame is output, and the second video frame includes the confidence of each pixel point in the second video frame.

[0180] The mean value of the confidence of all pixel points in each second video frame can be calculated to obtain the confidence of the second video frame. By calculating the confidence by the mean value, the overall evaluation of the matting result of the video frame can be performed, and the higher the confidence, the higher the reliability of the matting effect of the model on the first video frame.

[0181] The sum of the product of the confidence and the weight of all pixel points in each second video frame can also be calculated to obtain the confidence of the second video frame. The neural network model can set the weight of the pixel point in combination with the position of the pixel point, and when the pixel point is in the region of the target object, a larger weight corresponding to the pixel point can be set. By calculating the confidence by the weight, the weight of the important region (such as the region of the target object) can be improved, so that the important region contributes more to the confidence of the video frame.

[0182] 403、According to the confidence of at least one second video frame, at least one third video frame is selected from at least one first video frame.

[0183] The second video frame obtained in the foregoing steps after the matting processing of the first video frame and the confidence of the second video frame are further filtered according to the confidence of the second video frame. When the confidence of the second video frame is less than a third threshold, the corresponding first video frame is taken as a third video frame. The third threshold is used to filter the first video frame, which can be adjusted according to actual application.

[0184] In the video matting task, when the confidence output by the model is less than the third threshold, it indicates that the matting model is less ideal for the matting result of the frame. These frames are filtered out from the original video for further correction to optimize the matting effect of the frames.

[0185] 404、In response to the operation on at least one of the at least one key frame or the at least one third video frame, at least one frame is corrected to obtain at least one fourth video frame.

[0186] The key frames or frames with low confidence are screened from the original video, and the user confirms whether these frames need to be corrected. Further, the frames confirmed by the user to need to be corrected are screened and corrected. In response to the correction operation of the user, the video frame is subjected to matting processing, including steps of noise reduction, separation, smoothing, feathering of the edge of the target object, and the like, to optimize the details of the matting of the target object, to obtain a fourth video frame after matting processing.

[0187] 405. Inputting the at least one fourth video frame into the first neural network model for training to obtain a second neural network model.

[0188] The corrected key frames or frames with low confidence are used as a training set. The training set data includes video frames and the confidence of the video frames. The corrected video frames and the confidence of the video frames are input into the first neural network model for training. At the same time, the model parameters are adjusted by an optimization algorithm to minimize the loss function. The model performance is evaluated on a validation set. Hyperparameter tuning and model optimization are performed. The final performance of the model is evaluated on a test set. The accuracy and robustness of the model are checked to obtain an optimized video matting model (second neural network model).

[0189] 406. Inputting the at least one first video frame into the second neural network model to output at least one fifth video frame.

[0190] The original video is input into the optimized neural network model for video matting processing to obtain at least one fifth video frame after matting processing.

[0191] The processed frames of images are recombined into a video stream. Video encoding libraries such as OpenCV and FFmpeg can be used to encode the frame sequence into a video file. Appropriate video encoding formats such as H.264 and HEVC and quality parameters are selected to combine the at least one fifth video frame into a video output after matting, and to ensure the compression rate and picture quality balance of the output video.

[0192] It should be noted that step 401 can be executed synchronously with step 402, and step 402 can be executed before step 403. Therefore, the execution order can be steps 401, 402, and 403, or 402, 401, and 403, or 402, 403, and 401, and the specific execution order is not limited herein.

[0193] In summary, as shown in FIG. 7, the key frames in the video are obtained according to the type of the target object in the video. When the target object is of a type with more changes in posture such as a human or an animal, the dynamic key points of the target object are identified by using a neural network model. The displacement or speed of the dynamic key points is calculated by using an optical flow algorithm. When the speed or displacement of the dynamic key points exceeds a first threshold value, the video frame is used as a key frame.

[0194] When the target object is a static object type such as a building, a cup, a flowerpot, or the like, a salient object region corresponding to the static object is obtained through a salient object detection model, a feature difference degree of the salient object region between different video frames is calculated through an inter-frame difference analysis method, and when the feature difference degree exceeds a second threshold value, the video frame is taken as a key frame. The feature difference degree includes a feature value difference degree, a color histogram difference degree, or an edge feature difference degree.

[0195] The original video is input into a matting model, and each frame image after matting and the confidence of the frame image are output. Video frames corresponding to a confidence less than a third threshold value are screened as third video frames. Frames confirmed by a user to have a problem are screened from the key frames and the third video frames and are corrected. The corrected video frames are taken as a training set and are input into the matting model for training, so as to optimize parameters of the matting model and obtain an optimized matting model.

[0196] The original video is processed using the optimized matting model, the confidence output by the matting model can be improved, a more refined matting video is obtained, and the matting effect of the model is improved. In addition, the corrected video frames are taken as the training set. Even if the video scene changes, the corrected video frames in the changed scene can still be taken as the training set, so that the model is not limited by the current video scene.

[0197] Please refer to FIG. 8, which is a structural schematic diagram of a video processing apparatus provided by an embodiment of the present application. As shown in FIG. 8, the video processing apparatus 800 includes an acquisition unit 801, an image separation unit 802, a screening unit 803, a correction unit 804, a training unit 805, and an output unit 806.

[0198] The acquisition unit 801 is configured to acquire at least one key frame in at least one first video frame.

[0199] The image separation unit 802 is configured to input the at least one first video frame into a first neural network model to obtain at least one second video frame and a confidence of each second video frame. The first neural network model is configured to perform image separation on a target object in the at least one first video frame.

[0200] The screening unit 803 is configured to screen at least one third video frame from the at least one first video frame according to the confidence of the at least one second video frame.

[0201] The correction unit 804 is configured to, in response to an operation on at least one of the at least one key frame or the at least one third video frame, correct the at least one frame to obtain at least one fourth video frame.

[0202] The training unit 805 is configured to input the at least one fourth video frame into the first neural network model for training, to obtain a second neural network model.

[0203] The output unit 806 is configured to input the at least one first video frame into the second neural network model, and output at least one fifth video frame.

[0204] In some optional embodiments, the acquisition unit 801 is specifically configured to acquire, according to the target object, at least one key frame in the at least one first video frame.

[0205] In some optional embodiments, the acquisition unit 801 is specifically configured to, in a case where the target object belongs to a first type, input the at least one first video frame into a third neural network model, to obtain at least one dynamic key point of the at least one first video frame; and take, as a key frame, a first video frame in which displacement or speed of the dynamic key point exceeds a first threshold value.

[0206] In some optional embodiments, the acquisition unit 801 is specifically configured to, in a case where the target object belongs to a second type, input the at least one first video frame into a fourth neural network model, to obtain a salient object region of the at least one first video frame; and take, as a key frame, a first video frame in which feature difference degree of the salient object region exceeds a second threshold value.

[0207] In some optional embodiments, the feature difference degree includes one or more of the following: difference degree of a feature value, difference degree of a color histogram, and difference degree of an edge feature.

[0208] In some optional embodiments, the image separation unit 802 is specifically configured to input the at least one first video frame into the first neural network model, to obtain at least one second video frame, each second video frame including at least one first pixel point; and calculate a mean value of confidence of the at least one first pixel point in each second video frame, to obtain a confidence of each second video frame.

[0209] In some optional embodiments, the image separation unit 802 is specifically configured to input the at least one first video frame into the first neural network model, to obtain at least one second video frame, each second video frame including at least one first pixel point; and obtain, according to the confidence and a weight value of the at least one first pixel point in each second video frame, a confidence of each second video frame.

[0210] In some optional embodiments, the screening unit 803 is specifically configured to take, as a third video frame, a first video frame corresponding to a second video frame whose confidence is less than a third threshold value.

[0211] In some optional embodiments, the first neural network model is an encoder-decoder structure, the output of the decoder includes the R channel value, the G channel value, the B channel value, the alpha channel value and the confidence of the second video frame, and the first neural network model includes a restnet, a mobilenet, an LRASSP or a convGRU network structure.

[0212] The obtaining unit 801, the image separation unit 802, the screening unit 803, the correction unit 804, the training unit 805 and the output unit 806 can be implemented by software or by hardware. For example, the implementation of the obtaining unit 801 is described below. The implementation of the image separation unit 802, the screening unit 803, the correction unit 804, the training unit 805 and the output unit 806 can be similar to the implementation of the obtaining unit 801.

[0213] As an example of a software function unit, the obtaining unit 801 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining unit 801 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code can be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers for running the code can be distributed in the same availability zone (AZ) or in different AZs, and each AZ includes one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0214] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Generally, one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-zone communication between two VPCs in the same region or between VPCs in different regions, and the interconnection between VPCs is realized through the communication gateway.

[0215] As an example of a hardware functional unit, the obtaining unit 801 can include at least one computing device, such as a server or the like. Alternatively, the obtaining unit 801 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0216] The multiple computing devices included in the obtaining unit 801 can be distributed in the same region or in different regions. The multiple computing devices included in the obtaining unit 801 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the obtaining unit 801 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0217] Referring to FIG. 9, FIG. 9 is a structural schematic diagram of a computing device provided by an embodiment of the present application. The computing device 900 includes a processor 901, a communication interface 902, a bus 903, and a memory 904. The processor 901, the communication interface 902, and the memory 904 communicate with each other through the bus 903, and in actual application, communication can also be realized by wireless transmission or other means, which is not limited here.

[0218] The computing device 900 can be a server or a terminal device, and it should be understood that the number of processors and memories in the computing device 900 is not limited in the present application.

[0219] The processor 901 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0220] The communication interface 902 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, and the like to enable communication between the computing device 900 and other devices or communication networks.

[0221] The bus 903 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is shown in FIG. 9, but it does not mean that there is only one bus or only one type of bus. The bus 903 can include a path for transmitting information between various components (e.g., the memory 904, the processor 901, the communication interface 902) of the computing device 900.

[0222] The memory 904 can include volatile memory, such as random access memory (RAM) and non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0223] The memory 904 stores executable program code, and the processor 901 executes the executable program code to respectively implement the functions of the aforementioned acquisition unit 801, the image separation unit 802, the screening unit 803, the correction unit 804, the training unit 805, and the output unit 806, thereby implementing the video processing method. That is, the memory 904 stores instructions for executing the video processing method.

[0224] The embodiments of the present application also provide a computing device cluster, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some optional embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0225] Please refer to FIG. 10 and FIG. 11, both of which are structural schematic diagrams of the computing device cluster provided by the embodiments of the present application.

[0226] As shown in FIG. 10, the computing device cluster includes at least one computing device 900. The same instructions for performing the video processing method provided by the embodiments of the present application can be stored in the memory 904 of one or more computing devices 900 in the computing device cluster.

[0227] In some possible implementations, partial instructions for performing the video processing method can also be stored in the memory 904 of one or more computing devices 900 in the computing device cluster, respectively. In other words, the combination of the memories 904 of the one or more computing devices can collectively perform the instructions for performing the video processing method.

[0228] It should be noted that the memories 904 in different computing devices 900 in the computing device cluster can store different instructions, respectively, for performing partial functions of the video processing apparatus. That is, the instructions stored in the memories 904 in different computing devices 900 can implement the functions of one or more of the obtaining unit 801, the image separation unit 802, the screening unit 803, the correction unit 804, the training unit 805, and the output unit 806.

[0229] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 11 shows one possible implementation. As shown in FIG. 11, two computing devices 900A and 900B are connected through a network. Specifically, the communication interfaces in the respective computing devices are connected to the network. In this type of possible implementation, the memory 904 in the computing device 900A stores instructions for performing the functions of the obtaining unit 801. Meanwhile, the memory 904 in the computing device 900B stores instructions for performing the functions of the image separation unit 802, the screening unit 803, the correction unit 804, the training unit 805, and the output unit 806.

[0230] The connection manner between the computing device cluster shown in FIG. 11 can be that, in the video processing method provided by the present application, a large amount of data needs to be stored, and therefore it is considered to assign the functions of the obtaining unit 801 to the computing device 900A and assign the functions of the image separation unit 802, the screening unit 803, the correction unit 804, the training unit 805, and the output unit 806 to the computing device 900B.

[0231] It should be understood that the functions of the computing device 900A shown in FIG. 11 can also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B can also be completed by multiple computing devices 900.

[0232] The embodiment of the present application further provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection mode of the computing device cluster described with reference to FIG. 10 and FIG. 11, which will not be repeated here.

[0233] The embodiment of the present application further provides a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the video processing method described above.

[0234] The embodiment of the present application further provides a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the video processing method described above.

[0235] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0236] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of video processing, the method comprising: The method comprises the following steps: acquiring at least one key frame in at least one first video frame; inputting the at least one first video frame into a first neural network model to obtain at least one second video frame and a confidence degree of each second video frame, the first neural network model being used for image separation of a target object in the at least one first video frame; screening at least one third video frame from the at least one first video frame according to the confidence degree of the at least one second video frame; in response to an operation on at least one of the at least one key frame or the at least one third video frame, correcting the at least one frame to obtain at least one fourth video frame; inputting the at least one fourth video frame into the first neural network model for training to obtain a second neural network model; inputting the at least one first video frame into the second neural network model to output at least one fifth video frame.

2. The video processing method of claim 1, wherein, The method comprises the following steps: acquiring at least one key frame in at least one first video frame according to the target object.

3. The video processing method of claim 2, wherein, The method comprises the following steps: in a case where the target object belongs to a first type, inputting the at least one first video frame into a third neural network model to obtain at least one dynamic key point of the at least one first video frame; taking, as a key frame, the first video frame in which a displacement or a speed of the dynamic key point exceeds a first threshold value.

4. The video processing method of claim 2, wherein, The method comprises the following steps: in a case where the target object belongs to a second type, inputting the at least one first video frame into a fourth neural network model to obtain a salient object region of the at least one first video frame; taking, as a key frame, the first video frame in which a feature difference degree of the salient object region exceeds a second threshold value.

5. The video processing method of claim 4, wherein, The feature difference degree comprises one or more of the following: a feature value difference degree, a color histogram difference degree, and an edge feature difference degree.

6. The video processing method of any of claims 1-5, wherein, The method comprises the following steps: inputting the at least one first video frame into the first neural network model to obtain the at least one second video frame, each second video frame comprising at least one first pixel point; calculating a mean value of the confidence degrees of the at least one first pixel point in each second video frame to obtain the confidence degree of each second video frame.

7. The video processing method of any of claims 1-5, wherein, The method comprises the following steps: inputting the at least one first video frame into the first neural network model to obtain the at least one second video frame, each second video frame comprising at least one first pixel point; obtaining the confidence degree of each second video frame according to the confidence degrees and weight values of the at least one first pixel point in each second video frame.

8. The video processing method of any of claims 1-7, wherein, The filtering at least one third video frame from the at least one first video frame according to the confidence of the at least one second video frame comprises: When the confidence of the second video frame is less than a third threshold value, the first video frame corresponding to the second video frame is taken as the third video frame.

9. The video processing method of any of claims 1-8, wherein, The first neural network model is an encoder-decoder structure, and an output of the decoder comprises R channel values, G channel values, B channel values, alpha channel values of the second video frame and the confidence, and the first neural network model comprises a restnet, a mobilenet, an LRASSP or a convGRU network structure.

10. A video processing apparatus, comprising: The method comprises: An acquisition unit is configured to acquire at least one key frame in at least one first video frame; An image separation unit is configured to input the at least one first video frame into a first neural network model to obtain at least one second video frame and a confidence of each second video frame, the first neural network model being configured to perform image separation on a target object in the at least one first video frame; A filtering unit is configured to filter at least one third video frame from the at least one first video frame according to the confidence of the at least one second video frame; A correction unit is configured to correct at least one frame of the at least one key frame or the at least one third video frame in response to an operation on the at least one frame to obtain at least one fourth video frame; A training unit is configured to input the at least one fourth video frame into the first neural network model to train the first neural network model to obtain a second neural network model; An output unit is configured to input the at least one first video frame into the second neural network model to output at least one fifth video frame.

11. The video processing apparatus of claim 10, wherein, The acquisition unit is specifically configured to acquire at least one key frame in the at least one first video frame according to the target object.

12. The video processing apparatus of claim 11, wherein, The acquisition unit is specifically configured to: In a case where the target object belongs to a first type, input the at least one first video frame into a third neural network model to obtain at least one dynamic key point of the at least one first video frame; Take the first video frame whose displacement or speed of the dynamic key point exceeds a first threshold value as the key frame.

13. The video processing apparatus of claim 11, wherein, The acquisition unit is specifically further configured to: In a case where the target object belongs to a second type, input the at least one first video frame into a fourth neural network model to obtain a salient object region of the at least one first video frame; Take the first video frame whose feature difference degree of the salient object region exceeds a second threshold value as the key frame.

14. The video processing apparatus of claim 13, wherein, The feature difference degree comprises one or more of the following: A difference degree of feature values, a difference degree of color histograms, and a difference degree of edge features.

15. The video processing apparatus of any of claims 10-14, wherein, The image separation unit is specifically configured to: Input the at least one first video frame into the first neural network model to obtain the at least one second video frame, each second video frame comprising at least one first pixel point; Calculate a mean value of the confidence of the at least one first pixel point in each second video frame to obtain the confidence of each second video frame.

16. The video processing apparatus of any of claims 10-14, wherein, The image separation unit is specifically configured to: inputting the at least one first video frame into the first neural network model to obtain the at least one second video frame, each second video frame including at least one first pixel point; obtaining a confidence of each second video frame according to a confidence and a weight of the at least one first pixel point in each second video frame.

17. The video processing apparatus of any of claims 10-16, wherein, The screening unit is specifically configured to take the first video frame corresponding to the second video frame with a confidence less than a third threshold as a third video frame.

18. The video processing apparatus of any of claims 10-17, wherein, The first neural network model is an encoder-decoder structure, and an output of the decoder includes an R channel value, a G channel value, a B channel value, an alpha channel value, and the confidence of the second video frame, and the first neural network model includes a restnet, a mobilenet, an LRASSP, or a convGRU network structure.

19. A cluster of computing devices, characterized in that, The at least one computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method of any one of claims 1 to 9.

20. A computer program product comprising instructions, characterized in that, The instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1 to 9.

Images