Video processing method and device, equipment and medium

Abstract

The invention relates to a video processing method and device, equipment and a medium, and the method comprises the steps: carrying out the target detection of a to-be-processed video, and obtaining a detection frame of a target video frame and a first confidence coefficient corresponding to the detection frame; inputting the target video frame into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame; according to the saliency prediction map corresponding to the target video frame and a first confidence coefficient corresponding to a detection frame of the target video frame, determining a target area, the target area being an area where the main target object is located in the target video frame; and carrying out image processing on the target video frame according to a mode of reserving the target area so as to carry out conversion between a horizontal screen frame and a vertical screen frame on the target video frame. According to the technical scheme disclosed by the invention, the key information in the picture is prevented from being lost after the to-be-processed video is converted from the horizontal screen to the vertical screen.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a video processing method, device, equipment and medium. Background Art

[0002] With the development of the Internet and the emergence of various short video applications, short videos have become the main source of information for people in their fragmented time. Short videos are usually vertical videos. For a large number of horizontal video resources, there is a need to convert horizontal video resources into vertical short videos (i.e. convert the video frames in the video from horizontal frames to vertical frames) for traffic diversion and promotion.

[0003] At present, when converting a horizontal video to a vertical video, there is a loss of picture information. Whether the key information in the picture (i.e., the main target object) can be retained affects the video conversion effect. How to retain the key information in the picture during video conversion is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a video processing method, device, equipment and medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0006] Performing target detection on the video to be processed to obtain a detection frame of a target video frame and a first confidence level corresponding to the detection frame, wherein the target video frame is a video frame including a target object in the video to be processed, and the detection frame is used to indicate an area in the target video frame where the target object identified by target detection is located;

[0007] Inputting the target video frame into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame, wherein the saliency prediction map includes a saliency region, and the saliency region is a region in the target video frame where the target object identified by the saliency prediction model is located;

[0008] Determine a target area according to the saliency prediction map corresponding to the target video frame and the first confidence level corresponding to the detection frame of the target video frame, wherein the target area is an area where a main target object is located in the target video frame;

[0009] The target video frame is subjected to image processing in a manner of retaining the target area, so as to convert the target video frame into a horizontal frame and a vertical frame.

[0010] In a second aspect, an embodiment of the present disclosure provides a video processing device, including:

[0011] A first detection module, configured to perform target detection on the video to be processed, and obtain a detection frame of a target video frame and a first confidence level corresponding to the detection frame, wherein the target video frame is a video frame including a target object in the video to be processed, and the detection frame is used to indicate an area in the target video frame where the target object identified by target detection is located;

[0012] A first prediction module is used to input the target video frame into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame, wherein the saliency prediction map includes a saliency region, and the saliency region is a region in the target video frame where the target object identified by the saliency prediction model is located;

[0013] A first determination module is used to determine a target area according to the saliency prediction map corresponding to the target video frame and the first confidence level corresponding to the detection frame of the target video frame, wherein the target area is an area where a main target object is located in the target video frame;

[0014] The first image processing module is used to perform image processing on the target video frame in a manner of retaining the target area, so as to convert the target video frame into a horizontal screen frame and a vertical screen frame.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the video processing method described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the video processing method described in the first aspect is implemented.

[0017] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art: it is capable of performing target detection on a video to be processed, obtaining a detection frame of a target video frame and a first confidence level corresponding to the detection frame, wherein the target video frame is a video frame including a target object in the video to be processed, and the detection frame is used to indicate an area in the target video frame where the target object identified by target detection is located; the target video frame is input into a pre-trained saliency prediction model for processing, and a saliency prediction map corresponding to the target video frame is obtained, wherein the saliency prediction map includes a saliency area, and the saliency area is an area in the target video frame where the target object identified by the saliency prediction model is located; the target area is determined according to the saliency prediction map corresponding to the target video frame and the first confidence level corresponding to the detection frame of the target video frame, wherein the target area is an area where the main target object is located in the target video frame; image processing is performed on the target video frame in a manner of retaining the target area, so as to convert the target video frame into a horizontal frame and a vertical frame. Therefore, the target area of ​​the video frame is comprehensively determined based on target detection and saliency prediction, the main body of the video frame can be accurately identified, and the video frame can be cropped based on the target area to achieve horizontal and vertical screen conversion of the video, which can retain the main body of the picture and avoid the loss of key information in the picture after the video is converted from horizontal to vertical. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of another video processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0025] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0026] Figure 1 This is a flow chart of a video processing method provided in an embodiment of the present disclosure. The method provided in an embodiment of the present disclosure can be executed by a video processing device, which can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities.

[0027] like Figure 1 As shown, the video processing method provided by the embodiment of the present disclosure may include:

[0028] S110, performing target detection on the video to be processed to obtain a detection frame of the target video frame and a first confidence level corresponding to the detection frame, wherein the target video frame is a video frame including a target object in the video to be processed, and the detection frame is used to indicate an area in the target video frame where the target object identified by target detection is located.

[0029] In this embodiment, the video to be processed includes but is not limited to variety show videos, film and television videos, promotional videos, etc. The video to be processed includes multiple video frames. The video to be processed can be a complete video or a video clip in a complete video.

[0030] As an example, a complete video is divided into multiple video segments (i.e., shot segments) through shot detection, each video segment corresponds to a shot scene, the video segments are used as videos to be processed, target detection is performed on the videos to be processed, and target detection results of each video frame in the videos to be processed are obtained.

[0031] As an example, target detection is performed on a video to be processed, including: inputting each video frame in the video to be processed into a target detection model for processing, and obtaining a target detection result of the video frame. The target detection model can be implemented based on machine learning and deep learning. The input of the target detection model is an image, and the target detection result of the image is output. The target detection result includes a detection frame and a first confidence level of the target object. The detection frame is used to indicate the area of ​​the target object in the video frame. The first confidence level indicates the probability that the target object exists in the detection frame. The target object can be determined according to the actual application scenario, and no specific restrictions are made here.

[0032] Exemplarily, taking variety show videos as an example, the target object can be a person, including real people and / or virtual characters. Target detection is performed on the video to be processed to obtain the target detection results of the video frames in the video to be processed, including: inputting the video frames into a pre-trained character detection model for character detection, and generating a detection frame of the person in the video frame. The model training process includes: obtaining a training sample set, the training sample set includes areas marked with characters, training the character detection model through the training sample set, and then, in the process of converting the video to be processed from horizontal to vertical screen, inputting the video frame into the character detection model, for the video frame in which the character is recognized (i.e., the target video frame), the character detection model outputs the detection frame of the character and the corresponding first confidence, and for the video frame in which the character is not recognized, the character detection model output is empty, thereby obtaining the character detection results of each video frame.

[0033] Optionally, in order to reduce misidentification, for video frames in which people are identified, the size of each detection frame output by the person detection model is calculated, and the size of each detection frame is compared with a preset threshold. If the size of the detection frame is smaller than the preset threshold, the detection frame is deleted from the person detection result.

[0034] S120, inputting the target video frame into the pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame, wherein the saliency prediction map includes a saliency region, and the saliency region is a region in the target video frame where the target object identified by the saliency prediction model is located.

[0035] In this embodiment, for each target video frame, the target video frame is input into the saliency prediction model for processing to obtain a corresponding saliency prediction map. The input of the saliency prediction model is an image, and the output is a saliency prediction map of the image. The saliency prediction map is used to indicate whether each position in the image belongs to a saliency region. For example, the saliency prediction map is consistent with the width and height of the image input into the saliency prediction model, and the value of each position in the saliency prediction map is used to indicate whether the position belongs to a saliency region.

[0036] As an example, the saliency prediction model can be implemented based on machine learning and deep learning, and the model training process includes: obtaining a sample image, wherein the sample image is annotated with a saliency region where the target object is located; and training the saliency prediction model based on the sample image. Specifically, the sample image is input into the initial model for processing, and a saliency prediction map of the sample image is output, and the loss is calculated through the saliency region in the output saliency prediction map and the saliency region annotated in the sample image, so as to adjust the parameters of the initial model through loss back propagation until the model converges and obtains the saliency prediction model. It can be understood that by comparing with the annotated saliency region, it can be ensured that the model learns the correct saliency region, improves the accuracy of the target object positioning, and dynamically adjusts the parameters of the model according to the change of the loss during the training process, which can improve the training efficiency.

[0037] S130. Determine a target area according to a saliency prediction map corresponding to the target video frame and a first confidence level corresponding to a detection frame of the target video frame, wherein the target area is an area where a main target object is located in the target video frame.

[0038] In this embodiment, the value of each position in the saliency prediction map is used to indicate whether the corresponding position belongs to a salient area. For the detection frame, the area corresponding to the detection frame is determined from the saliency prediction map, and the value of each position in the area is multiplied by the feature in the detection frame (the feature is the feature in the target video frame) to obtain the second confidence of the detection frame. The target area is determined according to the second confidence of each detection frame, wherein the target area is the area where the main target object in the video to be processed is located, and the main target object is the main object shown in the video to be processed. In other words, the importance, criticality and focus of the main target object are stronger than other target objects, that is, the main target object is the most important and core part of the picture, and has significant importance in the image. For example, the main target object is the main character (main character and / or main object). As an example, when the second confidence of the detection box meets a preset condition, the detection box is determined to be a target area. For example, the preset condition may be that the second confidence of the detection box is greater than a threshold. For another example, the preset condition may be that the second confidence of the detection box is the maximum value among the second confidences of all detection boxes in the video to be processed. For another example, the preset condition may be that the second confidence of the detection box is the maximum value among the second confidences of all detection boxes in the target video frame in which it is located.

[0039] S140: performing image processing on the target video frame in a manner of retaining the target area, so as to convert the target video frame into a horizontal frame and a vertical frame.

[0040] In this embodiment, if the video to be processed is a horizontal video (i.e., the video frame is a horizontal screen frame), for the target video frame in the horizontal video, image processing is performed on the target video frame in a manner of retaining the target area, so as to convert the target video frame from a horizontal screen frame to a vertical screen frame, thereby converting the horizontal video into a vertical video (i.e., the video frame is a vertical screen frame). If the video to be processed is a vertical video, for the target video frame in the horizontal video, image processing is performed on the target video frame in a manner of retaining the target area, so as to convert the target video frame from a vertical screen frame to a horizontal screen frame, thereby converting the vertical video into a horizontal video. Among them, a horizontal screen frame refers to a video frame whose width is greater than the height, i.e., the size in the vertical direction is smaller than the size in the horizontal direction; a vertical screen frame refers to a video frame whose width is less than the height, i.e., the size in the vertical direction is larger than the size in the horizontal direction.

[0041] According to the embodiments of the present disclosure, the target area is comprehensively determined based on target detection and saliency prediction, the main body of the picture of the video to be processed can be accurately identified, and video processing is achieved by cropping the video frame based on the target area. The main body of the picture can be retained, and the key information in the picture can be avoided from being lost after the video is converted from horizontal to vertical screen, which solves the problem of loss of picture information when converting horizontal video to vertical video. Furthermore, various types of horizontal videos can be converted into vertical videos through this method, which can be applied to accurately and quickly generate vertical short videos. For example, hot short videos can be generated based on variety show and film and television videos to meet the needs of attracting traffic and promotion. There is no need for users to manually edit the video, which reduces labor costs.

[0042] In one embodiment of the present disclosure, a target video frame is input into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame, including: using the saliency prediction model to perform the following processing on each target video frame:

[0043] S121. Perform feature extraction on a target video frame to obtain a plurality of first feature maps of different sizes of the target video frame.

[0044] In this embodiment, a convolutional neural network in a saliency prediction model is used to extract features of a target video frame to obtain a plurality of first image features of different sizes of the target video frame.

[0045] As an example, the width of the target video frame is w, and the height is h. The target video frame is feature extracted through a convolutional neural network to obtain first feature maps with sizes of w / 8*h / 8, w / 16*h / 16, w / 32*h / 32, and w / 64*h / 64, respectively.

[0046] S122. For each first feature graph, perform sequence generation processing on the first feature graph to obtain a first feature sequence, and process the first feature sequence based on a multi-head self-attention mechanism to obtain a second feature sequence corresponding to the first feature graph.

[0047] As an example, the first feature map is flattened (ie, Flattening) to obtain a first feature sequence.

[0048] As an example, the first feature map is processed by patch embedding to obtain the first feature sequence. For example, in Vision Transformer (ViT), the first feature map is divided into patches of fixed size, and each patch is regarded as a separate sequence element without flattening the entire first feature map. In this way, local spatial information can be retained, which is suitable for visual tasks.

[0049] As an example, the first feature map is processed by convolutional embedding to obtain the first feature sequence. The features are extracted by the convolution layer, and the first feature map is not directly flattened, but the convolution kernel is used to generate feature embedding. This method retains some spatial structure information and is also suitable for visual tasks.

[0050] In this embodiment, the first feature sequence is processed based on the attention algorithm to obtain a second feature sequence corresponding to the first feature map. Attention algorithms include but are not limited to Multi-head Self Attention (MSA), Convolutional Attention, Sparse Attention, and Convolutional Neural Network + Self-Attention Mechanism. Among them, Convolutional Attention combines convolution operation and attention mechanism, and uses a combination of local convolution operation and global self-attention to retain the local receptive field of convolution and obtain global dependency. Sparse Attention reduces the amount of calculation by performing attention calculation only between specific positions. Representative methods include Big Bird, Longformer, etc.

[0051] As an example, after obtaining the first feature map, the first feature map is processed using a 1×1×Y convolution kernel, and the feature dimension of the first feature map is expanded to the Y dimension while keeping the size of the first feature map unchanged, and then the first feature map expanded to the Y dimension is input into the Transformer architecture, and the first feature map expanded to the Y dimension is expanded into the form of a feature sequence. Then, the similarity of each position of the first feature sequence with other positions in the first feature sequence is calculated by MSA, and all positions of the first feature sequence are weighted and summed based on the calculation results to improve the representation ability of the features. MSA repeats the above calculations multiple times, and each calculation uses a different weight matrix, so that the model can extract information from multiple subspaces and connect the outputs of all subspaces to generate a second feature sequence.

[0052] S123. For each first feature map, perform feature map generation processing on the second feature sequence corresponding to the first feature map to obtain a second feature map corresponding to the first feature map, wherein the first feature map and the second feature map have the same size.

[0053] In this embodiment, the second feature sequence is restored (reshaped) to its original size to obtain a second feature map.

[0054] As an example, the width of the target video frame is w and the height is h. The feature extraction of the target video frame obtains four first feature maps of size w / 8*h / 8, w / 16*h / 16, w / 32*h / 32, and w / 64*h / 64. Each first feature map is processed with a 1×1×768 convolution kernel to obtain four first feature maps of w / 8*h / 8*768, w / 16*h / 16*768, w / 32*h / 32*768, and w / 64*h / 64*768, thereby converting the number of feature channels to 768 to match the input dimension of the Transformer. Furthermore, the four first feature maps of w / 8*h / 8*768, w / 16*h / 16*768, w / 32*h / 32*768, and w / 64*h / 64*768 are input into the Transformer architecture for expansion to obtain the corresponding four first feature sequences. For each first feature sequence, the first feature sequence is processed by MSA, wherein the first feature sequence X is linearly transformed to obtain the query vector Q, the key vector K, and the value vector V, Q=XW Q , K = XW K , V = XW V , W Q , W K , W V is a trainable weight matrix. The query vector Q and the key vector K are used to calculate the attention score (similarity) and normalized by softmax. The formula is as follows:

[0055]

[0056] Among them, d k is the dimension of the key vector, used to scale the attention scores.

[0057] The above steps are repeated h times, each time using a different weight matrix to obtain h sub-outputs, which are connected to generate the output corresponding to the feature sequence (i.e., the second feature sequence), and the formula is as follows:

[0058] MultiHead(Q,K,V)=Conact(head1,head2,…,head h )W o

[0059] Among them, head i =Attention(QW Qi , KW Ki , VW Vi ), W o is a trainable weight matrix.

[0060] Four second feature sequences are obtained for the four first feature sequences respectively, and the four second feature sequences are restored to the size of the corresponding first feature maps to obtain four second feature maps of w / 8*h / 8*768, w / 16*h / 16*768, w / 32*h / 32*768, and w / 64*h / 64*768.

[0061] S124, fusing multiple second feature maps of different sizes to obtain a fused feature map.

[0062] In this embodiment, a plurality of second feature maps of different sizes are fused by using a skip connection (i.e., stepwise upsampling and feature fusion) to obtain a fused feature map. For example, the plurality of second feature maps are arranged in ascending order of size as the second feature maps. Figure 1 , Second feature Figure 2 , ..., the second feature map m, for the second feature with the smallest size Figure 1 Upsample to convert the second feature Figure 1 The size is changed to match the next second feature in the sorting Figure 2 Consistent, the second feature after the size transformation Figure 1 With the second feature Figure 2The fusion is performed to obtain fusion feature 1, and then the fusion feature i is upsampled to transform the size of the fusion feature i to be consistent with the second feature map i+2, and the size-transformed fusion feature i is fused with the second feature map i+2 to obtain fusion feature i+1, until all second feature maps are fused to obtain the final fusion feature map, thereby enabling the fusion feature map to obtain the contextual relationship of features of different dimensions, making the feature representation richer and the representation capability stronger.

[0063] As an example, for the four second feature maps of w / 8*h / 8*768, w / 16*h / 16*768, w / 32*h / 32*768, and w / 64*h / 64*768, the second feature map of w / 64*h / 64*768 is processed by a BN (BatchNormal) layer and a Relu activation layer in sequence, and then upsampled by two to transform the second feature map into w / 32*h / 32*768, and the transformed second feature map is fused with the initial second feature map of w / 32*h / 32*768, for example, by The fused feature is obtained by element-wise multiplication. The fused feature of w / 32*h / 32*768 is upsampled to transform the fused feature into w / 16*h / 16*768, and the transformed fused feature is fused with the second feature map of w / 16*h / 16*768 to obtain the fused feature of w / 16*h / 16*768. By analogy, the fused feature of w / 8*h / 8*768 is obtained. Furthermore, the fused feature of w / 8*h / 8*768 is successively passed through the BN layer, the Relu activation layer and upsampling to obtain the fused feature of w*h*768, that is, the fused feature map is obtained.

[0064] S125. Predict the target object in the target video frame according to the fused feature map to obtain a saliency prediction map corresponding to the target video frame.

[0065] As an example, the fusion feature map corresponding to the target video frame is input into the activation layer for processing to generate a saliency prediction map of the target video frame. The saliency prediction map is consistent with the width and height of the target video frame, and the value of each position in the saliency prediction map is the confidence of the position, which is used to indicate whether the position belongs to the saliency region. For example, the value of each position in the saliency prediction map is 0 or 1, 0 indicates that it does not belong to the saliency region, and 1 indicates that it belongs to the saliency region. The activation layer is implemented using a Sigmoid layer.

[0066] As an example, the fused feature map corresponding to the target video frame is input into the Softmax layer or the Tanh layer for processing to generate a saliency prediction map of the target video frame. Among them, Softmax is usually used for multi-classification problems, but it can also be used for saliency map prediction. If the goal is to make multi-class saliency judgments for each pixel or region, Softmax can be used to normalize the output so that the predicted value at each position represents the saliency probability of different categories. This method is a soft attention mechanism because the generated probability value is a probability value between 0 and 1, while Sigmoid is a hard attention mechanism and the generated value is either 0 or 1. The Tanh layer uses Thresholding (thresholding processing) and does not use a specific activation function. Instead, it thresholds the feature map and regards the part above a certain threshold as a saliency area. For example, thresholding the response of the convolution feature map can directly obtain a saliency map.

[0067] Optionally, predicting a target object in a target video frame according to the fused feature map to obtain a saliency prediction map corresponding to the target video frame includes:

[0068] Upsampling the fused feature map so that the size of the fused feature map is the same as the size of the target video frame corresponding to the fused feature map;

[0069] The fused feature map after upsampling is processed to generate a saliency prediction map, wherein the saliency prediction map includes a salient area and a non-salient area, the value of the pixel in the salient area is 1, and the value of the pixel in the non-salient area is 0.

[0070] It can be understood that the upsampling process can enlarge the low-resolution fused feature map to the same size as the target video frame, thereby maintaining high spatial resolution. The high-resolution saliency prediction map can better retain the detail information in the image and help to accurately locate the salient area.

[0071] As an example, the fused feature of w*h*768 (i.e., fused feature map) is processed by a 1×1×1 convolution kernel so that the number of channels of the fused feature becomes 1, and the fused feature is further input to the activation layer for processing. Optionally, the upsampling by two is implemented in the following way: a transposed convolution layer is defined, the number of input channels and the number of output channels of the transposed convolution layer are both 768, the convolution kernel size is 4, the stride is 2, and the padding is 1.

[0072] It can be understood that since feature maps of different sizes can retain detailed information at different levels, the model can more accurately locate the salient area when generating the saliency prediction map, and the self-attention mechanism can capture long-distance dependencies in the feature map, which helps the model understand complex scenes. In addition, the fusion of multiple second feature maps of different sizes can integrate information at different scales and improve the robustness and generalization ability of the model.

[0073] In one embodiment of the present disclosure, determining a target area according to a saliency prediction map corresponding to a target video frame and a first confidence level corresponding to a detection frame of the target video frame includes:

[0074] For each detection box, perform dot multiplication between the detection box and the corresponding area of ​​the detection box in the saliency prediction map to obtain the second confidence level corresponding to the detection box;

[0075] For each detection frame, the first confidence and the second confidence of the detection frame are weightedly fused to generate a fused confidence of the detection frame;

[0076] The target area is determined by fusing the detection box with the largest confidence among all detection boxes in all target video frames.

[0077] In this embodiment, the fusion confidence is used to indicate the probability that the detection frame is the target area. When the fusion confidence of the detection frame satisfies a preset condition, the detection frame is determined to be the target area. For example, the preset condition may be that the fusion confidence of the detection frame is greater than a threshold. For another example, the preset condition may be that the fusion confidence of the detection frame is the maximum value among the fusion confidences of all detection frames in the video to be processed. For another example, the preset condition may be that the fusion confidence of the detection frame is the maximum value among the fusion confidences of all detection frames in the target video frame in which it is located.

[0078] As an example, for each detection frame in the video to be processed, the second confidence of the detection frame is obtained by multiplying the value of each position in the detection frame of the saliency prediction map with the feature point in the detection frame, and then, weighted fusion is performed according to the first confidence and the second confidence of the detection frame to generate the fusion confidence of each detection frame, and the detection frame with the largest fusion confidence in the video to be processed is determined as the target area of ​​the video to be processed. Wherein, fusion confidence = a*first confidence + (1-a)*second confidence, and the weight a can be set as needed, and no specific limitation is made here. Of course, for each target video frame, the second confidence of the detection frame can also be obtained by multiplying the value of each position in the detection frame of the saliency prediction map with the feature point in the detection frame, and then, weighted fusion is performed according to the first confidence and the second confidence of the detection frame to generate the fusion confidence of each detection frame, and the detection frame with the largest fusion confidence in the target video frame is determined as the target area of ​​the target video.

[0079] It can be understood that by calculating the multi-scale feature representation, the saliency prediction map of the target video frame is obtained to assist in determining the center point of the character subject under each shot, and by integrating the first confidence of the target detection and the second confidence of the saliency prediction in a weighted form, the main body of the picture is comprehensively determined, which can accurately identify the salient areas in complex scenes such as variety shows and movies, further improving the accuracy of target area recognition.

[0080] In some embodiments of the present disclosure, image processing is performed on a target video frame in a manner of retaining a target area to convert the target video frame into a horizontal screen frame and a vertical screen frame, including:

[0081] Using the height of the target video frame as the height of the cropping area, wherein the cropping area is the area to be cropped from the target video frame;

[0082] Determine the width of the cropping area according to the height of the cropping area and the center position of the target area, so that the center position of the target area coincides with the center position of the cropping area;

[0083] The cropping area is cropped from the target video frame according to the width and height of the cropping area and the center position of the target area, so as to convert the target video frame from a horizontal screen frame to a vertical screen frame.

[0084] As an example, taking the video to be processed as a horizontal video, the height of the target video frame is used as the height of the cropping area, the width of the cropping area is determined according to the aspect ratio of the vertical frame and the height of the cropping area, the center position of the target area is used as the center position of the cropping area to determine the cropping area in the video frame, and the cropping area is cropped out from the target video frame to convert the target video frame from a horizontal frame to a vertical frame.

[0085] It can be understood that by setting the center position of the target area to coincide with the center position of the cropped area, after the target video frame is converted from a horizontal frame to a vertical frame, the target area is located in the center of the picture, that is, the key information in the picture is located in the center of the picture.

[0086] In one embodiment of the present disclosure, the saliency prediction model processes to obtain saliency prediction maps of N target video frames, where the N target video frames are determined from the video to be processed at a preset interval, and N is a positive integer. The method further includes:

[0087] Tracking the target areas of the N target video frames using a tracking algorithm to determine the target areas of other video frames in other video frames of the video to be processed, wherein the other video frames are video frames other than the N target video frames;

[0088] For each other video frame, image processing is performed on the other video frame in a manner of retaining the target area of ​​the other video frame, so as to convert the other video frame into a horizontal screen frame and a vertical screen frame.

[0089] In this embodiment, for a video to be processed containing M video frames, the saliency prediction model processes to obtain saliency prediction maps of N target video frames, the N target video frames are determined from the M video frames at preset intervals, M and N are positive integers and M is greater than N, and the tracking algorithm determines the corresponding area of ​​the target area of ​​the target video frame in other video frames by feature comparison, as the target area of ​​other video frames. As an example, for a video to be processed with 25 frames per second, target detection and saliency prediction are performed for the first video frame of each second to determine the target area of ​​the video frame, and for the other 24 video frames of each second, feature extraction is performed based on the target area in the first video frame of this second, and feature comparison is performed in the other 24 video frames based on the extracted features to determine the corresponding area that meets the similarity condition, and the target area of ​​the other 24 video frames is determined based on the corresponding area. As a result, it is not necessary to perform target detection and saliency prediction operations for each other video frame, thereby improving processing efficiency, and by determining the target area of ​​other video frames through the tracking algorithm, the video effect is smoother and the visual effect is improved.

[0090] In one embodiment of the present disclosure, a saliency prediction model is used to process saliency prediction maps of N target video frames, where the N target video frames are determined from a video to be processed at preset intervals, and N is a positive integer. The method further includes: for other video frames between two adjacent target video frames among the N target video frames, a moving trajectory of the target area in the other video frames is determined according to the position of the target area in the two adjacent target video frames, and the position of the target area in the other video frames is determined according to the moving trajectory.

[0091] As an example, for a video to be processed with 25 frames per second, target detection and saliency prediction are performed on the first video frame of each second to determine the target area of ​​the video frame. For the other 24 video frames of each second, the moving trajectory of the target area in the 24 video frames is determined based on the position of the target area in the first video frame of this second and the position of the target area in the first video frame of the next second. Based on the constraint condition that the moving trajectory moves uniformly, the target area in the 24 video frames is determined. In this way, the target area of ​​each video frame is obtained, and for other video frames, it is not necessary to perform target detection and saliency prediction operations on each video frame, thereby improving processing efficiency.

[0092] The video processing method provided by the embodiment of the present disclosure is described in detail below with reference to a specific example. Figure 2 A flowchart of another video processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, in the first step, shot detection is used to divide the video to be processed into multiple shot segments. Each shot corresponds to a scene, which generally contains several characters. In the first step, the characters in each shot are obtained through human body detection. If there is no person (real person or cartoon character) in the scene, it returns null. If there is a character, it returns the detection frame and the first confidence corresponding to several characters. In the first step, the video frame with the detection frame is calculated for saliency prediction. The specific algorithm steps are as follows:

[0093] 1) The input image (i.e., the video frame with the detection box) has a width of w and a height of h. The convolutional neural network is used to extract features and obtain the first feature maps of w / 8*h / 8, w / 16*h / 16, w / 32*h / 32, and w / 64*h / 64 respectively.

[0094] 2) Based on each first feature map, a 1×1×768 convolution kernel is used to keep the size of the first feature map unchanged while expanding the feature dimension to a uniform 768 dimension (the number of feature channels is converted to 768 or other specific dimension sizes to ensure that it matches the input dimension of the Transformer model and makes full use of its architectural design), namely w / 8*h / 8*768, w / 16*h / 16*768, w / 32*h / 32*768, and w / 64*h / 64*768.

[0095] 3) For each feature map after 1×1×768 convolution, the Transformer architecture is used to flatten the feature map into a sequence form to obtain the corresponding first feature sequence, and then the multi-head self-attention (MSA) is used to improve the representation ability of the feature. The multi-head self-attention mechanism allows the saliency prediction model to calculate the similarity of each position of the first feature sequence with other positions, thereby weighted summing all positions of the first feature sequence. This mechanism enables the saliency prediction model to capture the dependencies between the various parts of the first feature sequence. Multi-head self-attention repeats this calculation multiple times, using different weight matrices each time, so that the saliency prediction model can extract information from multiple subspaces. Each subspace is called a "head". The outputs of all heads are connected to form the final output (i.e., the second feature sequence corresponding to the first feature sequence).

[0096] 4) For each output after passing through Transformer (i.e., the second feature sequence), reshape it into the original input size, i.e., w / 8*h / 8*768, w / 16*h / 16*768, w / 32*h / 32*768, w / 64*h / 64*768, and obtain the corresponding second feature map. Use skip-connection to fuse the second feature map, so that the final fused feature (i.e., fused feature map) can obtain the contextual relationship of features of different dimensions, making the feature representation richer and the representation ability stronger. Here is an example of a skip-connection operation. For example, the output dimension after Transformer is w / 64*h / 64*768, which passes through the BN (BatchNormal) layer and the Relu activation layer, and then through ×2 upsampling (upsampling by 2, defining a transposed convolution layer, the number of input channels and the number of output channels are both 768, the convolution kernel size is 4, the stride is 2, and the padding is 1. This configuration can double the spatial size of the feature map), and the output is w / 32*h / 32*768, which can be obtained by element-wise product with the output w / 32*h / 32*768 of the previous Transformer to obtain the fused features. Similarly, the feature map that continues to be upsampled to a dimension of w / 16*h / 16*768 can be fused with the feature map with an output dimension of w / 16*h / 16*768 after Transformer, and so on.

[0097] 5) The final fused features are upsampled multiple times and then passed through the Sigmoid layer to obtain a saliency prediction map. At this time, the saliency prediction map is regressed to the width and height of the input image, but the features become 0 or 1 after passing through the Sigmoid layer. 1 represents that this is a salient area, that is, it represents the association of contextual features of different dimensions, and this may be a key area.

[0098] In the fourth step, the saliency prediction map and the area where the detection box is located are calculated to obtain the second confidence corresponding to several human body detection boxes in each image. A high score means that the detection box may be the subject area.

[0099] The fifth step is to perform weighted fusion on the first confidence and the second confidence to obtain fusion confidence = a*first confidence + (1-a)*second confidence, where the weight is the empirical value obtained from the experiment, and sort the fusion confidences corresponding to several characters. The detection frame with the highest fusion confidence is the main character under the lens, and the detection frame is used to calculate the center point information corresponding to the main character; the final vertical video is obtained by combining the center point of the image sequence and the horizontal to vertical aspect ratio (the video height is fixed, and the image width, that is, the width of the vertical video, can be calculated through the center point and image height).

[0100] Figure 3 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the video processing device includes:

[0101] A first detection module 310 is used to perform target detection on the video to be processed, and obtain a detection frame of a target video frame and a first confidence level corresponding to the detection frame, wherein the target video frame is a video frame including a target object in the video to be processed, and the detection frame is used to indicate an area in the target video frame where the target object identified by target detection is located;

[0102] A first prediction module 320 is used to input the target video frame into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame, wherein the saliency prediction map includes a saliency region, and the saliency region is a region in the target video frame where the target object identified by the saliency prediction model is located;

[0103] A first determination module 330 is used to determine a target area according to the saliency prediction map corresponding to the target video frame and the first confidence level corresponding to the detection frame of the target video frame, wherein the target area is an area where a main target object is located in the target video frame;

[0104] The first image processing module 340 is configured to perform image processing on the target video frame in a manner of retaining the target area, so as to convert the target video frame into a horizontal frame and a vertical frame.

[0105] Optionally, the first prediction module 320 includes a first extraction submodule, a first processing submodule, a second processing submodule, a first fusion submodule and a first prediction submodule, wherein the first prediction module is specifically used to perform the following processing on each target video frame using the saliency prediction model:

[0106] A first extraction submodule, used to extract features from a target video frame to obtain a plurality of first feature maps of different sizes of the target video frame;

[0107] A first processing submodule is used to perform sequence generation processing on each first feature map to obtain a first feature sequence, and process the first feature sequence based on a multi-head self-attention mechanism to obtain a second feature sequence corresponding to the first feature map;

[0108] A second processing submodule is used to perform feature map generation processing on a second feature sequence corresponding to each first feature map to obtain a second feature map corresponding to the first feature map, wherein the first feature map and the second feature map have the same size;

[0109] A first fusion submodule is used to fuse multiple second feature maps of different sizes to obtain a fused feature map;

[0110] The first prediction submodule is used to predict the target object in the target video frame according to the fused feature map to obtain a saliency prediction map corresponding to the target video frame.

[0111] Optionally, the first prediction submodule is specifically used to perform upsampling processing on the fused feature map so that the size of the fused feature map is the same as the size of the target video frame corresponding to the fused feature map;

[0112] The fused feature map after upsampling is processed to generate a saliency prediction map, wherein the saliency prediction map includes a salient area and a non-salient area, the value of the pixel in the salient area is 1, and the value of the pixel in the non-salient area is 0.

[0113] Optionally, the first determining module 330 includes:

[0114] A third processing submodule is used to perform a dot multiplication process on each of the detection boxes and a corresponding area of ​​the detection box in the saliency prediction map to obtain a second confidence level corresponding to the detection box;

[0115] A second fusion submodule, configured to perform weighted fusion of the first confidence level and the second confidence level of each detection frame to generate a fusion confidence level of the detection frame;

[0116] The first determination submodule is used to determine the target area according to the detection frame with the largest fusion confidence among all the detection frames of all the target video frames.

[0117] Optionally, the device further comprises:

[0118] A first acquisition module is used to acquire a sample image, wherein a salient area where a target object is located is marked in the sample image;

[0119] The first training module is used to train the saliency prediction model based on the sample image.

[0120] Optionally, the first image processing module 340 is specifically configured to use the height of the target video frame as the height of the cropping area, wherein the cropping area is an area to be cropped from the target video frame;

[0121] Determine the width of the cropping area according to the height of the cropping area and the center position of the target area, so that the center position of the target area coincides with the center position of the cropping area;

[0122] The cropping area is cropped from the target video frame according to the width and height of the cropping area and the center position of the target area, so as to convert the target video frame from a horizontal screen frame to a vertical screen frame.

[0123] Optionally, the saliency prediction model processes to obtain saliency prediction maps of N target video frames, where the N target video frames are determined from the video to be processed according to a preset interval, and N is a positive integer. The device further includes:

[0124] A tracking module, used for tracking the target areas of the N target video frames using a tracking algorithm to determine the target areas of other video frames in other video frames of the video to be processed, wherein the other video frames are video frames other than the N target video frames;

[0125] The second image processing module is used to perform image processing on each other video frame in a manner of retaining a target area of ​​the other video frame, so as to convert the other video frame into a horizontal screen frame and a vertical screen frame.

[0126] The video processing device provided in the embodiment of the present disclosure can execute any video processing method provided in the embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. The contents not described in detail in the embodiment of the device of the present disclosure can refer to the description in any method embodiment of the present disclosure.

[0127] Figure 4 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0128] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0129] The memory 402 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 402 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 402 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 402 is a non-volatile solid-state memory. In a particular embodiment, the memory 402 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0130] The processor 401 can perform each step in any of the above method embodiments by reading and executing the computer program instructions stored in the memory 402.

[0131] In one example, the electronic device may further include a transceiver 403 and a bus 404. Figure 4 As shown, the processor 401, the memory 402 and the transceiver 403 are connected via a bus 404 and communicate with each other.

[0132] The bus 404 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 404 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0133] The embodiment of the present disclosure further provides a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the time synchronization method provided by the embodiment of the present disclosure.

[0134] The above-mentioned storage medium may, for example, include a memory 402 of computer program instructions, and the above-mentioned instructions may be executed by a processor 401 of an electronic device to complete the time synchronization method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0135] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, characterized in that: include: Performing target detection on the video to be processed to obtain a detection frame of a target video frame and a first confidence level corresponding to the detection frame, wherein the target video frame is a video frame including a target object in the video to be processed, and the detection frame is used to indicate an area in the target video frame where the target object identified by target detection is located; Inputting the target video frame into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame, wherein the saliency prediction map includes a saliency region, and the saliency region is a region in the target video frame where the target object identified by the saliency prediction model is located; Determine a target area according to the saliency prediction map corresponding to the target video frame and the first confidence level corresponding to the detection frame of the target video frame, wherein the target area is an area where a main target object is located in the target video frame; The target video frame is subjected to image processing in a manner of retaining the target area, so as to convert the target video frame into a horizontal frame and a vertical frame.

2. The method according to claim 1, characterized in that The step of inputting the target video frame into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame includes: using the saliency prediction model to perform the following processing on each target video frame: Performing feature extraction on the target video frame to obtain a plurality of first feature maps of different sizes of the target video frame; For each of the first feature graphs, performing sequence generation processing on the first feature graph to obtain a first feature sequence, and processing the first feature sequence based on a multi-head self-attention mechanism to obtain a second feature sequence corresponding to the first feature graph; For each of the first feature graphs, performing feature graph generation processing on a second feature sequence corresponding to the first feature graph to obtain a second feature graph corresponding to the first feature graph, wherein the first feature graph and the second feature graph have the same size; Fusing multiple second feature maps of different sizes to obtain a fused feature map; The target object in the target video frame is predicted according to the fused feature map to obtain the saliency prediction map corresponding to the target video frame.

3. The method according to claim 2, characterized in that The step of predicting the target object in the target video frame according to the fused feature map to obtain the saliency prediction map corresponding to the target video frame includes: Performing upsampling processing on the fused feature map so that the size of the fused feature map is the same as the size of the target video frame corresponding to the fused feature map; The fused feature map after upsampling is processed to generate the saliency prediction map, wherein the saliency prediction map includes the saliency area and the non-saliency area, the value of the pixel in the saliency area is 1, and the value of the pixel in the non-saliency area is 0.

4. The method according to claim 1, characterized in that The determining the target area according to the saliency prediction map corresponding to the target video frame and the first confidence level corresponding to the detection frame of the target video frame includes: For each of the detection boxes, performing a dot multiplication process on the detection box and a corresponding area of ​​the detection box in the saliency prediction map to obtain a second confidence level corresponding to the detection box; For each of the detection frames, weightedly fuse the first confidence and the second confidence of the detection frame to generate a fused confidence of the detection frame; The target area is determined according to a detection frame with the largest fusion confidence among all the detection frames of all the target video frames.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a sample image, wherein a salient area where a target object is located is marked in the sample image; The saliency prediction model is trained based on the sample image.

6. The method according to claim 1, characterized in that The performing image processing on the target video frame in a manner of retaining the target area to convert the target video frame into a horizontal screen frame and a vertical screen frame includes: Using the height of the target video frame as the height of the cropping area, wherein the cropping area is an area to be cropped from the target video frame; Determining the width of the cropping area according to the height of the cropping area and the center position of the target area, so that the center position of the target area coincides with the center position of the cropping area; The cropping area is cropped from the target video frame according to the width and height of the cropping area and the center position of the target area, so as to convert the target video frame from a horizontal frame to a vertical frame.

7. The method according to claim 1, characterized in that The saliency prediction model processes to obtain the saliency prediction maps of N target video frames, wherein the N target video frames are determined from the video to be processed according to a preset interval, and N is a positive integer. The method further includes: Using a tracking algorithm to track the target areas of the N target video frames, so as to determine the target areas of the other video frames in other video frames of the video to be processed, wherein the other video frames are video frames other than the N target video frames; For each of the other video frames, image processing is performed on the other video frame in a manner of retaining the target area of ​​the other video frame, so as to convert the other video frame into a horizontal screen frame and a vertical screen frame.

8. A device for converting video from horizontal to vertical, characterized in that: include: A first detection module, configured to perform target detection on the video to be processed, and obtain a detection frame of a target video frame and a first confidence level corresponding to the detection frame, wherein the target video frame is a video frame including a target object in the video to be processed, and the detection frame is used to indicate an area in the target video frame where the target object identified by target detection is located; A first prediction module is used to input the target video frame into a pre-trained saliency prediction model for processing to obtain a saliency prediction map corresponding to the target video frame, wherein the saliency prediction map includes a saliency region, and the saliency region is a region in the target video frame where the target object identified by the saliency prediction model is located; A first determination module is used to determine a target area according to the saliency prediction map corresponding to the target video frame and the first confidence level corresponding to the detection frame of the target video frame, wherein the target area is an area where a main target object is located in the target video frame; The first image processing module is used to perform image processing on the target video frame in a manner of retaining the target area, so as to convert the target video frame into a horizontal screen frame and a vertical screen frame.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the video processing method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the video processing method described in any one of claims 1 to 7 is implemented.

Images