Video image processing method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a video image processing method and device, electronic equipment and a computer readable medium. A specific embodiment of the method comprises the following steps: adjusting initial video data into a stable frame sequence and an unstable frame sequence; calculating a stable frame feature point coordinate set of each stable frame in the stable frame sequence; calculating an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence; constructing a video frame image training sample set; training the initial video stable image recognition model according to the video frame image training sample set; performing video stable image recognition processing on each video frame in the to-be-detected video according to the video stable image recognition model; and performing video image stabilization processing on the to-be-detected video according to the video stabilization image recognition result sequence. According to the embodiment, the motion mode is accurately distinguished in most unstable videos, and the pixel-by-pixel stable distortion graph is generated, so that an accurate stable frame sequence is generated.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computers, and in particular to a video image processing method, device, electronic device, and computer-readable medium. Background Art

[0002] In recent years, the task of video stabilization has attracted widespread attention from researchers, and some work has achieved excellent results. However, there are still some challenges in this task that have not been well resolved, including the difficulty in ensuring the frame quality of the generated video, the frequent occurrence of local abnormal distortion and texture loss in the generated frames, and the problem of black edges appearing after the video frames are distorted, which will cause some boundary information to be lost.

[0003] It is worth noting that the steps before the image distortion part of the video stabilization task are all aimed at obtaining a distortion map that can be used for distortion, and the distortion map is obtained through motion estimation and trajectory smoothing, or solved by other methods. The existing pixel-level stabilization methods based on deep learning do not perform trajectory estimation and other processes, but generate and estimate the distortion map by learning the video sequence. Specifically, these methods use unstable and stable video pairs as input, obtain the motion patterns of stable and unstable video sequences and corresponding stable video sequences by learning the hidden information between frames, and further learn the transition from unstable motion mode to stable motion mode to estimate the distortion map of the change from unstable frame to stable frame.

[0004] However, although existing methods can achieve this transformation, they are still too rough and still lack the ability to crop and grasp local information, resulting in excessive cropping or local blur. Existing methods are limited by the datasets of stable and unstable video pairs used, which have limited motion patterns. In some complex motion scenes, the model performance is poor.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a video image processing method, an apparatus, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a video image processing method, the method comprising: adjusting initial video data into a stable frame sequence and an unstable frame sequence, wherein a stable frame in the stable frame sequence corresponds to an unstable frame in the unstable frame sequence; calculating a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence; calculating an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence; constructing a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence; training an initial video stable image recognition model according to the video frame image training sample set to obtain a trained video stable image recognition model; in response to receiving a video to be detected, performing video stable image recognition processing on each video frame in the video to be detected according to the video stable image recognition model to obtain a video stable image recognition result sequence, wherein one video frame corresponds to one video stable image recognition result; performing video stabilization processing on the video to be detected according to the video stable image recognition result sequence to obtain an optimized video.

[0009] In a second aspect, some embodiments of the present disclosure provide a video image processing device, the device comprising: an adjustment unit, configured to adjust initial video data into a stable frame sequence and an unstable frame sequence, wherein a stable frame in the stable frame sequence corresponds to an unstable frame in the unstable frame sequence; a first calculation unit, configured to calculate a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence; a second calculation unit, configured to calculate an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence; a construction unit, configured to construct a stable frame feature point coordinate set sequence based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence. A label set sequence is used to construct a video frame image training sample set; a training unit is configured to train an initial video stabilized image recognition model according to the video frame image training sample set to obtain a trained video stabilized image recognition model; a recognition unit is configured to, in response to receiving a video to be detected, perform video stabilized image recognition processing on each video frame in the video to be detected according to the video stabilized image recognition model to obtain a video stabilized image recognition result sequence, wherein one video frame corresponds to one video stabilized image recognition result; and a stabilization unit is configured to perform video stabilization processing on the video to be detected according to the video stabilized image recognition result sequence to obtain an optimized video.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the video image processing method of some embodiments of the present disclosure, the overall framework of the pixel-level video stabilization algorithm based on the dual branches of CNN and Transformer is elaborated in detail, and the specific design details of the CNN branch, the Transformer branch and the hierarchical adaptive fusion module (HAFM) are described. The hierarchical adaptive fusion module (HAFM) can use convolution features and attention features to better learn the motion information in the frame sequence, establish a direct interaction between the convolution features and the attention features, and model a higher quality inter-frame motion pattern. The hierarchical adaptive fusion module (HAFM) deeply integrates local detail information and global visual information, thereby effectively refining the information in each receptive field. The model can accurately identify motion patterns in most unstable videos and generate pixel-by-pixel stable distortion maps, thereby generating accurate stable frame sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the video image processing method according to the present disclosure;

[0015] Figure 2 is a schematic structural diagram of some embodiments of the video image processing device according to the present disclosure;

[0016] Figure 3 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure;

[0017] Figure 4 is an exemplary grid point change diagram in the video image processing method according to the present disclosure;

[0018] Figure 5are some structural flow charts of the video stabilization image recognition model in the video image processing method disclosed in the present invention;

[0019] Figure 6 It is a single-layer structure diagram of a network TUNet generated in the video image processing method disclosed in the present invention;

[0020] Figure 7 is a structural flow chart of HAFM in the video image processing method according to the present disclosure;

[0021] Figure 8 is a structural flow chart of a discriminator in the video image processing method according to the present disclosure;

[0022] Fig. 9 It is a flow chart of the calculation of the self-attention score in the video image processing method disclosed in the present invention. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0025] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0029] Figure 11 is a flow chart of some embodiments of the video image processing method according to the present disclosure. 100 of some embodiments of the video image processing method according to the present disclosure is shown. The video image processing method comprises the following steps:

[0030] Step 101: adjust initial video data into a stable frame sequence and an unstable frame sequence.

[0031] In some embodiments, the execution subject (e.g., a computing device) of the video image processing method may adjust the initial video data into a stable frame sequence and an unstable frame sequence. Among them, the stable frame in the stable frame sequence corresponds to the unstable frame in the unstable frame sequence. The initial video data may refer to the video to be processed. For example, video editing software may be used to perform video stabilization processing on each frame of video image in the initial video data to obtain a stable frame sequence. Video editing software may be used to perform video jitter processing on each frame of video image in the initial video data to obtain an unstable frame sequence.

[0032] As an example, the video data frame can be adjusted to a stable frame sequence and unstable frame sequences Where n is the number of frames in the initial video data, For subsequent training and calculation, the height and width of the video frame can be adjusted to 256, that is, H = W = 256, C = 3, where H, W, and C represent the height, width, and number of channels of the frame, respectively.

[0033] Step 102 , calculating a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence.

[0034] In some embodiments, the execution subject may calculate a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence. The stable frame feature point coordinates may refer to the SURF (Speed ​​Up Robust Features) feature point coordinates of the stable frame. For example, the stable frame feature point coordinate set of each stable frame in the stable frame sequence may be calculated by constructing a Hessian matrix or detecting extreme values ​​in a scale space.

[0035] Constructing the Hessian matrix: The core of the SURF algorithm is to construct the Hessian matrix. The Hessian matrix is ​​a second-order derivative matrix that is used to describe the local curvature of an image. The Hessian matrix can be constructed by calculating the second-order partial derivatives of the image. To simplify the calculation, SURF uses the determinant approximation of the Hessian matrix to detect feature points.

[0036] Detection of extreme values ​​in scale space: After constructing the Hessian matrix, potential feature points are identified by detecting extreme points in scale space. These extreme points need to be the largest or smallest at all scales to ensure the scale and rotation invariance of the feature points.

[0037] Step 103, calculating an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence.

[0038] In some embodiments, the execution subject may calculate the unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence. The unstable frame feature point coordinates may refer to the SURF (Speed ​​Up Robust Features) feature point coordinates of the unstable frame. For example, the unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence may be calculated by constructing a Hessian matrix or by detecting extreme values ​​in a scale space.

[0039] Step 104: construct a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence.

[0040] In some embodiments, the execution subject may construct a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence.

[0041] In practice, the above execution entity can construct a video frame image training sample set through the following steps:

[0042] In the first step, for each stable frame in the stable frame sequence, the following processing steps are performed:

[0043] 1. Determine a stable frame feature point coordinate set of the stable frame. The stable frame feature point coordinate set may be P: {(x1, y1), (x2, y2), ..., (x n ,y n )}. P can represent a stable frame feature point coordinate set.

[0044] 2. Determine the stable frame feature point coordinate set of the next stable frame of the stable frame as the adjacent stable frame feature point coordinate set. The adjacent stable frame feature point coordinate set can be P′: {(x′1, y′1), (x′2, y′2), …, (x′ n , y′ n )}. P′ can represent the coordinate set of feature points in the unstable frame.

[0045] 3. Normalize the coordinates of each stable frame feature point in the stable frame feature point coordinate set to obtain a normalized stable frame feature point coordinate set. For example, each stable frame feature point coordinate is normalized to [-1, 1].

[0046] 4. Normalize the coordinates of each adjacent stable frame feature point in the adjacent stable frame feature point coordinate set to obtain a normalized adjacent stable frame feature point coordinate set. For example, each adjacent stable frame feature point coordinate is normalized to [-1, 1].

[0047] 5. Construct a stable frame affine matrix between the stable frame and the next stable frame according to the normalized stable frame feature point coordinate set and the normalized adjacent stable frame feature point coordinate set.

[0048] For example, construct a stable frame affine matrix between the stable frame and the next stable frame

[0049]

[0050] A can refer to the stable frame affine matrix

[0051] 6. Constructing a stable frame feature point coordinate vector of the stable frame according to the normalized stable frame feature point coordinate set.

[0052] For example, the coordinate vector of the feature point in the stable frame can be:

[0053] B=[x1,x2,...,x n , y1, y2, ..., y n ] T ,

[0054] Among them, B represents the coordinate vector of the feature point of the stable frame, and T represents the matrix transpose.

[0055] In addition, in order to obtain the affine vector X, a linear equation can also be constructed: A*X=B.

[0056] Where X = [a1, a2, a3, a4, a5, a6] T .

[0057] In the second step, for each unstable frame in the unstable frame sequence, the following processing steps are performed:

[0058] 1. Determine an unstable frame feature point coordinate set of the unstable frame.

[0059] 2. Determine an unstable frame feature point coordinate set of the next unstable frame of the unstable frame as an adjacent unstable frame feature point coordinate set.

[0060] 3. Normalize the coordinates of each unstable frame feature point in the unstable frame feature point coordinate set to obtain a normalized unstable frame feature point coordinate set.

[0061] 4. Normalizing the coordinates of each adjacent unstable frame feature point in the adjacent unstable frame feature point coordinate set to obtain a normalized adjacent unstable frame feature point coordinate set.

[0062] 5. Construct an unstable frame affine matrix between the unstable frame and the next unstable frame according to the normalized unstable frame feature point coordinate set and the normalized adjacent unstable frame feature point coordinate set.

[0063] 6. Constructing an unstable frame feature point coordinate vector of the unstable frame according to the normalized unstable frame feature point coordinate set.

[0064] Here, for the processing method of the unstable frame, refer to the processing steps of the stable frame, which will not be repeated here.

[0065] In the third step, for each stable frame in the stable frame sequence, the following processing steps are performed:

[0066] 1. Determine an unstable frame corresponding to the stable frame as an associated unstable frame.

[0067] 2. Calculate the correlation affine matrix between the stable frame and the associated unstable frame. For example, the correlation affine matrix can be expressed as Here, the construction of the associated affine matrix can refer to the construction method of the stable frame affine matrix, which will not be repeated here.

[0068] 3. Convert the associated affine matrix into a corresponding video frame distortion map. The video frame distortion map can be used Indicates. Among them, is the deformed X,

[0069] 4. Determine the stable frame, the associated unstable frame, the video frame distortion map, the stable frame feature point coordinate set corresponding to the stable frame, the adjacent stable frame feature point coordinate set, the stable frame affine matrix, the stable frame feature point coordinate vector, and the unstable frame feature point coordinate set corresponding to the associated unstable frame, the adjacent unstable frame feature point coordinate set, the unstable frame affine matrix, and the unstable frame feature point coordinate vector as video frame image training samples.

[0070] Step 105 , training the initial video-stabilized image recognition model according to the video frame image training sample set to obtain a trained video-stabilized image recognition model.

[0071] In some embodiments, the execution subject may train the initial video stabilization image recognition model according to the video frame image training sample set to obtain a trained video stabilization image recognition model. The initial video stabilization image recognition model may be a neural network model including an initial convolutional neural network layer, an initial attention mechanism layer, and a hierarchical adaptive fusion module.

[0072] It should be noted that, during the training process, each sampled frame (video frame image training sample) and 15 consecutive frames before and after it (video frame image training samples) are trained, that is, θ=15 in the algorithm. Grid represents the predicted video frame distortion map, A′ represents the video frame distortion map predicted without using the grid, and U′ and P represent the frames obtained by distorting the grid and A′. and Represent matrices of all -1 and all 1 respectively.

[0073] In practice, the above execution entity can train the initial video stabilization image recognition model through the following steps:

[0074] The first step is to select a target video frame image training sample from the video frame image training sample set. A video frame image training sample can be randomly selected from the video frame image training sample set as the target video frame image training sample.

[0075] In the second step, the target video frame image training sample is input into the initial video stabilized image recognition model to obtain the initial video stabilized image recognition result. The initial video stabilized image recognition result includes: the initial video stabilized image content recognition result, the initial video stabilized image shape recognition result, and the initial video stabilized image frame time recognition result. The initial video stabilized image content recognition result may refer to the recognition result of the content represented by the video stabilized image in the target video frame image training sample. The initial video stabilized image shape recognition result may refer to the recognition result of the shape of the video stabilized image in the target video frame image training sample. The initial video stabilized image frame time recognition result may refer to the recognition result of the time of the video stabilized image in the target video frame image training sample.

[0076] As an example, the training process of the initial video stabilization image recognition model is shown in the following figure:

[0077]

[0078] The third step is to determine the loss information between the initial video stabilized image recognition result and the corresponding sample label based on a preset loss function.

[0079] The third step may include the following sub-steps:

[0080] The first sub-step is to determine the image content loss information between the initial video stabilized image recognition result and the corresponding sample label based on the content loss function included in the loss function. The sample label may include: a video stabilized image content information label, a video stabilized image shape label, and a video stabilized image frame time label.

[0081] For example, the content loss function can be:

[0082] L content =∑ i∈[t，t+1] MSE(S i -P i )+L VGG (S i , P i ),

[0083] Among them, S represents the basic facts of the stabilized frame (the content information label of the video stabilized image), P is the predicted stabilized frame (the initial video stabilized image recognition result), and MSE is the mean square error function. This paper uses the MSE loss function to penalize the deviation between the predicted value and the true value. Using the pre-trained VGG 16 The maximum pooling layer before the fully connected layer of the network is used to calculate L VGG (S i , P i )The mean square loss in feature space is as follows:

[0084] L VGG (S i , P i )=MSE(VGG 16 (S i )-VGG 16 (P i )).

[0085] The second sub-step is to determine the image shape loss information between the initial video stabilized image recognition result and the corresponding sample label based on the image shape loss function included in the loss function.

[0086] For example, an image shape loss function can be:

[0087] L shape =L point (γ,S i , P i )+L grid (S i , P i ).

[0088] Among them, L point (γ,S i , P i) calculates the loss between the feature points in the two frames, which is used to penalize some abnormal feature point changes after distortion. grid (S i , P i ) indicates that the grid loss is as follows:

[0089] L grid (S i , P i )=L relative (S i , P i )+L adjacent (S i , P i ).

[0090] In addition, the sparse grid vector difference method between adjacent points was used to handle shape changes during the training process, and the feasibility of this method was further verified in subsequent experiments. Figure 4 shown.

[0091] Essentially, this is a mathematical model that quantifies the differences in sparse points between adjacent frames. It computes the vector differences between four adjacent points at a specific midpoint within the grid and aggregates them separately. relative (S i , P i ) quantifies the change in the relationship between the vector differences between adjacent points with relative positions, such as Figure 4 (a). Their vector summation is calculated as follows:

[0092]

[0093] Among them, δ is the set of points in the grid within the frame, n is the number of points in the grid, and point and Located at point v i The upper and lower positions, or point v i The left and right position of v i Non-adjacent adjacent grid points. The geometric meaning of the result is the length of vector 1 in the figure. L adjacent (S i , P i ) Use as Figure 4 The dot product of the vectors shown in (b) is used to evaluate the change between adjacent points, which is calculated as follows:

[0094]

[0095] in, and It is point v i The geometric meaning of the result is n in the figure, that is, the projection size of the vector.

[0096] The third sub-step is to determine the image frame time loss information between the initial video stabilized image recognition result and the corresponding sample label based on the image frame time loss function included in the loss function.

[0097] For example, the image frame temporal loss function can be:

[0098]

[0099] in, The predicted frame P i-1 Twisted to P i function.

[0100] The fourth sub-step is to combine the image content loss information, the image shape loss information and the image frame time loss information into loss information.

[0101] In step 4, in response to determining that the loss information satisfies a convergence condition, the initial video-stabilized image recognition model is determined as a trained video-stabilized image recognition model. The convergence condition may refer to the loss values ​​represented by the image content loss information, the image shape loss information, and the image frame time loss information being all less than corresponding loss thresholds.

[0102] Optionally, a model structure of the initial video-stabilized image recognition model is determined.

[0103] In some embodiments, the above-mentioned execution subject may determine the model structure of the initial video stabilization image recognition model through the following steps:

[0104] The first step is to determine the initial convolutional neural network layer and the initial attention mechanism layer of the initial video stabilization image recognition model.

[0105] The second step is to determine the hierarchical adaptive fusion module connecting the initial convolutional neural network layer and the initial attention mechanism layer.

[0106] The structural description of the initial video stabilization image recognition model, such as Figure 5 The example is described as follows:

[0107] (1) A new framework, TranStableNet, is proposed, which combines Transforme and CNN branches and strikes a balance between processing speed and stability performance.

[0108] (2) Tranformer-UNet (TUNet) is proposed, which is able to generate a warp map that encapsulates global context and local feature information. The warp map incorporates various semantic information of the image to achieve a more robust and stable geometric transformation, thereby guiding the reconstruction of the video stabilizer.

[0109] (3) A Hierarchical Adaptive Fusion Module (HAFM) is proposed, which improves the quality of the warped map by integrating attention visual features and multi-level visual features in a multi-scale adaptive fusion manner.

[0110] (4) This paper designs a discriminator STSD (SwinTransformerStable Discriminator) based on SwinTransformerV2, which adopts the strategy of pixel-by-pixel rendering and high-quality image reconstruction. It is a key component to alleviate black edges and achieve a full field of view (FOV). The framework achieves new optimal performance on several benchmark datasets. A large number of experimental results demonstrate the effectiveness of the proposed method.

[0111] At present, in order to solve the above challenges, this paper analyzes the motion difference between stable frame sequences and unstable frame sequences, and explores the collaborative modeling of high-quality inter-frame motion features by CNN and Transformer with the idea of ​​multi-level feature interaction as the core. In addition, this paper deeply integrates high-level semantics and low-level visual detail information to obtain more refined distortion features. Specifically, the overall framework of the pixel-level video stabilization algorithm TranStableNet based on CNN and Transformer dual branches is as follows Figure 5 As shown in the figure, the model first extracts the motion features from two adjacent unstable frame sequences through two TUNets (Tranformer UNet) with shared parameters. Each network will obtain distortion maps in both horizontal and vertical directions. Then, STSD (Swin Transformer Stable Discriminator) is used to discriminate the estimated distortion map from the stable frame of the corresponding central frame and the estimated stable frame.

[0112] This application gives an unstable video and a corresponding stable video as a video pair, and takes an unstable frame sequence centered on unstable adjacent frames Ut and Ut+1, that is, the sequence [Ut-Δt, Ut+Δt] and [Ut-Δt+1, Ut+Δt+1]. The model first processes the two sequences separately through two TUNet (Tranformer UNet) networks with shared parameters to obtain two adjacent distortion maps Wt and wt+1, wherein TUNet uses CNN and Transformer dual branches to learn the mapping relationship from unstable video to stable video, and generates a pixel-by-pixel distortion map to process the video content more finely. Then, the stable frame obtained with the distortion map and the real stable frame are discriminated by a discriminator based on Swin Transformer, and multiple rounds of iterations are performed to enhance the generation and discrimination capabilities of the model. The method proposed in this paper constructs a dual-branch network to better learn the information in the video, and achieves more accurate stable frame prediction through the discriminator. The model only uses TUNet for image stabilization in the subsequent reasoning process.

[0113] Specifically, the single-layer structure of the generated network TUNet is as follows Figure 6 As shown in the figure, a dual-branch network of UNet and Transformer is used, and a hierarchical adaptive fusion module (HAFM) is used to connect the dual branches. The CNN branch (initial convolutional neural network layer) and the Transforme (initial attention mechanism layer) branch are composed of 7 3×3 convolution blocks, 7 deconvolution blocks, and 8 Transformer blocks with the same structure. The Transformer branches between layers are isolated from each other. In the convolution branch, corresponding convolution and deconvolution blocks are designed to share tensor information downward.

[0114] The description of the generator is as follows:

[0115] (1) Transformer branch: This paper first processes the input frame sequence into an appropriate size and then embeds it into the size required by the Transformer branch. The Transformer branch uses a 12-head multi-head attention mechanism, and the rest is consistent with the original Transformer encoder, that is, it includes a multi-head attention mechanism layer, a feedforward neural network layer, and a residual normalization process. In the middle part of the corresponding CNN branch, as the depth of the downsampling part increases, the tensor size of the convolutional features in the middle part is not enough to support the sampling process of patch embedding, so HAFM is no longer used for feature coupling in this part.

[0116] (2) CNN branch: The CNN branch adopts the UNet structure. Figure 6As shown in the figure, the resolution of the feature map decreases with the depth of the network, while the number of channels increases. The left side is the feature extraction network (encoder), and the right side is the feature fusion network (decoder). The encoder is used to capture the contextual information of the image, while the decoder is used to restore the details and generate prediction results with the same resolution as the input image, so as to achieve the modeling of local details and global context at the same time.

[0117] During the downsampling process, a 3×3 spatial convolution and a residual connection between the input and output are used, and then the features are coupled with the Transformer branch through HAFM, and finally batch normalization and ReLU activation function are performed. At the end of downsampling, the intermediate result is passed through the fully connected layer to obtain the first output A^ of the convolution branch.

[0118] In the above, 4×4 deconvolution is used, and the features are coupled with the Transformer branch through HAFM. Then, the spatial information that may be lost in the decoder stage is restored through the skip connection layer. The second output W^ is obtained after upsampling. After that, the output T^ of the Transformer branch is unified to the shape of the distorted graph and added to the two outputs to obtain the result W required for the generation network part.

[0119] (3) HAFM: The key in the TUNet (Transformer UNet) structure is how to couple the convolutional features in the CNN branch with the embedded features in the Transformer. To solve this important problem, this paper designs a hierarchical adaptive fusion module (HAFM) to fuse the convolutional features in the CNN branch with the embedded features in the Transformer. The HAFM designed in this paper has the following structure: Figure 7 As shown, first, using convolutional modules along the branches, HAFM gradually integrates feature maps and patch embeddings in an interactive manner. In order to keep a consistent embedding size, an asymmetric sampling step strategy is adopted during the interaction.

[0120] Specifically, since the convolution feature dimension is C×H×W (C, H, W represent the number of channels, height and width of the tensor respectively), and the shape of the embedding is (T+1)×E (T and E represent the number of image embeddings and embedding dimension respectively), it involves the conversion between two-dimensional and three-dimensional features. Since the dimension size of the embedded feature remains unchanged, while the convolution feature keeps changing with the up- and down-sampling process, the direct connection method fails.

[0121] To achieve this, HAFM uses upsampling and downsampling to connect the two, and in the feature conversion process, different step sizes are used for up and down sampling, controlling the convolution and deconvolution of each step to maintain consistent feature size through different step sizes. In terms of spatial dimensions, HAFM uses different embedding sizes to ensure that the embedding dimensions are the same after each sampling.

[0122] The description of the discriminator is as follows:

[0123] TUNet can generate relatively stable frame sequences after the first 20 iterations of training, but there will be abnormal distortions in local details and the proportions of objects and people inside the image. Therefore, a discriminator STSD (Swin Transformer Stable Discriminator) is designed for pixel-by-pixel rendering and high-quality image reconstruction. Compared with traditional convolutional layers, Transformer blocks have better performance in existing GAN models. Since the introduction of the Swin Transformer as a whole has high computational complexity and resource consumption, the discriminator designed in this paper has a structure diagram as shown below. Figure 8 shown.

[0124] Two Swin Transformer V2 blocks are used in the discriminator for feature extraction. When using traditional self-attention score calculation in visual models, the similarity of pixel pairs is calculated as the dot product of the query and key vectors. The learned attention maps of some blocks and heads are often dominated by a few pixel pairs. The method used by Swin Transformer V2 has been shown to alleviate this phenomenon. The Swin Transformer V2 block is different from the traditional self-attention score calculation method. Fig. 9 As shown, it introduces a scaled cosine attention method to deal with this phenomenon, as shown below:

[0125] Attention(Q,K,V)=softmax(Sim(Q,K))·V.

[0126] Among them, Sim(Q, K) is the scaled cosine similarity calculation function as shown below:

[0127] Sim(Q,K)=cosine(Q·K) / τ+B.

[0128] Among them, cosine(Q·K) calculates the cosine similarity between the query vector Q and the key vector K, and τ is a learnable scalar used to adjust the size of the similarity. is the relative position deviation term of each head.

[0129] Step 106 , in response to receiving the video to be detected, performing video-stabilized image recognition processing on each video frame in the video to be detected according to the video-stabilized image recognition model to obtain a video-stabilized image recognition result sequence.

[0130] In some embodiments, the execution subject may, in response to receiving the video to be detected, perform video stabilization image recognition processing on each video frame in the video to be detected according to the video stabilization image recognition model to obtain a video stabilization image recognition result sequence. Wherein, one video frame corresponds to one video stabilization image recognition result. The video to be detected may refer to the video to be detected.

[0131] For example, each video frame in the video to be detected can be input into a video stabilization image recognition model to obtain a video stabilization image recognition result sequence. The video stabilization image recognition result can indicate whether the corresponding video frame is a stable frame.

[0132] Step 107: Perform video stabilization processing on the video to be detected according to the video stabilization image recognition result sequence to obtain an optimized video.

[0133] In some embodiments, the execution subject may perform video stabilization processing on the video to be detected according to the video stabilization image recognition result sequence to obtain an optimized video. The optimized video may be a video composed of adjusting each video frame in the video to be detected into each stabilized frame.

[0134] For example, the reasoning process of the image stabilization algorithm is as follows:

[0135] The above execution body can be based on the needs of the network. After obtaining the height, width, H, W and number of frames n, TranStableNet scales the frame sequence to 256*256. Since this article needs to combine the information of the previous and next frames for image stabilization, the unstable frame sequence is first divided into three parts, namely the first 15 frames, the middle part and the last 15 frames. For ease of processing, a sliding window list with a length of 31 is introduced. The generator network TUNet (Transformer UNet) is used to restrict the unstable frame sequence of the input network. When the window is initialized, the first 15 frames are initialized as the first frame, and the last 16 frames are the first 16 frames:

[0136]

[0137] In each processing, the middle frame of the window is the current frame. In each subsequent processing, the first frame in the window is discarded and the 15th frame after the current frame is added. If the current frame is the last 15 frames, the last frame is added. Each processed frame is cropped and scaled to restore to its original size and added to the stable frame sequence as a stable frame.

[0138] Specifically, the process of video stabilization can be shown as follows:

[0139]

[0140] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a video image processing device. These video image processing device embodiments are Figure 1 Corresponding to the method embodiments shown, the video image processing device can be specifically applied to various electronic devices.

[0141] like Figure 2 As shown, the video image processing apparatus 200 of some embodiments includes: an adjustment unit 201, a first calculation unit 202, a second calculation unit 203, a construction unit 204, a training unit 205, a recognition unit 206 and an image stabilization unit 207. The adjustment unit 201 is configured to adjust the initial video data into a stable frame sequence and an unstable frame sequence, wherein a stable frame in the stable frame sequence corresponds to an unstable frame in the unstable frame sequence; the first calculation unit 202 is configured to calculate a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence; the second calculation unit 203 is configured to calculate an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence; the construction unit 204 is configured to construct a video frame image training unit 204 according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence. training sample set; a training unit 205, configured to train an initial video-stabilized image recognition model according to the video frame image training sample set to obtain a trained video-stabilized image recognition model; a recognition unit 206, configured to, in response to receiving a video to be detected, perform video-stabilized image recognition processing on each video frame in the video to be detected according to the video-stabilized image recognition model to obtain a video-stabilized image recognition result sequence, wherein one video frame corresponds to one video-stabilized image recognition result; and a stabilization unit 207, configured to perform video-stabilization processing on the video to be detected according to the video-stabilized image recognition result sequence to obtain an optimized video.

[0142] It is understandable that the units described in the video image processing device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the video image processing device 200 and the units included therein, and will not be described in detail here.

[0143] Reference below Figure 3, which shows a schematic diagram of the structure of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0144] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0145] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0146] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0147] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0148] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0149] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: adjusts the initial video data into a stable frame sequence and an unstable frame sequence, wherein a stable frame in the stable frame sequence corresponds to an unstable frame in the unstable frame sequence; calculates a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence; calculates an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence; constructs a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence; trains an initial video stable image recognition model according to the video frame image training sample set to obtain a trained video stable image recognition model; in response to receiving a video to be detected, performs video stable image recognition processing on each video frame in the video to be detected according to the video stable image recognition model to obtain a video stable image recognition result sequence, wherein one video frame corresponds to one video stable image recognition result; performs video stabilization processing on the video to be detected according to the video stable image recognition result sequence to obtain an optimized video.

[0150] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: an adjustment unit, a first calculation unit, a second calculation unit, a construction unit, a training unit, a recognition unit and an image stabilization unit. Among them, the names of these units do not constitute limitations on the units themselves in certain circumstances, for example, the construction unit may also be described as "a unit for constructing a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence".

[0153] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0154] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A video image processing method, comprising: Adjusting the initial video data into a stable frame sequence and an unstable frame sequence, wherein the stable frames in the stable frame sequence correspond to the unstable frames in the unstable frame sequence; Calculating a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence; Calculating an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence; Constructing a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence; Training an initial video stabilization image recognition model according to the video frame image training sample set to obtain a trained video stabilization image recognition model; In response to receiving the video to be detected, performing video-stabilized image recognition processing on each video frame in the video to be detected according to the video-stabilized image recognition model to obtain a video-stabilized image recognition result sequence, wherein one video frame corresponds to one video-stabilized image recognition result; According to the video stabilization image recognition result sequence, the video to be detected is subjected to video stabilization processing to obtain an optimized video.

2. The method according to claim 1, wherein: The step of constructing a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence comprises: For each stable frame in the stable frame sequence, the following processing steps are performed: Determining a stable frame feature point coordinate set of the stable frame; Determine a stable frame feature point coordinate set of a stable frame next to the stable frame as an adjacent stable frame feature point coordinate set; Normalizing the coordinates of each stable frame feature point in the stable frame feature point coordinate set to obtain a normalized stable frame feature point coordinate set; Normalizing the coordinates of each adjacent stable frame feature point in the adjacent stable frame feature point coordinate set to obtain a normalized adjacent stable frame feature point coordinate set; Constructing a stable frame affine matrix between the stable frame and the next stable frame according to the normalized stable frame feature point coordinate set and the normalized adjacent stable frame feature point coordinate set; Constructing a stable frame feature point coordinate vector of the stable frame according to the normalized stable frame feature point coordinate set; For each unstable frame in the unstable frame sequence, the following processing steps are performed: Determining an unstable frame feature point coordinate set of the unstable frame; Determine an unstable frame feature point coordinate set of an unstable frame next to the unstable frame as an adjacent unstable frame feature point coordinate set; Normalizing the coordinates of each unstable frame feature point in the unstable frame feature point coordinate set to obtain a normalized unstable frame feature point coordinate set; Normalizing the coordinates of each adjacent unstable frame feature point in the adjacent unstable frame feature point coordinate set to obtain a normalized adjacent unstable frame feature point coordinate set; Constructing an unstable frame affine matrix between the unstable frame and the next unstable frame according to the normalized unstable frame feature point coordinate set and the normalized adjacent unstable frame feature point coordinate set; According to the normalized unstable frame feature point coordinate set, an unstable frame feature point coordinate vector of the unstable frame is constructed.

3. The method according to claim 2, wherein: The step of constructing a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence further includes: For each stable frame in the stable frame sequence, the following processing steps are performed: determining an unstable frame corresponding to the stable frame as an associated unstable frame; Calculating an associated affine matrix between the stable frame and the associated unstable frame; Converting the associated affine matrix into a corresponding video frame distortion map; The stable frame, the associated unstable frame, the video frame distortion map, the stable frame feature point coordinate set corresponding to the stable frame, the adjacent stable frame feature point coordinate set, the stable frame affine matrix, the stable frame feature point coordinate vector, and the unstable frame feature point coordinate set corresponding to the associated unstable frame, the adjacent unstable frame feature point coordinate set, the unstable frame affine matrix, and the unstable frame feature point coordinate vector are determined as video frame image training samples.

4. The method according to claim 1, wherein: The step of training the initial video stabilized image recognition model according to the video frame image training sample set to obtain the trained video stabilized image recognition model comprises: Selecting a target video frame image training sample from the video frame image training sample set; Input the target video frame image training sample into the initial video stabilized image recognition model to obtain the initial video stabilized image recognition result, wherein the initial video stabilized image recognition result includes: the initial video stabilized image content recognition result, the initial video stabilized image shape recognition result, and the initial video stabilized image frame time recognition result; Based on a preset loss function, determining loss information between the initial video stabilized image recognition result and the corresponding sample label; In response to determining that the loss information satisfies a convergence condition, the initial video-stabilized image recognition model is determined as a trained video-stabilized image recognition model.

5. The method according to claim 4, wherein: The determining, based on a preset loss function, loss information between the initial video stabilized image recognition result and the corresponding sample label includes: Based on a content loss function included in the loss function, determining image content loss information between the initial video stabilized image recognition result and the corresponding sample label; Based on the image shape loss function included in the loss function, determining image shape loss information between the initial video stabilized image recognition result and the corresponding sample label; Based on the image frame time loss function included in the loss function, determining the image frame time loss information between the initial video stabilized image recognition result and the corresponding sample label; The image content loss information, the image shape loss information and the image frame time loss information are combined into loss information.

6. The method according to claim 1, wherein: Before training the initial video-stabilized image recognition model according to the video frame image training sample set to obtain the trained video-stabilized image recognition model, the method further includes: Determining a model structure of the initial video stabilization image recognition model; And the model structure of the initial video stabilization image recognition model is determined, including: Determining an initial convolutional neural network layer and an initial attention mechanism layer of the initial video stabilized image recognition model; Determine the hierarchical adaptive fusion module connecting the initial convolutional neural network layer and the initial attention mechanism layer.

7. A video image processing device, comprising: an adjusting unit configured to adjust the initial video data into a stable frame sequence and an unstable frame sequence, wherein a stable frame in the stable frame sequence corresponds to an unstable frame in the unstable frame sequence; A first calculation unit is configured to calculate a stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence; A second calculation unit is configured to calculate an unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence; A construction unit, configured to construct a video frame image training sample set according to the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence; A training unit is configured to train an initial video-stabilized image recognition model according to the video frame image training sample set to obtain a trained video-stabilized image recognition model; The recognition unit is configured to, in response to receiving the video to be detected, perform video-stabilized image recognition processing on each video frame in the video to be detected according to the video-stabilized image recognition model to obtain a video-stabilized image recognition result sequence, wherein one video frame corresponds to one video-stabilized image recognition result; The image stabilization unit is configured to perform video stabilization processing on the video to be detected according to the video stabilization image recognition result sequence to obtain an optimized video.

8. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images