Video image processing method and apparatus, electronic device, and computer readable medium

Abstract

This disclosure discloses video image processing methods, apparatuses, electronic devices, and computer-readable media. One specific embodiment of the method includes: adjusting initial video data into a stable frame sequence and an unstable frame sequence; calculating the coordinate set of stable frame feature points for each stable frame in the stable frame sequence; calculating the coordinate set of unstable frame feature points for each unstable frame in the unstable frame sequence; constructing a video frame image training sample set; training an initial video stabilization image recognition model based on the video frame image training sample set; performing video stabilization image recognition processing on each video frame in the video to be detected based on the video stabilization image recognition model; and performing video stabilization processing on the video to be detected based on the video stabilization image recognition result sequence. This embodiment accurately identifies motion patterns in most unstable videos and generates pixel-by-pixel stable distortion maps, thereby generating an accurate stable frame sequence.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computers, and more particularly to video image processing methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0002] In recent years, video stabilization has attracted widespread attention from researchers, and some works have achieved remarkable results. However, some challenges in this task have not yet been well resolved, including the difficulty in guaranteeing the quality of frames in the generated video, the frequent occurrence of local abnormal distortions and texture loss in the generated frames, and the black border problem that appears after video frames are distorted, which leads to the loss of some boundary information.

[0003] It's worth noting that the steps preceding image distortion in video stabilization are all aimed at obtaining a distortion map that can be used for distortion. This distortion map is obtained either through motion estimation and trajectory smoothing, or through other methods. Existing deep learning-based pixel-level stabilization methods do not perform trajectory estimation; instead, they generate and estimate the distortion map by learning from the video sequence. Specifically, these methods use unstable and stable video pairs as input. By learning inter-frame hidden information, they obtain the motion patterns of stable and unstable video sequences and their corresponding stable video sequences. Furthermore, they learn the transition from unstable motion patterns to stable motion patterns to estimate the distortion map of the change from unstable to stable frames.

[0004] However, while existing methods can achieve this transformation, they remain too coarse, lacking in cropping and local information handling, leading to problems such as excessive cropping or local blurring. Existing methods are also limited by the datasets containing stable and unstable video pairs, which have a limited range of motion patterns, resulting in suboptimal model performance in some complex motion scenes.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide video image processing methods, apparatuses, electronic devices, and computer-readable media to solve one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a video image processing method, which includes: adjusting initial video data into a stable frame sequence and an unstable frame sequence, wherein stable frames in the stable frame sequence correspond to unstable frames in the unstable frame sequence; calculating the stable frame feature point coordinate set of each stable frame in the stable frame sequence to obtain a stable frame feature point coordinate set sequence; calculating the unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence to obtain an unstable frame feature point coordinate set sequence; constructing a video frame image training sample set based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence; training an initial video stable image recognition model based on the video frame image training sample set to obtain a trained video stable image recognition model; responding to receiving a video to be detected, performing video stable image recognition processing on each video frame in the video to be detected according to the video stable image recognition model to obtain a video stable image recognition result sequence, wherein one video frame corresponds to one video stable image recognition result; and performing video stabilization processing on the video to be detected based on the video stable image recognition result sequence to obtain an optimized video.

[0009] Secondly, some embodiments of this disclosure provide a video image processing apparatus, comprising: an adjustment unit configured to adjust initial video data into a stable frame sequence and an unstable frame sequence, wherein stable frames in the stable frame sequence correspond to unstable frames in the unstable frame sequence; a first calculation unit configured to calculate a stable frame feature point coordinate set for each stable frame in the stable frame sequence, thereby obtaining a stable frame feature point coordinate set sequence; a second calculation unit configured to calculate an unstable frame feature point coordinate set for each unstable frame in the unstable frame sequence, thereby obtaining an unstable frame feature point coordinate set sequence; and a construction unit configured to construct the unstable frame feature point coordinate set sequence based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence. A standard set sequence is used to construct a training sample set of video frame images. A training unit is configured to train an initial video-stabilized image recognition model based on the training sample set of video frame images to obtain a trained video-stabilized image recognition model. A recognition unit is configured to, in response to receiving a video to be detected, perform video-stabilized image recognition processing on each video frame in the video to be detected according to the video-stabilized image recognition model to obtain a video-stabilized image recognition result sequence, wherein one video frame corresponds to one video-stabilized image recognition result. An image stabilization unit is configured to perform video image stabilization processing on the video to be detected according to the video-stabilized image recognition result sequence to obtain an optimized video.

[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0012] The above embodiments of this disclosure have the following beneficial effects: Through the video image processing methods of some embodiments of this disclosure, based on the overall framework of a pixel-level video stabilization algorithm with dual branches of CNN and Transformer, the specific design details of the CNN branch, Transformer branch, and Hierarchical Adaptive Fusion Module (HAFM) are elaborated. The Hierarchical Adaptive Fusion Module (HAFM) can utilize convolutional features and attention features to better learn motion information in frame sequences, establish a direct interaction between convolutional features and attention features, and model higher-quality inter-frame motion patterns. The Hierarchical Adaptive Fusion Module (HAFM) deeply integrates local detail information and global visual information, thereby effectively refining the information in each receptive field. The model can accurately identify motion patterns in most unstable videos and generate pixel-by-pixel stable distortion maps, thereby generating accurate stable frame sequences. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the video image processing method according to the present disclosure;

[0015] Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the video image processing apparatus according to the present disclosure;

[0016] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure;

[0017] Figure 4 This is an exemplary grid point variation diagram in the video image processing method according to this disclosure;

[0018] Figure 5These are some structural flowcharts of the video stable image recognition model in the video image processing method of this disclosure;

[0019] Figure 6 This is a single-layer structure diagram of the TUNet generation network in the video image processing method of this disclosure;

[0020] Figure 7 This is a structural flowchart of HAFM in the video image processing method according to this disclosure;

[0021] Figure 8 This is a structural flowchart of the discriminator in the video image processing method according to the present disclosure;

[0022] Figure 9 This is a flowchart of the self-attention score calculation in the video image processing method of this disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1This is a flowchart of some embodiments of the video image processing method according to the present disclosure. A flowchart 100 of some embodiments of the video image processing method according to the present disclosure is shown. The video image processing method includes the following steps:

[0030] Step 101: Adjust the initial video data into a stable frame sequence and an unstable frame sequence.

[0031] In some embodiments, the execution entity of the video image processing method (e.g., a computing device) can adjust the initial video data into a stable frame sequence and an unstable frame sequence. The stable frames in the stable frame sequence correspond to the unstable frames in the unstable frame sequence. The initial video data can refer to the video to be processed. For example, video editing software can be used to perform video stabilization processing on each frame of the initial video data to obtain a stable frame sequence. Alternatively, video editing software can be used to perform video jitter processing on each frame of the initial video data to obtain an unstable frame sequence.

[0032] As an example, video data frames can be adjusted into a stable frame sequence. and unstable frame sequences Where n is the number of frames in the initial video data. For subsequent training and computation, the height and width of the video frame can be adjusted to 256, i.e., H = W = 256, C = 3, where H, W, and C represent the height, width, and number of channels of the frame, respectively.

[0033] Step 102: Calculate the set of stable frame feature point coordinates for each stable frame in the stable frame sequence to obtain the sequence of stable frame feature point coordinate sets.

[0034] In some embodiments, the execution entity described above can calculate the set of stable frame feature point coordinates for each stable frame in the stable frame sequence, thereby obtaining a sequence of stable frame feature point coordinate sets. The stable frame feature point coordinates can refer to the SURF (Speed ​​Up Robust Features) feature point coordinates of the stable frame. For example, the set of stable frame feature point coordinates for each stable frame in a stable frame sequence can be calculated by constructing a Hessian matrix or by detecting extrema in scale space.

[0035] Constructing the Hessian matrix: The core of the SURF algorithm is constructing the Hessian matrix. The Hessian matrix is ​​a second-order derivative matrix used to describe the local curvature of an image. The Hessian matrix can be constructed by calculating the second-order partial derivatives of the image. To simplify computation, SURF uses an approximation of the determinant of the Hessian matrix to detect feature points.

[0036] Scale-space extremum detection: After constructing the Hessian matrix, potential feature points are identified by detecting extrema in scale space. These extrema need to be either maximum or minimum at all scales to ensure the scale and rotation invariance of the feature points.

[0037] Step 103: Calculate the unstable frame feature point coordinate set for each unstable frame in the unstable frame sequence to obtain the unstable frame feature point coordinate set sequence.

[0038] In some embodiments, the execution entity described above can calculate the unstable frame feature point coordinate set for each unstable frame in the unstable frame sequence, thereby obtaining a sequence of unstable frame feature point coordinate sets. The unstable frame feature point coordinates can refer to the SURF (Speed ​​Up Robust Features) feature point coordinates of the unstable frames. For example, the unstable frame feature point coordinate set for each unstable frame in the unstable frame sequence can be calculated by constructing a Hessian matrix or by using extremum detection in scale space.

[0039] Step 104: Construct a video frame image training sample set based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence.

[0040] In some embodiments, the aforementioned execution entity may construct a video frame image training sample set based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence.

[0041] In practice, the aforementioned implementing entity can construct a training sample set of video frame images through the following steps:

[0042] The first step is to perform the following processing steps for each stable frame in the stable frame sequence:

[0043] 1. Determine the stable frame feature point coordinate set of the stable frame. The stable frame feature point coordinate set can be P: {(x1, y1), (x2, y2), ..., (x...} n y n P can represent the set of feature point coordinates of a stable frame.

[0044] 2. Determine the set of stable frame feature point coordinates for the next stable frame, as the set of adjacent stable frame feature point coordinates. The set of adjacent stable frame feature point coordinates can be P′: {(x′1, y′1), (x′2, y′2), ..., (x′...} n y′ n P′ can represent the set of feature point coordinates for an unstable frame.

[0045] 3. Normalize the coordinates of each stable frame feature point in the set of stable frame feature point coordinates to obtain a normalized set of stable frame feature point coordinates. For example, the coordinates of each stable frame feature point are normalized to [-1, 1].

[0046] 4. Normalize the coordinates of each adjacent stable frame feature point in the adjacent stable frame feature point coordinate set to obtain a normalized adjacent stable frame feature point coordinate set. For example, the coordinates of each adjacent stable frame feature point are normalized to [-1, 1].

[0047] 5. Based on the normalized stable frame feature point coordinate set and the normalized adjacent stable frame feature point coordinate set, construct the stable frame affine matrix between the stable frame and the next stable frame.

[0048] For example, constructing a stable frame affine matrix between the stable frame and the next stable frame.

[0049]

[0050] A can refer to the affine matrix of the stable frame.

[0051] 6. Based on the normalized stable frame feature point coordinate set, construct the stable frame feature point coordinate vector of the stable frame.

[0052] For example, the coordinate vector of feature points in a stable frame can be:

[0053] B = [x1, x2, ..., x] n y1, y2, ..., y n ] T ,

[0054] Where B represents the coordinate vector of the feature points in the stable frame, and T represents the matrix transpose.

[0055] In addition, to obtain the affine vector X, a linear equation can be constructed: A*X=B.

[0056] Where X = [a1, a2, a3, a4, a5, a6] T .

[0057] The second step involves performing the following processing steps for each unstable frame in the unstable frame sequence:

[0058] 1. Determine the set of unstable frame feature point coordinates for the unstable frame.

[0059] 2. Determine the set of unstable frame feature point coordinates for the next unstable frame of the unstable frame, and use it as the set of adjacent unstable frame feature point coordinates.

[0060] 3. Normalize the coordinates of each unstable frame feature point in the unstable frame feature point coordinate set to obtain a normalized unstable frame feature point coordinate set.

[0061] 4. Normalize the coordinates of each adjacent unstable frame feature point in the adjacent unstable frame feature point coordinate set to obtain a normalized adjacent unstable frame feature point coordinate set.

[0062] 5. Based on the normalized unstable frame feature point coordinate set and the normalized adjacent unstable frame feature point coordinate set, construct the unstable frame affine matrix between the unstable frame and the next unstable frame.

[0063] 6. Based on the normalized unstable frame feature point coordinate set, construct the unstable frame feature point coordinate vector of the unstable frame.

[0064] The handling of unstable frames is the same as that for stable frames, and will not be repeated here.

[0065] Third, for each stable frame in the stable frame sequence, perform the following processing steps:

[0066] 1. The unstable frame corresponding to the stable frame is identified as the associated unstable frame.

[0067] 2. Calculate the association affine matrix between the stable frame and the associated unstable frame. For example, the association affine matrix can be calculated using... This is indicated by the diagram. The construction of the associated affine matrix can be found in the section on the construction of the affine matrix for stable frames, and will not be repeated here.

[0068] 3. Convert the associated affine matrix into the corresponding video frame distortion diagram. The video frame distortion diagram can be generated using... It indicates. Among them, That is, the transformed X.

[0069] 4. The stable frame, the associated unstable frame, the video frame distortion map, the stable frame feature point coordinate set corresponding to the stable frame, the adjacent stable frame feature point coordinate set, the stable frame affine matrix, the stable frame feature point coordinate vector, and the unstable frame feature point coordinate set corresponding to the associated unstable frame, the adjacent unstable frame feature point coordinate set, the unstable frame affine matrix, and the unstable frame feature point coordinate vector are determined as video frame image training samples.

[0070] Step 105: Train the initial video stable image recognition model based on the video frame image training sample set to obtain the trained video stable image recognition model.

[0071] In some embodiments, the aforementioned execution entity can train an initial video-stabilized image recognition model based on the video frame image training sample set to obtain a trained video-stabilized image recognition model. The initial video-stabilized image recognition model may be a neural network model including an initial convolutional neural network layer, an initial attention mechanism layer, and a hierarchical adaptive fusion module.

[0072] It should be noted that during the training process, each sampled frame (video frame image training sample) and 15 consecutive frames before and after it (video frame image training samples) are used for training, that is, θ = 15 in the algorithm. Grid represents the predicted video frame distortion map, A′ represents the video frame distortion map predicted without using the grid, and U′ and P represent the frames obtained by distortion deformation through Grid and A′. and These represent matrices consisting entirely of -1 and 1, respectively.

[0073] In practice, the aforementioned implementing entity can train the initial video-stabilized image recognition model through the following steps:

[0074] The first step is to select a target video frame image training sample from the video frame image training sample set. A video frame image training sample can be randomly selected from the video frame image training sample set as the target video frame image training sample.

[0075] The second step involves inputting the target video frame image training samples into the initial video stabilized image recognition model to obtain the initial video stabilized image recognition results. These initial video stabilized image recognition results include: initial video stabilized image content recognition results, initial video stabilized image shape recognition results, and initial video stabilized image frame time recognition results. The initial video stabilized image content recognition results can refer to the recognition results of the content represented by the video stabilized image in the target video frame image training samples. The initial video stabilized image shape recognition results can refer to the recognition results of the shape of the video stabilized image in the target video frame image training samples. The initial video stabilized image frame time recognition results can refer to the recognition results of the time of the video stabilized image in the target video frame image training samples.

[0076] As an example, the training process of the initial video-stabilized image recognition model is illustrated in the following figure:

[0077]

[0078] The third step is to determine the loss information between the initial stable video image recognition result and the corresponding sample label based on the preset loss function.

[0079] The third step mentioned above may include the following sub-steps:

[0080] The first sub-step involves determining the image content loss information between the initial video-stabilized image recognition result and the corresponding sample labels, based on the content loss function included in the loss function. The sample labels may include: video-stabilized image content information labels, video-stabilized image shape labels, and video-stabilized image frame time labels.

[0081] For example, the content loss function could be:

[0082] L content =∑ i∈[t，t+1] MSE(S i -P i )+L VGG (S i P i ),

[0083] Where S represents the basic facts of the stable frame (video-stabilized image content information label), P is the predicted stable frame (initial video-stabilized image recognition result), and MSE is the mean squared error function. This paper uses the MSE loss function to penalize the deviation between the predicted and the true values. A pre-trained VGG algorithm is used. 16 L is calculated using the max pooling layer before the fully connected layer of the network. VGG (S i P i The mean squared loss in the feature space is as follows:

[0084] L VGG (S i P i =MSE(VGG) 16 (S i -VGG 16 (P i )).

[0085] The second sub-step involves determining the image shape loss information between the initial video stable image recognition result and the corresponding sample label, based on the image shape loss function included in the loss function.

[0086] For example, the image shape loss function could be:

[0087] L shape =L point (γ, S) i P i )+L grid (S i P i ).

[0088] Among them, L point (γ, S) i P i) Calculate the loss between feature points in two frames to penalize changes in some aberrational feature points after distortion. L grid (S i P i The mesh loss is represented as follows:

[0089] L grid (S i P i ) = L relative (S i P i )+L adjacent (S i P i ).

[0090] Furthermore, the sparse mesh vector difference method between adjacent points was used to handle shape changes during training, and the feasibility of this method was further verified in subsequent experiments. Figure 4 As shown.

[0091] Essentially, this is a mathematical model that quantifies the differences in sparse points between adjacent frames. It calculates the vector difference between four neighboring points located at a specific midpoint within the grid and aggregates them individually. relative (S i P i This quantifies the changes in the relationship between vector differences between adjacent points with relative positions, such as... Figure 4 As shown in (a). Their vector summation is calculated as follows:

[0092]

[0093] Where δ is the set of points within the grid in the frame, and n is the number of points within the grid. and Located at point v i The vertical position, or point v i The left and right positions of v i Non-adjacent adjacent grid points. The geometric interpretation of the result is the length of vector 1 in the graph. L adjacent (S i P i Use as Figure 4 (b) uses the dot product of the vectors shown to evaluate the change between adjacent points, calculated as follows:

[0094]

[0095] in, and It's point v i The adjacent points are adjacent in position. The geometric meaning of the result is n in the graph, that is, the size of the vector projection.

[0096] The third sub-step involves determining the image frame time loss information between the initial video stable image recognition result and the corresponding sample label based on the image frame time loss function included in the loss function.

[0097] For example, the image frame time loss function could be:

[0098]

[0099] in, It is to predict the frame P i-1 Twisted into P i The function.

[0100] The fourth sub-step involves merging the image content loss information, the image shape loss information, and the image frame time loss information into a single loss information.

[0101] Fourth, in response to determining that the loss information satisfies the convergence condition, the initial video-stabilized image recognition model is determined as the trained video-stabilized image recognition model. The convergence condition may refer to the fact that the loss values ​​represented by the image content loss information, the image shape loss information, and the image frame time loss information are all less than the corresponding loss thresholds.

[0102] Optionally, the model structure of the initial video-stabilized image recognition model is determined.

[0103] In some embodiments, the execution entity described above may determine the model structure of the initial video-stabilized image recognition model through the following steps:

[0104] The first step is to determine the initial convolutional neural network layer and the initial attention mechanism layer of the initial video-stabilized image recognition model.

[0105] The second step is to determine the hierarchical adaptive fusion module that connects the initial convolutional neural network layer and the initial attention mechanism layer.

[0106] The structure of the initial video-stabilized image recognition model is explained as follows: Figure 5 The example is described as follows:

[0107] (1) A new framework, TranStableNet, is proposed, which combines Transformer and CNN branches and achieves a balance between processing speed and stability performance.

[0108] (2) Tranformer-UNet (TUNet) was proposed, which can generate warp maps that encapsulate global context and local feature information. The warp maps contain various semantic information of the image to achieve more robust and stable geometric transformations, thereby guiding the reconstruction of video stabilizers.

[0109] (3) A Hierarchical Adaptive Fusion Module (HAFM) is proposed, which improves the quality of distortion maps by integrating attentional visual features and multi-level visual features in a multi-scale adaptive fusion manner.

[0110] (4) This paper designs a discriminator based on SwinTransformerV2, STSD (SwinTransformerStable Discriminator), which employs a strategy of pixel-by-pixel rendering and high-quality image reconstruction. STSD serves as a key component for mitigating black borders and achieving a complete field of view (FOV). The framework achieves state-of-the-art performance on several benchmark datasets. Extensive experimental results demonstrate the effectiveness of the proposed method.

[0111] To address the aforementioned challenges, this paper analyzes the motion differences between stable and unstable frame sequences. Centered on the idea of ​​multi-level feature interaction, it explores the collaborative modeling of high-quality inter-frame motion features using CNN and Transformer. Furthermore, this paper deeply integrates high-level semantic and low-level visual detail information to obtain more refined distortion features. Specifically, the overall framework of the pixel-level video stabilization algorithm TranStableNet, based on a dual-branch CNN and Transformer architecture, is as follows: Figure 5 As shown, the model first extracts motion features from two adjacent unstable frame sequences using two shared-parameter TUNet (Tranformer UNet) networks. Each network generates distortion maps in both the horizontal and vertical directions. Then, STSD (Swin Transformer Stable Discriminator) is used to distinguish the estimated distortion maps from the corresponding stable frames of the center frame and the estimated stable frames.

[0112] This application provides an unstable video and its corresponding stable video as video pairs. It takes unstable frame sequences centered on unstable adjacent frames Ut and Ut+1, namely the sequences [Ut-Δt, Ut+Δt] and [Ut-Δt+1, Ut+Δt+1]. The model first processes the two sequences separately using two TUNet (Transformer UNet) networks with shared parameters to obtain two adjacent distortion maps Wt and wt+1. The TUNet uses a dual-branch CNN and Transformer architecture to learn the mapping relationship from the unstable video to the stable video, generating pixel-by-pixel distortion maps for more refined processing of the video content. Then, a discriminator based on the Swing Transformer is used to distinguish between the stable frames obtained from the distortion maps and the true stable frames. Multiple iterations are performed to enhance the model's generation and discrimination capabilities. The proposed method constructs a dual-branch network to better learn information in the video and achieves more accurate stable frame prediction through the discriminator. In subsequent inference processes, the model only uses TUNet for image stabilization.

[0113] Specifically, the single-layer structure of the generative network TUNet is as follows: Figure 6 As shown, a dual-branch network using UNet and Transformer, along with a hierarchical adaptive fusion module (HAFM) connecting the two branches, is employed. The CNN branch (initial convolutional neural network layer) and the Transformer branch (initial attention mechanism layer) each consist of 7 3×3 convolutional blocks, 7 deconvolutional blocks, and 8 Transformer blocks with identical structures. The Transformer branches between layers are isolated from each other. Within the convolutional branches, corresponding convolutional and deconvolutional blocks are designed to share tensor information downwards.

[0114] The generator is described below:

[0115] (1) Transformer Branch: This paper first processes the input frame sequence to an appropriate size, and then embeds it into the size required by the Transformer branch. The Transformer branch adopts a 12-head multi-head attention mechanism, and the rest of the part is consistent with the original Transformer encoder part, that is, it includes a multi-head attention mechanism layer, a feedforward neural network layer, and a residual normalization process. In the middle part of the corresponding CNN branch, as the depth of the downsampling part increases, the tensor size of the convolutional features in the middle part is insufficient to support the sampling process of patch embedding. Therefore, HAFM is no longer used for feature coupling in this part.

[0116] (2) CNN Branches: CNN branches adopt the UNet structure, such as... Figure 6As shown, the resolution of the feature maps decreases with network depth, while the number of channels increases. The left side represents the feature extraction network (encoder), and the right side represents the feature fusion network (decoder). The encoder captures the contextual information of the image, while the decoder recovers details and generates predictions at the same resolution as the input image, enabling simultaneous modeling of local details and global context.

[0117] During downsampling, a 3×3 spatial convolution and a residual connection between the input and output are used. Then, HAFM is used to couple features with the Transformer branch. Finally, batch normalization and ReLU activation are applied. At the end of downsampling, the intermediate results are passed through a fully connected layer to obtain the first output A^ of the convolution branch.

[0118] In the upsampling, a 4×4 deconvolution was used. Similarly, HAFM was used to couple features with the Transformer branch. Then, skip connection layers were used to recover the spatial information that might be lost in the decoder stage. The second output W^ was obtained at the end of the upsampling. After that, the output T^ of the Transformer branch was unified to the shape of the distorted image and then added to these two outputs to obtain the result W required by the generator network.

[0119] (3) HAFM: A key aspect of the TUNet (Transformer UNet) architecture is how to couple the convolutional features from the CNN branches with the embedding features from the Transformer. To address this crucial issue, this paper designs a Hierarchical Adaptive Fusion Module (HAFM) to fuse the convolutional features from the CNN branches with the embedding features from the Transformer. The structure of the HAFM designed in this paper is as follows: Figure 7 As shown, HAFM first uses convolutional modules along the branches to progressively integrate feature maps and patch embeddings in an interactive manner. To maintain a consistent embedding size, an asymmetric sampling stride strategy is employed during the interaction.

[0120] Specifically, since the convolutional feature dimension is C×H×W (C, H, and W represent the number of channels, height, and width of the tensor, respectively), while the embedding shape is (T+1)×E (T and E represent the number of image embeddings and the embedding dimension, respectively), it involves the mutual conversion between two-dimensional and three-dimensional features. Because the dimension of the embedding features remains constant, while the convolutional features change continuously during upsampling and downsampling, direct connection methods become ineffective.

[0121] To achieve this, HAFM uses upsampling and downsampling to establish the connection between them. During feature transformation, different strides are used for upsampling and downsampling, controlling the convolution and deconvolution at each step to maintain a consistent feature size through different strides. In terms of spatial dimensions, HAFM employs different embedding sizes to ensure that the embedding dimension is the same after each sampling.

[0122] The discriminator is described below:

[0123] After the first 20 iterations of training, TUNet can generate relatively stable frame sequences, but abnormal distortions occur in local details and the proportions of objects and people within the image. Therefore, a discriminator, STSD (Swin Transformer Stable Discriminator), was designed for pixel-by-pixel rendering and high-quality image reconstruction. Compared to traditional convolutional layers, Transformer blocks have better performance in existing GAN models. Since incorporating the entire Swing Transformer has high computational complexity and resource consumption, the discriminator designed in this paper has the following structure diagram: Figure 8 As shown.

[0124] Two Swin Transformer V2 blocks are used in the discriminator for feature extraction. In traditional self-attention score calculations in visual models, the similarity term for pixel pairs is calculated as the dot product of the query and key vectors. The learned attention maps for some blocks and heads are often dominated by a few pixel pairs. The method used by Swin Transformer V2 has been shown to alleviate this phenomenon. The Swin Transformer V2 blocks differ from traditional self-attention score calculation methods in that… Figure 9 As shown, it introduces the scaled cosine attention method to handle this phenomenon, as follows:

[0125] Attention(Q,K,V)=softmax(Sim(Q,K))·V.

[0126] Where Sim(Q, K) is the scaling cosine similarity calculation function, as shown below:

[0127] Sim(Q,K)=cosine(Q·K) / τ+B.

[0128] Here, cosine(Q·K) calculates the cosine similarity between the query vector Q and the key vector K, and τ is a learnable scalar used to adjust the magnitude of the similarity. This is the relative positional deviation term for each head.

[0129] Step 106: In response to receiving the video to be detected, perform video stabilization image recognition processing on each video frame in the video to be detected according to the video stabilization image recognition model to obtain a video stabilization image recognition result sequence.

[0130] In some embodiments, the execution entity may, in response to receiving a video to be detected, perform video stabilization image recognition processing on each video frame in the video to be detected according to the video stabilization image recognition model, to obtain a sequence of video stabilization image recognition results. Each video frame corresponds to one video stabilization image recognition result. The video to be detected can refer to the video to be detected.

[0131] For example, individual video frames from the video to be detected can be input into a video stabilization image recognition model to obtain a sequence of video stabilization image recognition results. The video stabilization image recognition results can indicate whether the corresponding video frame is a stable frame.

[0132] Step 107: Based on the video stable image recognition result sequence, perform video stabilization processing on the video to be detected to obtain an optimized video.

[0133] In some embodiments, the aforementioned execution entity can perform video stabilization processing on the video to be detected based on the video stabilization image recognition result sequence to obtain an optimized video. The optimized video can be a video composed of each video frame in the video to be detected adjusted into a series of stable frames.

[0134] For example, the reasoning process of the image stabilization algorithm is as follows:

[0135] The aforementioned execution entity can scale the frame sequence to 256*256 pixels after obtaining the frame height (H), width (W), and frame number (n) according to the network's needs. Since this paper requires image stabilization processing by combining information from preceding and following frames, the unstable frame sequence is first divided into three parts: the first 15 frames, the middle part, and the last 15 frames. For ease of processing, a sliding window list of length 31 is introduced. The generator network TUNet (Transformer UNet) is used to constrain the unstable frame sequence input to the network. During window initialization, the first 15 frames are initialized as frame 1, and the last 16 frames are the first 16 frames.

[0136]

[0137] During each processing iteration, the middle frame of the window becomes the current frame. Subsequent processing discards the first frame in the window and adds the 15th frame after the current frame. If the current frame is one of the last 15 frames, then the last frame is added. Each processed frame is then cropped and scaled back to its original size and added to the stable frame sequence as a stable frame.

[0138] Specifically, the video stabilization process can be illustrated as follows:

[0139]

[0140] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a video image processing apparatus, which are similar to... Figure 1 Corresponding to the method embodiments shown, this video image processing apparatus can be specifically applied to various electronic devices.

[0141] like Figure 2 As shown, a video image processing apparatus 200 in some embodiments includes: an adjustment unit 201, a first calculation unit 202, a second calculation unit 203, a construction unit 204, a training unit 205, a recognition unit 206, and an image stabilization unit 207. The adjustment unit 201 is configured to adjust initial video data into a stable frame sequence and an unstable frame sequence, wherein stable frames in the stable frame sequence correspond to unstable frames in the unstable frame sequence; the first calculation unit 202 is configured to calculate the stable frame feature point coordinate set of each stable frame in the stable frame sequence, obtaining a stable frame feature point coordinate set sequence; the second calculation unit 203 is configured to calculate the unstable frame feature point coordinate set of each unstable frame in the unstable frame sequence, obtaining an unstable frame feature point coordinate set sequence; the construction unit 204 is configured to construct a video frame image training unit based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence. Training sample set; training unit 205 is configured to train the initial video stable image recognition model based on the training sample set of the video frame images to obtain the trained video stable image recognition model; recognition unit 206 is configured to, in response to receiving a video to be detected, perform video stable image recognition processing on each video frame in the video to be detected according to the video stable image recognition model to obtain a video stable image recognition result sequence, wherein one video frame corresponds to one video stable image recognition result; image stabilization unit 207 is configured to perform video image stabilization processing on the video to be detected according to the video stable image recognition result sequence to obtain an optimized video.

[0142] It is understandable that the units described in the video image processing apparatus 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the video image processing apparatus 200 and the units contained therein, and will not be repeated here.

[0143] The following is for reference. Figure 3This document illustrates a schematic diagram of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0144] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0145] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0146] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0147] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0148] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0149] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: adjust initial video data into a stable frame sequence and an unstable frame sequence, wherein stable frames in the stable frame sequence correspond to unstable frames in the unstable frame sequence; calculate the stable frame feature point coordinate set for each stable frame in the stable frame sequence, obtaining a stable frame feature point coordinate set sequence; calculate the unstable frame feature point coordinate set for each unstable frame in the unstable frame sequence, obtaining an unstable frame feature point coordinate set sequence; construct a video frame image training sample set based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence; train an initial video stable image recognition model based on the video frame image training sample set, obtaining a trained video stable image recognition model; in response to receiving a video to be detected, perform video stable image recognition processing on each video frame in the video to be detected according to the video stable image recognition model, obtaining a video stable image recognition result sequence, wherein one video frame corresponds to one video stable image recognition result; and perform video stabilization processing on the video to be detected based on the video stable image recognition result sequence, obtaining an optimized video.

[0150] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including: an adjustment unit, a first calculation unit, a second calculation unit, a construction unit, a training unit, a recognition unit, and an image stabilization unit. The names of these units do not necessarily limit the specific unit; for example, the construction unit can be described as "a unit that constructs a video frame image training sample set based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence."

[0153] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0154] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A video image processing method, comprising: The initial video data is adjusted into a stable frame sequence and an unstable frame sequence, where the stable frames in the stable frame sequence correspond to the unstable frames in the unstable frame sequence. Calculate the set of stable frame feature point coordinates for each stable frame in the stable frame sequence to obtain the sequence of stable frame feature point coordinate sets; Calculate the unstable frame feature point coordinate set for each unstable frame in the unstable frame sequence to obtain the unstable frame feature point coordinate set sequence; A video frame image training sample set is constructed based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence. Based on the video frame image training sample set, the initial video stable image recognition model is trained to obtain the trained video stable image recognition model. In response to receiving a video to be detected, video stabilization image recognition processing is performed on each video frame in the video to be detected according to the video stabilization image recognition model to obtain a video stabilization image recognition result sequence, wherein one video frame corresponds to one video stabilization image recognition result; Based on the video-stabilized image recognition result sequence, the video to be detected is subjected to video stabilization processing to obtain an optimized video; The video-stabilized image recognition model consists of a generator and a discriminator. The generator includes a Transformer branch, a CNN branch, and a hierarchical adaptive fusion module (HAFM). The CNN branch adopts a UNet structure. The HAFM fuses the convolutional features from the CNN branch with the embedded features from the Transformer branch. The HAFM uses upsampling and downsampling to connect the two branches. During feature transformation, different strides are used for upsampling and downsampling to control the convolution and deconvolution at each step, maintaining a consistent feature size through different strides. In terms of spatial dimension, the HAFM uses different embedding sizes to ensure that the embedding dimension is the same after each sampling. The discriminator uses two Swing Transformer V2 blocks for feature extraction.

2. The method according to claim 1, wherein, The step of constructing a video frame image training sample set based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence includes: For each stable frame in the stable frame sequence, the following processing steps are performed: Determine the set of stable frame feature point coordinates for the stable frame; Determine the set of stable frame feature point coordinates for the next stable frame of the stable frame, and use it as the set of feature point coordinates for adjacent stable frames; The coordinates of each stable frame feature point in the set of stable frame feature points are normalized to obtain a normalized set of stable frame feature point coordinates. The coordinates of each adjacent stable frame feature point in the adjacent stable frame feature point coordinate set are normalized to obtain a normalized adjacent stable frame feature point coordinate set. Based on the normalized stable frame feature point coordinate set and the normalized adjacent stable frame feature point coordinate set, construct the stable frame affine matrix between the stable frame and the next stable frame; Based on the normalized stable frame feature point coordinate set, construct the stable frame feature point coordinate vector of the stable frame; For each unstable frame in the unstable frame sequence, perform the following processing steps: Determine the set of unstable frame feature point coordinates for the unstable frame; Determine the set of unstable frame feature point coordinates for the next unstable frame, and use it as the set of adjacent unstable frame feature point coordinates. The coordinates of each unstable frame feature point in the unstable frame feature point coordinate set are normalized to obtain a normalized unstable frame feature point coordinate set. The coordinates of each adjacent unstable frame feature point in the adjacent unstable frame feature point coordinate set are normalized to obtain a normalized adjacent unstable frame feature point coordinate set. Based on the normalized unstable frame feature point coordinate set and the normalized adjacent unstable frame feature point coordinate set, construct the unstable frame affine matrix between the unstable frame and the next unstable frame; Based on the normalized unstable frame feature point coordinate set, construct the unstable frame feature point coordinate vector of the unstable frame.

3. The method according to claim 2, wherein, The step of constructing a video frame image training sample set based on the stable frame feature point coordinate set sequence and the unstable frame feature point coordinate set sequence further includes: For each stable frame in the stable frame sequence, the following processing steps are performed: The unstable frame corresponding to the stable frame is identified as the associated unstable frame; Calculate the association affine matrix between the stable frame and the associated unstable frame; The associated affine matrix is ​​converted into the corresponding video frame distortion map; The stable frame, the associated unstable frame, the video frame distortion map, the stable frame feature point coordinate set corresponding to the stable frame, the adjacent stable frame feature point coordinate set, the stable frame affine matrix, the stable frame feature point coordinate vector, and the unstable frame feature point coordinate set corresponding to the associated unstable frame, the adjacent unstable frame feature point coordinate set, the unstable frame affine matrix, and the unstable frame feature point coordinate vector are determined as video frame image training samples.

4. The method according to claim 1, wherein, The step of training the initial video-stabilized image recognition model based on the video frame image training sample set to obtain the trained video-stabilized image recognition model includes: Select target video frame image training samples from the video frame image training sample set; The target video frame image training samples are input into the initial video stable image recognition model to obtain the initial video stable image recognition result, wherein the initial video stable image recognition result includes: initial video stable image content recognition result, initial video stable image shape recognition result, and initial video stable image frame time recognition result; Based on a preset loss function, the loss information between the initial video stable image recognition result and the corresponding sample label is determined; In response to determining that the loss information satisfies the convergence condition, the initial video-stabilized image recognition model is determined as the trained video-stabilized image recognition model.

5. The method according to claim 4, wherein, The loss information determined based on a preset loss function between the initial video-stabilized image recognition result and the corresponding sample label includes: Based on the content loss function included in the loss function, the image content loss information between the initial video stable image recognition result and the corresponding sample label is determined; Based on the image shape loss function included in the loss function, the image shape loss information between the initial video stable image recognition result and the corresponding sample label is determined; Based on the image frame time loss function included in the loss function, the image frame time loss information between the initial video stable image recognition result and the corresponding sample label is determined; The image content loss information, the image shape loss information, and the image frame time loss information are combined into a single loss information.

6. A video image processing apparatus, comprising: The adjustment unit is configured to adjust the initial video data into a stable frame sequence and an unstable frame sequence, wherein the stable frames in the stable frame sequence correspond to the unstable frames in the unstable frame sequence. The first computing unit is configured to calculate the set of stable frame feature point coordinates for each stable frame in the stable frame sequence, thereby obtaining a sequence of stable frame feature point coordinate sets. The second calculation unit is configured to calculate the unstable frame feature point coordinate set for each unstable frame in the unstable frame sequence, and obtain the unstable frame feature point coordinate set sequence. The construction unit is configured to construct a video frame image training sample set based on the sequence of stable frame feature point coordinates and the sequence of unstable frame feature point coordinates. The training unit is configured to train the initial video stable image recognition model based on the training sample set of the video frame images, so as to obtain the trained video stable image recognition model. The recognition unit is configured to, in response to receiving a video to be detected, perform video stabilization image recognition processing on each video frame in the video to be detected according to the video stabilization image recognition model, and obtain a video stabilization image recognition result sequence, wherein one video frame corresponds to one video stabilization image recognition result; The image stabilization unit is configured to perform video stabilization processing on the video to be detected based on the video stabilization image recognition result sequence to obtain an optimized video; The video-stabilized image recognition model consists of a generator and a discriminator. The generator includes a Transformer branch, a CNN branch, and a hierarchical adaptive fusion module (HAFM). The CNN branch adopts a UNet structure. The HAFM fuses the convolutional features from the CNN branch with the embedded features from the Transformer branch. The HAFM uses upsampling and downsampling to connect the two branches. During feature transformation, different strides are used for upsampling and downsampling to control the convolution and deconvolution at each step, maintaining a consistent feature size through different strides. In terms of spatial dimension, the HAFM uses different embedding sizes to ensure that the embedding dimension is the same after each sampling. The discriminator uses two Swing Transformer V2 blocks for feature extraction.

7. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images