A method for generating a video, a terminal device, and a storage medium

Abstract

The application is suitable for the technical field of video generation, and provides a video generation method, a terminal device and a storage medium. The method comprises the following steps: after obtaining a latent variable tensor, performing global semantic extraction on the latent variable tensor to obtain an intermediate feature map; then performing full-map decoding processing on the intermediate feature map to obtain a basic video; and finally obtaining a final target video by extracting a region of interest in the basic video and correcting the region of interest. After obtaining the basic video, the application does not directly output the basic video, but corrects the region of interest in the basic video, so that the quality of the final target video is higher and the details are more perfect.

Description

Technical Field

[0001] This application belongs to the field of video generation technology, and in particular relates to a video generation method, terminal device and storage medium. Background Technology

[0002] With the rapid development of AI-powered video generation technology, methods for generating videos using trained models have been gradually applied to content creation, film and television production, virtual reality, and intelligent interaction. In a typical video generation chain, an encoder first extracts and compresses features to obtain latent variables; then, a decoder decodes these latent variables into video frames, thus obtaining the final video. Ensuring the quality of the generated video is a problem that needs to be solved. Summary of the Invention

[0003] This application provides a video generation method, terminal device, and storage medium, which can improve the quality of the generated video.

[0004] In a first aspect, embodiments of this application provide a method for generating a video, including: After receiving the latent variable tensor to be processed, global semantic extraction is performed on the latent variable tensor to obtain an intermediate feature map. The resolution of the intermediate feature map is greater than the resolution of the latent variable tensor but less than the target resolution. The latent variable tensor is the data after encoding the input data. The intermediate feature map is subjected to full-image decoding to obtain a base video, wherein the resolution of the base video is the target resolution; Based on the image features of each frame in the base video, the region of interest in each frame and the correction amount of the region of interest are determined, wherein the region of interest is the region that needs to be corrected; The region of interest in each frame of the image is corrected using the correction amount to obtain the target video.

[0005] In this application, after obtaining the latent variable tensor, global semantic extraction is first performed on the latent variable tensor to obtain an intermediate feature map; then, full-image decoding is performed on the intermediate feature map to obtain the base video; by extracting the region of interest (ROI) from the base video and refining the ROI, the final target video is obtained. This application does not directly output the base video after obtaining it, but rather refines the ROI within the base video to achieve higher quality and more detailed final target video.

[0006] Secondly, embodiments of this application provide a video generation apparatus, comprising: The feature extraction module is used to perform global semantic extraction on the latent variable tensor after receiving it to obtain an intermediate feature map. The resolution of the intermediate feature map is greater than the resolution of the latent variable tensor but less than the target resolution. The latent variable tensor is the data after encoding the input data. A decoding module is used to perform full-image decoding on the intermediate feature map to obtain a base video, wherein the resolution of the base video is the target resolution; The parameter determination module is used to determine the region of interest and the correction amount of the region of interest in each frame of the base video based on the image features of each frame of the base video, wherein the region of interest is the region that needs to be corrected; The video correction module is used to correct the region of interest in each frame of the image using the correction amount to obtain the target video.

[0007] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video generation method described in any one of the first aspects above.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the video generation method described in any one of the first aspects above.

[0009] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the video generation method described in any one of the first aspects.

[0010] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a video generation method provided in an embodiment of this application; Figure 2This is a flowchart illustrating a method for determining the correction amount of a region of interest according to an embodiment of this application; Figure 3 This is a flowchart illustrating a method for determining pixel residuals in a region of interest according to an embodiment of this application. Figure 4 This is a schematic diagram of the structure of a video generation apparatus provided in one embodiment of this application; Figure 5 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0013] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0014] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0017] A decoder consists of an encoder and a decoder. The encoder encodes the input data to obtain encoded data, and the decoder decodes the encoded data to obtain the desired target data. For a video generation decoder, the encoder generates latent variables, and the decoder decodes these latent variables to output the generated video. For example, a variational autoencoder (VAE) performs representation transformations between latent variable space and pixel / video space. The latent variables are compressed representations within the video generation model.

[0018] In existing technologies, decoders typically employ complex convolutional networks, residual networks, or spatiotemporal upsampling networks to improve high-resolution video reconstruction capabilities. However, these decoders generally suffer from high computational cost, high memory usage, and long inference latency. Especially when generating high-resolution, long-time-series videos, the decoding stage often becomes the main performance bottleneck in the end-to-end video generation chain. If the network complexity of the decoder is directly reduced to improve speed, it can easily lead to problems such as loss of image details, blurred edges, poor texture smoothing, and insufficient recovery of high-frequency information, resulting in poor video generation quality.

[0019] Taking the original VAE decoder as an example, the original VAE decoder consists of a normalization layer, an intermediate layer, a multi-level upsampling layer, and a high-resolution suffix. After the latent variables are input into the original VAE decoder, they are processed by the normalization layer, the intermediate layer, and the multi-level upsampling layer before high-resolution decoding is performed in the high-resolution suffix. When performing high-resolution decoding, the same amount of computation is required for each spatial location on the feature map, which cannot distinguish between background, smooth regions, fast-moving regions, and other detail-sensitive areas, resulting in excessive computational cost for the high-resolution suffix.

[0020] Therefore, in the process of decoding latent variables, how to improve the generation effect of video frames while reducing computing power and reducing problems such as edge blurring, texture loss, motion blur and temporal instability remains an important technical problem that needs to be solved in current video generation technology.

[0021] To address the aforementioned problems, this application proposes a video generation method. This method employs an improved decoder, which includes a base model, a lightweight network model, and a correction model. After decoding the latent variables to obtain high-dimensional features (i.e., intermediate feature maps), the base model does not directly perform high-resolution suffix processing on these features. Instead, it utilizes the lightweight network model to perform global decoding on the high-dimensional features, obtaining a base video. Then, the correction model refines the regions of interest (ROIs) within the base video, ultimately yielding the target video. The lightweight network model is simpler in structure than the high-resolution suffix model. The ROI refers to the image region requiring further refinement.

[0022] In one approach, the base model can be a prefix module in the original VAE decoder, which includes a normalization layer, an intermediate layer, and a multi-level upsampling layer.

[0023] The following is combined with Figure 1 The video generation method according to embodiments of this application will be described in detail. This method can be applied to a terminal device. The improved decoder described above exists in the terminal device.

[0024] Figure 1 A schematic flowchart illustrating the video generation method provided in this application is shown, with reference to... Figure 1 The method is described in detail below: S101, after receiving the latent variable tensor to be processed, global semantic extraction is performed on the latent variable tensor to obtain an intermediate feature map, wherein the resolution of the intermediate feature map is greater than the resolution of the latent variable tensor and less than the target resolution, and the latent variable tensor is the data after encoding the input data.

[0025] In this embodiment, the input data can be text data or voice data, etc.

[0026] Specifically, one way to obtain latent variable tensors includes: Obtain text information, which is the input data; extract key information from the text information to obtain key information of the text information, which represents the features of the video to be generated; encode the key information to obtain the latent variable tensor.

[0027] In this embodiment, a text encoder is used to encode key information to obtain a latent variable tensor.

[0028] For example, if a user inputs the text "generate a video of a bird eating insects", then the input data is "generate a video of a bird eating insects". Key information includes "bird, eating insects, etc." An encoder is used to encode this key information to obtain a latent variable tensor. The latent variable tensor can be represented as... Where B is the batch size, representing the amount of data processed in parallel during a single inference. This represents the number of channels. It is the time series length of the input latent variables. The height is a latent variable and has a proportional relationship with the height of the output video. The width is a latent variable and has a proportional relationship with the width of the output video.

[0029] In this embodiment, the target resolution is the resolution of the final generated video.

[0030] In this embodiment, after receiving the latent variable tensor to be processed, the temporal dimension and spatial resolution of the output video are determined. The latent variable tensor is then processed using the base model to obtain an intermediate feature map. This intermediate feature map is a high-dimensional feature map, not the video itself. The temporal dimension determines the number of frames in the video, and the spatial resolution determines the width and height of the images within the video.

[0031] In one approach, the base model can employ a prefix module of a VAE. The base model may include an inverse normalization layer, an initial convolutional layer, intermediate residual blocks, and multi-level upsampling layers; where the initial convolutional layer and intermediate residual blocks are both intermediate layers. The inverse normalization layer is used to restore the standardized latent variables to the true numerical range, obtaining inverse normalized data. The initial convolutional layer is used to extract basic video texture and contour features from the inverse normalized data, obtaining the first feature. The intermediate residual block is used to extract and fuse residual features from the first feature, obtaining the second feature, ensuring that the second feature retains the semantic information and spatiotemporal context required for subsequent basic decoding and local refinement of the region of interest. The multi-level upsampling layer is used to upsample the second feature at least once, obtaining an intermediate feature map. In one approach, the multi-level upsampling layer may include a first sampling layer and a second sampling layer; the first sampling layer is used to perform a first upsampling on the second feature, obtaining a low-resolution feature; the second sampling layer is used to perform a second upsampling on the low-resolution feature, obtaining a medium-resolution feature, which is denoted as the intermediate feature map. Intermediate feature maps can preserve sufficient semantics and spatiotemporal context.

[0032] In this embodiment, global semantic extraction of latent variable tensors includes: half-resolution global semantic extraction of latent variable tensors, where half-resolution refers to half of the target resolution.

[0033] As an example, after a second upsampling through the second sampling layer, the resulting shape is... The intermediate feature map, B is the batch size, representing the amount of data processed in parallel during a single inference, for example, B=2 means processing 2 independent videos simultaneously. C is the number of channels. To output the video time frame dimension, , This is the temporal length of the input latent variable, used to determine the output video duration. For example, It is 21:00 The value is 81. H is the full pixel height of the output video, for example, H is 720. W is the full pixel width of the output video, for example, W can be 1280.

[0034] S102, perform full-image decoding on the intermediate feature map to obtain the base video, wherein the resolution of the base video is the target resolution.

[0035] In this embodiment, the full-image decoding process of the intermediate feature map includes: performing full-image, frame-by-frame, and continuous decoding on the intermediate feature map. The resulting basic video is watchable, but the details are not perfect.

[0036] Specifically, a lightweight network module is used to perform full-image decoding on the intermediate feature map to obtain the basic video. The number of channels in the lightweight network module is less than a preset number of channels. The preset number of channels can be set as needed; for example, it can be 48 channels. Using the lightweight network module to perform full-image decoding on the intermediate feature map reduces computational load and improves data processing efficiency.

[0037] The lightweight network module uses a smaller number of channels, fewer residual blocks (the number of residual blocks is less than a preset value), and / or a lighter convolutional structure (the number of convolutions is less than a preset number). The lightweight network module is more lightweight than the high-resolution suffix in the original VAE. For example, the high-resolution suffix has 96 channels, while the lightweight network module can have 48 channels. The lightweight network module is more efficient in processing intermediate feature maps.

[0038] Because the decoding of the intermediate feature map is a full-image decoding process, that is, each frame is regenerated instead of reusing the content of the previous frame, the basic video covers the entire frame, avoiding problems such as background freezing, brightness banding and temporal discontinuity.

[0039] S103, based on the image features of each frame in the base video, determine the region of interest (ROI) and the correction amount of the ROI in each frame, wherein the ROI is the region that needs to be corrected.

[0040] In this embodiment, the region of interest (ROI) refers to the area requiring additional refinement, that is, a localized area in an image or video that is visually noticeable, has important details, and requires special processing. This contrasts with smooth, low-detail, and unimportant background areas. For example, ROIs may include faces, hands, text, object edges, fast-moving objects, and complex textures. ROIs may include solid-color walls, the sky, blurred static backgrounds, and smooth ground.

[0041] In one implementation, the method for determining the region of interest and the corresponding correction amount may include: Based on the image features of each frame in the base video, each pixel in each frame is scored, and the region of interest is determined according to the scores of each pixel. A correction prediction model is then used to predict the correction amount for the region of interest.

[0042] In another implementation, the method for determining the region of interest and the corresponding correction amount may include: Key points are pre-defined, such as facial key points, hand key points, wing key points, etc. Based on the image features of each frame in the base video, the positions of these key points in each frame are determined. A predetermined number of pixels are then expanded outwards from the key point positions to obtain the region of interest (ROI). Different pixel residuals are pre-defined for different key points; the pixel residuals of the key points corresponding to the ROI are used as the correction amount for the ROI.

[0043] Alternatively, fixed coordinate regions (such as the bottom subtitle area or the center character area) can be pre-configured. The region containing these fixed coordinate regions is then determined in each frame of the base video to obtain the region of interest (ROI). Pre-defined pixel residuals corresponding to different fixed coordinate regions are stored in advance, and the RGB pixel residuals of the fixed coordinate regions corresponding to the ROI serve as the correction amount for the ROI.

[0044] Alternatively, based on the image features of each frame in the base video, moving objects present in the base video are identified. The location region of the moving object in each frame is determined, and this location region is defined as the region of interest. The pixel residuals corresponding to different objects are pre-stored; the pixel residuals of the objects corresponding to the region of interest are used as the correction amount for the region of interest.

[0045] In practical applications, correction amounts can also be luminance correction factors, chromaticity correction factors, etc. Luminance correction factors can include exposure factors, contrast factors, and brightness factors. Chromaticity correction factors can include saturation factors and color cast factors.

[0046] S104, using the correction amount, the region of interest in each frame of the image is corrected to obtain the target video.

[0047] In one approach, the correction amount is superimposed or multiplied with the pixel values ​​of the region of interest to obtain the target video.

[0048] In this application, after obtaining the latent variable tensor, global semantic extraction is first performed on the latent variable tensor to obtain an intermediate feature map; then, full-image decoding is performed on the intermediate feature map to obtain the base video; by extracting the region of interest (ROI) from the base video and refining the ROI, the final target video is obtained. This application does not directly output the base video after obtaining it, but instead refines the ROI within the base video, resulting in a higher quality and more detailed final target video. Furthermore, refining only the ROI avoids the high-cost retouching process covering every spatial location, thus improving data processing speed.

[0049] In one possible implementation, such as Figure 2 As shown, the implementation process of step S103 may include: S1031, based on the image features of each frame, evaluate the distortion potential of each pixel in each frame to obtain a score value for each pixel in each frame.

[0050] In this embodiment, distortion potential can be determined by scoring factors. Scoring factors may include at least one of motion energy, spatial energy, texture complexity, and user-specified focus areas.

[0051] Specifically, runtime energy characterizes the intensity of motion of each pixel in a single frame image. The more intense the motion, the more likely the area is to exhibit blurring, ghosting, and temporal distortion.

[0052] The method for determining motion energy may include: obtaining the pixel value of the j-th pixel in the i-th frame and the pixel value of the j-th pixel in the (i-1)-th frame of the base video, where 1≤i≤N, N is the number of images in the base video, and 1≤j≤M, M is the number of pixels in the images; and determining the motion energy of the j-th pixel in the i-th frame based on the pixel difference between the pixel value of the j-th pixel in the i-th frame and the pixel value of the j-th pixel in the (i-1)-th frame.

[0053] The frames in the base video are sorted chronologically to obtain the sequence number of each frame. This is done using a formula. Determine the motion energy of each pixel in each frame of the image; Let be the motion energy of the j-th pixel in the i-th frame image, and let the spatial coordinates of the j-th pixel be (h, w), where h is the coordinate in the vertical direction and w is the coordinate in the horizontal direction; c is the channel index, which includes the index of the R (red) channel, the index of the G (green) channel and the index of the B (blue) channel. Let be the pixel value of the j-th pixel in the i-th frame of the image in channel c; Let be the pixel value of the j-th pixel in the (i-1)-th frame of the image in the c-channel.

[0054] In practical use, when i=1, let Alternatively, the (i-1)th frame can be replaced with a historical cached frame or a preset motion prior.

[0055] Specifically, spatial energy characterizes the magnitude of pixel variation in a single frame of an image. The method for determining spatial energy may include: obtaining the pixel value of the j-th pixel in the i-th frame of the base video and the pixel value of a target pixel in the i-th frame, wherein the target pixel is a pixel adjacent to the j-th pixel in the i-th frame; and determining the spatial energy of the j-th pixel in the i-th frame based on the pixel difference between the pixel value of the j-th pixel in the i-th frame and the pixel value of the target pixel in the i-th frame.

[0056] Spatial coordinates include vertical coordinates (i.e., ordinate, h) and horizontal coordinates (i.e., abscissa, w). Target pixels include the nth pixel and the mth pixel, wherein the nth pixel is in the same row as the jth pixel and is the right-hand neighbor of the jth pixel; the mth pixel is in the same column as the jth pixel and is the bottom-level neighbor of the jth pixel.

[0057] The spatial energy of the j-th pixel in the i-th frame includes both horizontal and vertical energy.

[0058] Specifically, the method for determining the horizontal energy includes: determining the horizontal energy of the j-th pixel in the i-th frame image based on the pixel difference between the pixel value of the n-th pixel and the pixel value of the j-th pixel in the i-th frame image.

[0059] That is, using the formula Determine the horizontal energy of the j-th pixel in the i-th frame of the image. Let be the horizontal energy of the j-th pixel in the i-th frame of the image; Let be the pixel value of the nth pixel in the i-th frame image, and let the spatial coordinates of the nth pixel be (h, w+1). Let be the pixel value of the j-th pixel in the i-th frame image, and let the spatial coordinates of the j-th pixel be (h, w).

[0060] The method for determining the vertical energy includes: determining the vertical energy of the j-th pixel in the i-th frame image based on the pixel difference between the pixel value of the m-th pixel and the pixel value of the j-th pixel in the i-th frame image.

[0061] That is, using the formula Determine the vertical energy of the j-th pixel in the i-th frame of the image. Let be the vertical energy of the j-th pixel in the i-th frame of the image; Let be the pixel value of the m-th pixel in the i-th frame image, and let the spatial coordinates of the m-th pixel be (h+1, w). Let be the pixel value of the j-th pixel in the i-th frame image, and let the spatial coordinates of the j-th pixel be (h, w).

[0062] Specifically, texture complexity characterizes the richness of texture, edges, and details of pixels in a single frame of an image within space.

[0063] Specifically, the user-specified key area represents the score of different areas set by the user. For example, the scores of key attention areas such as faces, products, subtitles, and hands are preset. The system checks whether the pixels in each frame of the image are within the user-specified key area. If the pixel is within the user-specified key area, the key area score of the pixel is determined according to the score corresponding to the user-specified key area.

[0064] In this embodiment, if there are multiple scoring factors, the scoring factors are weighted and summed to obtain the score value of each pixel in each frame of the image.

[0065] If a scoring factor exists, then the score corresponding to that scoring factor is determined as the score value for each pixel in each frame of the image.

[0066] Specifically, if the scoring factors may include motion energy and / or spatial energy, the method for determining the score value of each pixel based on motion energy and / or spatial energy includes: obtaining the score value of the j-th pixel in the i-th frame image using the motion energy and / or the spatial energy of the j-th pixel. For example, using the formula Determine the score value of the j-th pixel in the i-th frame of the image. Wherein, Let (h, w) be the score value of the j-th pixel in the i-th frame of the image; Let be the motion energy of the j-th pixel in the i-th frame; Let be the horizontal energy of the j-th pixel in the i-th frame of the image; Let be the vertical energy of the j-th pixel in the i-th frame of the image.

[0067] In this embodiment, after obtaining the score value of each pixel in each frame of the image, a comprehensive energy map of each frame of the image is generated based on the score value of each pixel. The score value of each pixel is marked in the comprehensive energy map.

[0068] S1032, Based on the score value of each pixel in each frame of the image, determine the region of interest in each frame of the image.

[0069] In one approach, for a given image frame, the image is divided into patches, and the region of interest is determined based on the score value of the pixels in each patch.

[0070] Specifically, each frame of the base video is divided into grids to obtain the tiles included in each frame; based on the score value of each pixel in the tile, a comprehensive score value for the tile is obtained; based on the magnitude of the comprehensive score value of each tile in each frame, the region of interest in each frame is determined. The comprehensive score value of the tile indicates the necessity of fine-tuning the tile; a higher comprehensive score value indicates that fine-tuning is more necessary.

[0071] All tiles are the same size; for example, each tile has a side length of P×P and a height of P. ; The height of the block, High-quality basic videos Let be the side length of the tile. The width of the tile is... ; The width of the tile. The width of the base video. The total number of tiles is... ; This represents the total number of tiles in a single frame of an image. For example, if H is 720 and P is 64, then... , , .

[0072] Calculate the average score of each pixel in the patch, and record this average as the patch's overall score. For example, using the formula... Calculate the overall score for each map tile. Among them, Q represents the overall score of the k-th patch in each frame of the image; Q is the total number of pixels in the k-th patch. This is the score value of the m-th pixel in the k-th patch.

[0073] Alternatively, the score values ​​of each pixel in the patch can be weighted and summed to obtain the overall score value of the patch.

[0074] Patches with a comprehensive score greater than a preset score are identified as regions of interest (ROIs). Alternatively, the comprehensive score values ​​are sorted in descending order, and the patches corresponding to the top K comprehensive score values ​​are selected as ROIs. K can be set as needed, for example, K can be set to 16, 32, or 64, etc. K is less than the total number of patches in each frame. Each ROI has a region index, which can be the spatial coordinates of the patch. In one implementation, the region index is a tensor index.

[0075] The coverage rate of the area of ​​interest is: ; For the coverage of the region of interest, For the number of regions of interest, This represents the total number of tiles in a single frame of an image.

[0076] In another approach, for a given image frame, pixels with a score greater than a preset value are connected to obtain connected regions, and each connected region is recorded as a region of interest.

[0077] In this application, the selection of regions of interest is related to the video content. The comprehensive score of regions of interest combines motion energy and spatial energy, making it more likely that the main edges, rapidly changing areas, texture areas, and text areas of the basic video can be refined, while smooth background areas reduce additional calculations.

[0078] S1033, using a pixel residual prediction model, predict the pixel residual of the region of interest, and determine the pixel residual as the correction amount of the region of interest.

[0079] In this embodiment, the pixel residual prediction model is a trained model. During training, the predicted pixel residual and the actual residual are obtained, and compared to obtain a comparison result. Based on the comparison result, the parameters in the pixel residual prediction model are corrected to obtain the trained pixel residual prediction model. The actual residual is the difference between the actual pixel value of the region of interest in the high-definition sample video and the pixel value of the region of interest in each frame of the generated base video. The predicted pixel residual is the pixel residual of the region of interest in each frame of the base video predicted by the pixel residual prediction model. For example, the original VAE decoder is used as the teacher model, and the improved decoder in this application is used as the student model. The lightweight network model and the corrected model in the improved decoder are trained to obtain the trained improved decoder. The corrected model contains the pixel residual prediction model. When training the improved decoder, the training loss may include at least one of pixel reconstruction loss, perceptual loss, structural similarity loss, temporal consistency loss, and region of interest weighted loss. Multiple training losses are weighted and summed to obtain the total loss. The region-of-interest (ROI) weighted loss can increase the weight of the refinement region based on the ROI mask, while retaining quality constraints outside the ROI to avoid a significant decrease in background quality.

[0080] The pixel residual prediction model can be a lightweight convolution, a depthwise separable convolution, a residual block, an attention module, or a local network optimized for edge hardware, etc., without any restrictions.

[0081] In one approach, the region of interest is input into a pixel residual prediction model to predict the pixel residual of the region of interest. The pixel residual represents the residual of grayscale values ​​or pixel values.

[0082] In another approach, the pixel residual of the region of interest is determined jointly based on the region of interest and its corresponding extended neighborhood region. Specifically, for example... Figure 3 As shown, the implementation process of step S1033 above may further include: S11, determine the neighborhood extension region of the region of interest from each frame of the base video, wherein the neighborhood extension region is the region surrounding and adjacent to the region of interest.

[0083] In practical use, the neighborhood extension region of the region of interest is determined from each frame of the base video based on the region index of the region of interest; or, the neighborhood extension region of the region of interest is determined from the intermediate feature map based on the region index of the region of interest.

[0084] Specifically, if the size of the region of interest is P×P, the neighborhood expansion region is the area surrounding the region of interest with a width of R. The neighborhood expansion magnification factor is: ;in, This is the neighborhood expansion amplification factor; This is the side length of the tile (i.e., the side length of the region of interest). Expand the width of the neighborhood.

[0085] S12, cropping a target region from the base video, wherein the target region includes the region of interest and the neighborhood extension region of the region of interest.

[0086] In this embodiment, if the size of the region of interest is P×P, and the neighborhood extension region is the area surrounding the region of interest with a size of R, then the size of the target region is (P+2R)×(P+2R). The coverage rate of the target region is: ; The coverage rate of the target area; Coverage of the region of interest; This is the neighborhood expansion amplification factor.

[0087] In another approach, the target region is cropped from the intermediate feature map.

[0088] S13, input the target region into the pixel residual prediction model to determine the pixel residual of the region of interest.

[0089] In this embodiment, the target region is input into the pixel residual prediction model so that the pixel residual prediction model can refer to the neighborhood context features of the region of interest when predicting the pixel residual of the region of interest, making the predicted pixel residual of the region of interest more accurate.

[0090] In practical use, multiple target regions can be input into the pixel residual prediction model at the same time, and the pixel residuals of the regions of interest in each target region can be output, reducing scheduling overhead.

[0091] In one possible implementation, step S104 may include: S1041, for each frame of image, determine whether the m-th region of interest overlaps with other regions of interest, wherein the other regions of interest are regions of interest other than the m-th region of interest.

[0092] As an example, in a basic video, the width of each frame is 100, and the side length of each tile is 30. When dividing a frame into tiles in the horizontal direction, the width of the first, second, and third tiles is 30, and the width of the fourth tile is 10. In order for the fourth tile to form a complete tile, the fourth tile extends 20 to the left. Therefore, there is an overlapping area between the third and fourth tiles.

[0093] S1042, if the m-th region of interest does not overlap with the other regions of interest, then the m-th region of interest is corrected using the correction amount of the m-th region of interest to obtain the corrected m-th region of interest.

[0094] In one approach, the corrected region of interest (ROI) is obtained by multiplying the correction amount by the pixel values ​​of each pixel in the i-th ROI.

[0095] In one approach, if the correction amount is a pixel residual, the pixel residual is superimposed on the pixel values ​​of each pixel in the i-th region of interest to obtain the corrected i-th region of interest.

[0096] In another approach, the first weight and the second weight of the i-th region of interest are obtained. The pixel value of the pixel in the i-th region of interest is multiplied by the first weight to obtain the first candidate value. The pixel residual of the i-th region of interest is multiplied by the second weight to obtain the second candidate value. The first candidate value and the second candidate value are superimposed to obtain the corrected i-th region of interest.

[0097] S1043, if the m-th region of interest overlaps with the other regions of interest, then for the non-overlapping regions in the m-th region of interest, the correction amount of the m-th region of interest is used to correct the non-overlapping regions; For the overlapping regions in the m-th region of interest, the correction amount of the n-th region of interest is obtained, where the n-th region of interest is the region of interest that overlaps with the m-th region of interest. The correction amounts of the m-th and n-th regions of interest are fused to obtain the target correction amount. The overlapping regions in the m-th region of interest are corrected using the target correction amount to obtain the corrected m-th region of interest. The fusion process may include weighted fusion and normalization. After correcting the regions of interest in each frame of the base video, the target video is obtained. The tensor implementation of the fusion process may include scattering accumulation and folding restoration; where scattering accumulation is to accumulate the pixel residuals corresponding to each region of interest and their corresponding weights to the corresponding positions of the base video image; folding restoration is to restore multiple regions of interest to a full image tensor according to their spatial positions, sum the overlapping positions, and then normalize according to the cumulative weights to obtain the full image residual.

[0098] In this embodiment, the specific correction method is described in the above step S1042, and will not be repeated here.

[0099] In another implementation, the pixels in the neighboring extended region of the region of interest are corrected.

[0100] For the neighborhood extension region of the m-th region of interest, obtain the third weight and the fourth weight corresponding to the neighborhood extension region. Multiply the pixel value of the pixel in the neighborhood extension region by the third weight to obtain the third candidate value. Multiply the pixel residual of the m-th region of interest by the fourth weight to obtain the fourth candidate value. Add the third candidate value and the fourth candidate value to obtain the corrected pixel value of the pixel in the neighborhood extension region.

[0101] If the neighboring extended region of the m-th region of interest overlaps with other regions of interest, the pixel values ​​of the pixels in the neighboring extended region and the pixel residuals of the other regions of interest are weighted and summed to obtain the corrected pixel value of the pixels in the neighboring extended region. Alternatively, the pixel values ​​of the pixels in the neighboring extended region, the pixel residuals of the other regions of interest, and the pixel residual of the m-th region of interest are weighted and summed to obtain the corrected pixel value of the pixels in the neighboring extended region.

[0102] If the neighboring extended regions of the m-th region of interest overlap with the neighboring extended regions of other regions of interest, the corrected pixel values ​​of the pixels in the neighboring extended regions are obtained by weighted summation, which will not be elaborated here.

[0103] In one possible implementation, steps S101 to S104 are all implemented in the improved decoder installed on the terminal device.

[0104] The improved decoder needs to be trained before it can be used.

[0105] When training the improved decoder, the input data is fed into the encoder, which encodes the input data to obtain a latent variable tensor. At this point, the input data can be video data.

[0106] The improved decoder is trained using latent variable tensors.

[0107] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0108] Corresponding to the video generation method described in the above embodiments, Figure 4 A structural block diagram of a video generation apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0109] Reference Figure 4 The device 200 may include: a feature extraction module 210, a decoding module 220, a parameter determination module 230, and a video correction module 240.

[0110] The feature extraction module 210 is used to perform global semantic extraction on the latent variable tensor after receiving it to obtain an intermediate feature map. The resolution of the intermediate feature map is greater than the resolution of the latent variable tensor and less than the target resolution. The latent variable tensor is the data after encoding the input data. Decoding module 220 is used to perform full-image decoding processing on the intermediate feature map to obtain a base video, wherein the resolution of the base video is the target resolution; The parameter determination module 230 is used to determine the region of interest and the correction amount of the region of interest in each frame of the base video based on the image features of each frame of the base video, wherein the region of interest is the region that needs to be corrected; The video correction module 240 is used to correct the region of interest in each frame of the image using the correction amount to obtain the target video.

[0111] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0113] This application also provides a terminal device, see [link to relevant documentation] Figure 5 The terminal device 400 may include: at least one processor 410, a memory 420, and a computer program stored in the memory 420 and executable on the at least one processor 410. When the processor 410 executes the computer program, it implements the steps in any of the above method embodiments, for example... Figure 1 Steps S101 to S104 in the illustrated embodiment. Alternatively, when the processor 410 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of the feature extraction module 210 to the video correction module 240 are shown.

[0114] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 420 and executed by processor 410 to complete this application. The one or more modules / units may be a series of computer program segments capable of performing a specific function, which are used to describe the execution process of the computer program in terminal device 400.

[0115] Those skilled in the art will understand that Figure 5 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0116] The processor 410 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0117] The memory 420 can be an internal storage unit of the terminal device or an external storage device, such as a plug-in hard drive, a smart media card (SMC), a secure digital (SD) card, or a flash card. The memory 420 is used to store the computer program and other programs and data required by the terminal device. The memory 420 can also be used to temporarily store data that has been output or will be output.

[0118] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0119] The video generation method provided in this application embodiment can be applied to terminal devices such as computers, tablets, laptops, netbooks, and personal digital assistants (PDAs). This application embodiment does not impose any restrictions on the specific type of terminal device.

[0120] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0122] In the embodiments provided in this application, it should be understood that the disclosed terminal devices, apparatuses, and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by one or more processors, it can implement the steps of the various method embodiments described above.

[0126] Similarly, as a computer program product, when the computer program product is run on a terminal device, it enables the terminal device to implement the steps in the above-described method embodiments.

[0127] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0128] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating video, characterized in that, include: After receiving the latent variable tensor to be processed, global semantic extraction is performed on the latent variable tensor to obtain an intermediate feature map. The resolution of the intermediate feature map is greater than the resolution of the latent variable tensor but less than the target resolution. The latent variable tensor is the data after encoding the input data. The intermediate feature map is subjected to full-image decoding to obtain a base video, wherein the resolution of the base video is the target resolution; Based on the image features of each frame in the base video, the region of interest in each frame and the correction amount of the region of interest are determined, wherein the region of interest is the region that needs to be corrected; The region of interest in each frame of the image is corrected using the correction amount to obtain the target video.

2. The method as described in claim 1, characterized in that, The step of determining the region of interest (ROI) and its correction amount in each frame of the base video based on the image features of each frame includes: Based on the image features of each frame, the distortion potential of each pixel in each frame is evaluated to obtain a score value for each pixel in each frame. Based on the score value of each pixel in each frame of the image, the region of interest in each frame of the image is determined; Using a pixel residual prediction model, the pixel residual of the region of interest is predicted, and the pixel residual is determined as the correction amount of the region of interest.

3. The method as described in claim 2, characterized in that, Determining the region of interest in each frame of an image based on the score value of each pixel in each frame includes: The image of each frame in the base video is divided into grids to obtain the tiles included in each frame of the base video; Based on the score values ​​of each pixel in the image patch, a comprehensive score value for the image patch is obtained; The region of interest in each frame of an image is determined based on the comprehensive score value of each tile in the image.

4. The method as described in claim 2, characterized in that, The process of evaluating the distortion potential of each pixel in each frame based on the image features of each frame to obtain a score value for each pixel in each frame includes: Obtain the pixel value of the j-th pixel in the i-th frame and the pixel value of the j-th pixel in the (i-1)-th frame of the base video, where 1≤i≤N, N is the number of images in the base video, and 1≤j≤M, M is the number of pixels in the image. Based on the pixel difference between the pixel value of the j-th pixel in the i-th frame and the pixel value of the j-th pixel in the (i-1)-th frame, the motion energy of the j-th pixel in the i-th frame is determined; and / or, Obtain the pixel value of the j-th pixel in the i-th frame of the base video and the pixel value of the target pixel in the i-th frame, wherein the target pixel is the pixel in the i-th frame that is adjacent to the j-th pixel; The spatial energy of the j-th pixel in the i-th frame image is determined based on the pixel difference between the pixel value of the j-th pixel in the i-th frame image and the pixel value of the target pixel in the i-th frame image. The score value of the j-th pixel in the i-th frame image is obtained by using the motion energy and / or the spatial energy of the j-th pixel in the i-th frame image.

5. The method as described in claim 4, characterized in that, The target pixels include the nth pixel and the mth pixel, wherein the nth pixel is in the same row as the jth pixel and is the right-side neighbor of the jth pixel; the mth pixel is in the same column as the jth pixel and is the bottom-side neighbor of the jth pixel; Determining the spatial energy of the j-th pixel in the i-th frame image based on the pixel difference between the pixel value of the j-th pixel in the i-th frame image and the pixel value of the target pixel in the i-th frame image includes: The horizontal energy of the j-th pixel in the i-th frame image is determined based on the pixel difference between the pixel value of the n-th pixel and the pixel value of the j-th pixel in the i-th frame image. Based on the pixel difference between the pixel value of the m-th pixel and the pixel value of the j-th pixel in the i-th frame image, the vertical energy of the j-th pixel in the i-th frame image is determined, and the spatial energy includes the horizontal energy and the vertical energy.

6. The method according to any one of claims 2 to 5, characterized in that, The step of using a pixel residual prediction model to predict the pixel residual of the region of interest includes: From each frame of the base video, a neighborhood extension region of the region of interest is determined, wherein the neighborhood extension region is the region surrounding and adjacent to the region of interest; A target region is cropped from the base video, wherein the target region includes the region of interest and the neighborhood extension region of the region of interest; The target region is input into the pixel residual prediction model to determine the pixel residual of the region of interest.

7. The method as described in claim 1, characterized in that, The process of performing full-image decoding on the intermediate feature map to obtain the basic video includes: The intermediate feature map is fully decoded using a lightweight network module to obtain the basic video, wherein the number of channels in the lightweight network module is less than the preset number of channels.

8. The method as described in claim 1, characterized in that, The step of using the correction amount to correct the region of interest in each frame of the image to obtain the target video includes: For each frame of image, determine whether the m-th region of interest overlaps with other regions of interest, wherein the other regions of interest are regions of interest other than the m-th region of interest; If the m-th region of interest does not overlap with the other regions of interest, then the m-th region of interest is corrected using the correction amount of the m-th region of interest to obtain the corrected m-th region of interest; If the m-th region of interest overlaps with other regions of interest, then for the non-overlapping regions within the m-th region of interest, the correction amount of the m-th region of interest is used to correct the non-overlapping regions. For the overlapping regions in the m-th region of interest, obtain the correction amount of the n-th region of interest, where the n-th region of interest is the region of interest that overlaps with the m-th region of interest; The correction values ​​of the m-th region of interest and the n-th region of interest are fused to obtain the target correction value; The overlapping regions in the m-th region of interest are corrected using the target correction amount to obtain the corrected m-th region of interest; After correcting the region of interest in each frame of the base video, the target video is obtained.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video generation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method as described in any one of claims 1 to 8.

Images