Video image quality enhancement method, device, equipment and program product

Abstract

The invention discloses a video image quality enhancement method, device, equipment and program product, and the method comprises the steps: segmenting a to-be-processed video in a space dimension and a time dimension, and forming video sub-blocks; when the video sub-blocks are input into a converter model for super-division processing, offset processing is executed by at least one of a time offset layer and a space offset layer, and super-division video sub-blocks are obtained; and combining the super-divided video sub-blocks to obtain a target video of which the image quality is enhanced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and relate to but are not limited to a method, device, equipment and program product for enhancing video quality. Background Art

[0002] Multi-frame video quality enhancement means restoring a high-quality video given a low-quality video. At present, deep learning-based technologies are dominant in this field. Among them, the Transformer-based model has obvious performance advantages over the previous convolutional neural model-based methods due to its unique modeling capabilities. At the same time, the single-frame method only uses the input of the current frame to reconstruct the picture, while the multi-frame method can use the time information between multiple frames that are adjacent to the current frame but usually inconsistent. Therefore, the Transformer-based multi-frame model is currently the best multi-frame quality optimization solution.

[0003] However, this solution requires huge computing power. On the one hand, the restoration of each high-quality frame requires multiple input frames as input, which requires the model to use a large amount of computing power to process the information contained in these input frames. On the other hand, since Transformer involves more matrix operations, its computing overhead increases significantly as the size of the feature map increases.

[0004] In order to save computing power, the related solutions reduce the number of reference frames, which means less available information and poorer recovery results. The computing power that this method can reduce is very limited, and due to the reduction of reference frames, the advantage of the multi-frame method that can use the information of adjacent frames will be lost.

[0005] Therefore, how to reduce the computing power overhead of the Transformer-based video multi-frame super-resolution algorithm while increasing the number of reference frames is a technical problem that needs to be urgently solved in video multi-frame image quality enhancement. Summary of the invention

[0006] In view of this, embodiments of the present application provide a video quality enhancement method, apparatus, device and program product.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides a method for enhancing video quality, comprising:

[0009] Divide the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks;

[0010] The video sub-block is input into the converter model for super-resolution processing, and at least one of the temporal offset layer and the spatial offset layer performs offset processing to obtain a super-resolution video sub-block;

[0011] Combine the super-resolution video sub-blocks to obtain the target video with enhanced image quality.

[0012] In a second aspect, an embodiment of the present application provides a video quality enhancement device, comprising:

[0013] A segmentation module, used for segmenting the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks;

[0014] A video processing module, configured to input the video sub-block into the converter model for super-resolution processing, and at the same time, perform offset processing by at least one of a temporal offset layer and a spatial offset layer to obtain a super-resolution video sub-block;

[0015] The combination module is used to combine the super-resolution video sub-blocks to obtain the target video with enhanced image quality.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes to split the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks; while inputting the video sub-blocks into the converter model for super-resolution processing, at least one of the temporal offset layer and the spatial offset layer performs offset processing to obtain super-resolution video sub-blocks; and combining the super-resolution video sub-blocks to obtain a target video with enhanced image quality.

[0017] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions, which, when executed by a processor, can realize segmenting a video to be processed in spatial and temporal dimensions to form video sub-blocks; while inputting the video sub-blocks into a converter model for super-resolution processing, at least one of a temporal offset layer and a spatial offset layer performs offset processing to obtain super-resolution video sub-blocks; and combining the super-resolution video sub-blocks to obtain a target video with enhanced image quality.

[0018] In a fifth aspect, a computer program product includes a computer program or instructions. When the computer program or instructions are executed by a processor, the video to be processed is segmented in the spatial dimension and the temporal dimension to form video sub-blocks; while the video sub-blocks are input into a converter model for super-resolution processing, at least one of the temporal offset layer and the spatial offset layer performs offset processing to obtain super-resolution video sub-blocks; and the super-resolution video sub-blocks are combined to obtain a target video with enhanced image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a method for enhancing video quality according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a video segmentation implementation process provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of video data division provided in an embodiment of the present application;

[0022] Figure 4A A block diagram of a solution for processing a video to be processed provided in an embodiment of the present application;

[0023] Figure 4B A schematic diagram of a time offset provided in an embodiment of the present application;

[0024] Figure 4C A schematic diagram of a horizontal space offset provided in an embodiment of the present application;

[0025] Figure 4D A schematic diagram of a vertical space offset provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of a video quality enhancement device provided in an embodiment of the present application;

[0027] Figure 6 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0029] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0032] The present application embodiment provides a method for enhancing video quality. Figure 1 As shown, the method includes:

[0033] Step S110, dividing the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks;

[0034] Here, the video to be processed is a low-quality video, that is, the video has poor picture quality, the video picture is blurred, jittery, and the color is distorted, which seriously affects the viewing experience. For example, a large number of noise points, blurred areas, or color distortions appear in the picture. Among them, the resolution of the video determines the clarity of the picture. If the resolution is too low, the picture will appear blurred and the details will be seriously lost. In some embodiments, a resolution threshold can be set to determine that a video below the resolution threshold is a low-quality video.

[0035] The spatial dimension refers to the picture information of the video frame, that is, the two-dimensional spatial structure contained in each frame of the image. Each frame of video can be regarded as an image, and each image includes two dimensions: height and width.

[0036] The time dimension refers to the motion information of the video screen, that is, the change relationship between video frames. Video is composed of a series of frames, each of which is captured at a specific point in time. Therefore, video contains a lot of time information.

[0037] In some embodiments, the video to be processed may be first segmented in the spatial dimension to obtain a plurality of sub-image sequences that have completed spatial segmentation; and then each sub-image sequence may be segmented in the temporal dimension to obtain a plurality of video sub-blocks that meet a temporal threshold.

[0038] In some embodiments, the video to be processed may be first segmented in the time dimension to obtain multiple video sequences that meet the time threshold; and then each video sequence may be segmented in the space dimension to obtain multiple video sub-blocks that have completed spatial segmentation.

[0039] Step S120, while inputting the video sub-block into the converter model for super-resolution processing, at least one of the temporal offset layer and the spatial offset layer performs offset processing to obtain a super-resolution video sub-block;

[0040] Here, Video Super Resolution (VSR), also known as Video Super Resolution, is a technology that uses deep learning technology to analyze video images and scenes, combined with video noise reduction, deblurring, sharpening, de-shaking and other image quality enhancement processes to improve video quality. It is a method to increase the resolution of video frames, and can obtain high-resolution images through a series of low-resolution images. Among them, a resolution threshold can be set to determine that images with resolutions higher than the resolution threshold are high-resolution images.

[0041] The Transformer model is a neural network architecture based on the self-attention mechanism, which performs well in processing sequence data. The core of the Transformer model is the self-attention mechanism, which allows the model to pay weighted attention to other elements in the sequence when processing each element in the sequence.

[0042] In the converter structure, the operation with the highest computational power consumption is the attention mechanism, which can be expressed as follows:

[0043]

[0044] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and d represents the dimension of the key vector.

[0045] If no spatial partitioning is performed, Q, K, and V are all tensors of shape (b, t, W*H, c), where c is the channel, b is the number of blocks, and t is the number of frames. A total of two matrix multiplications are performed, and the computational complexity is O(N 2 ), where N is generally the sequence length, and in visual tasks it is the spatial size of the feature map (W*H). Therefore, after completing the spatial division, only the matrix multiplication involved in the attention mechanism can be reduced In addition, due to the reduction of feature dimensions, the computational complexity of most operators in the model can be significantly reduced.

[0046] Here, temporal shift refers to the process of moving or adjusting time series data or video frames in the time dimension. This movement can be forward or backward, aiming to capture the temporal dynamics and dependencies in the data. Temporal shift can capture the temporal dynamics and dependencies in the data, thereby improving the performance of the model. Through temporal shift, the model can better understand the movement patterns and change trends in time series data or video data.

[0047] In some embodiments, a time offset layer can be set in the converter model to introduce a certain time dimension offset while performing super-resolution processing on the video sub-blocks, so as to form a time interaction between the video sub-blocks, so as to achieve the purpose of using information in a larger time range to enhance the image quality optimization effect of the current video sub-block. The time offset can be flexibly adjusted to better meet the needs of users.

[0048] Here, spatial shift refers to the process of moving or adjusting image data or spatially distributed data in the spatial dimension. This movement can be along the X-axis, Y-axis, or Z-axis (in three-dimensional space), aiming to capture spatial features, shape changes, or spatial relationships in the data. Spatial shift helps the model capture local features and spatial relationships in the image. In implementation, spatial shift can be achieved through convolution operations, pooling operations, or spatial transformer networks (STNs). Spatial shift can capture spatial features and shape changes in the data, thereby improving the performance of the model. Through spatial shift, the model can better understand the spatial relationships in the image or data and improve the accuracy of recognition and analysis.

[0049] In some embodiments, a spatial offset layer can be set in the converter model to introduce an offset of a certain spatial dimension while performing super-resolution processing on the video sub-blocks, thereby forming spatial interactions between the various video sub-blocks, so as to achieve the purpose of using information in a larger spatial range to enhance the image quality optimization effect of the current video sub-block. The spatial offset can be flexibly adjusted to better meet the needs of users.

[0050] In some embodiments, by setting a time offset layer and a space offset layer in the converter model, it is possible to introduce a certain time dimension and space dimension offset while performing super-resolution processing on the video sub-blocks, forming an interaction between the various video sub-blocks in time and space, so as to achieve the purpose of using information in a larger time range and space range to enhance the quality optimization effect of the current video sub-block. Among them, the time and space offsets can be flexibly adjusted to better meet the needs of users.

[0051] During the implementation process, video super-resolution supports parallel computing and can perform inference on each video sub-block in parallel, thus improving the efficiency of hardware utilization.

[0052] Step S130: Combine the super-resolution video sub-blocks to obtain a target video with enhanced image quality.

[0053] In the implementation process, the super-resolution video sub-blocks can be combined based on the spatial position and time position of the super-resolution video sub-blocks in the video to be processed to obtain a target video corresponding to the video to be processed. The image quality of the target video is better than that of the video to be processed, that is, the resolution of the target video is higher than that of the video to be processed.

[0054] In the embodiment of the present application, the video to be processed is firstly segmented in the spatial dimension and the temporal dimension to form video sub-blocks; then the video sub-blocks are input into the converter model for super-resolution processing, and at least one of the temporal offset layer and the spatial offset layer performs offset processing to obtain super-resolution video sub-blocks; finally, the super-resolution video sub-blocks are combined to obtain the target video with enhanced image quality. In this way, the video super-resolution method is simple to implement, and uses operations such as feature segmentation and offset supported in the algorithm framework, and the models used by each video sub-block are exactly the same, so only one image quality optimization model needs to be deployed. The computing power of the model can be significantly reduced. A large number of matrix operations are involved in the converter model structure, and when the size of the feature is reduced in the length and width dimensions, a large amount of computing overhead can be saved. Features on the spatiotemporal neighborhood can be fully utilized. Thanks to the temporal and / or spatial offset operations between the video sub-blocks by the temporal offset layer and / or the spatial offset layer, the reference frames of different spatiotemporal ranges that can be used by different layers can fully utilize the features of the temporal neighborhood and the spatial neighborhood to improve the image quality optimization effect. It is highly flexible, and the spatial size of the video sub-blocks, time length, embedding position of the offset operation, and offset size involved can all be adjusted to meet different computing power conditions and scenario requirements.

[0055] In some embodiments, the above step S110 "segmenting the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks" is as follows: Figure 2 As shown, this can be achieved by following the steps below:

[0056] Step S210: Segment the video to be processed in the time dimension based on a preset number of frames to obtain at least one group of video blocks that meet the preset number of frames;

[0057] Here, the video to be processed is composed of multiple continuous video frames. The most important reference information of each frame of video comes from its adjacent frames. If each frame is inferred to refer to all frames of the video, a lot of redundant calculations will be generated. Therefore, the frames of the video to be processed are first divided into video blocks, namely video clips.

[0058] During implementation, the preset number of frames corresponding to the time dimension division can be determined, and the video to be processed can be segmented in the time dimension based on the preset number of frames to obtain at least one set of video blocks that meet the preset number of frames. The preset number of frames can be determined based on the processing capability of the model for processing the video to be processed.

[0059] Figure 3A schematic diagram of video data division provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the schematic diagram includes: time segmentation 31 and space segmentation 32, wherein:

[0060] Time segmentation 31 is to divide the frames of the processed video into video blocks, each of which contains t frames (preset number of frames). When the total number of frames T cannot be divided by t, the remaining part is separately divided into a video segment.

[0061] Step S220: Divide each of the video blocks in a spatial dimension based on a preset length and a preset width to obtain at least one of the video sub-blocks.

[0062] Here, in order to reduce the size of the feature map during the converter model inference and save computing power, after the time dimension is divided, the spatial dimension is also fixed in size, such as Figure 3 The spatial segmentation 32 in the video block can be used to divide the video block into video sub-blocks of fixed size w×h.

[0063] The preset length may be a preset length in the Y-axis direction; the preset width may be a preset width in the X-axis direction.

[0064] During the implementation, the videos to be processed are divided into video sub-blocks, where T is the total number of frames of the video to be processed, and W and H are respectively the original width (width in the X-axis direction) and height (length in the Y-axis direction) of the video frame to be processed.

[0065] In the embodiment of the present application, the video to be processed is first divided in the time dimension based on a preset time period to obtain at least one group of video blocks that meet the preset time period; then each video block is divided in the space dimension based on a preset length and a preset width to obtain at least one video sub-block. In this way, the size of the obtained video sub-block in space and time is much smaller than the original video, the size of the feature map in the network is greatly reduced, and the computing power consumption is effectively reduced.

[0066] In some embodiments, the above step S210 "segmenting the to-be-processed video in the time dimension based on a preset time period to obtain at least one group of video blocks that meet the preset time period" can be implemented by the following steps:

[0067] Step 211: determining the preset number of frames based on a preset time period and a frame rate of the video to be processed;

[0068] During implementation, in some embodiments, a preset time period may be determined first, and then a preset number of frames may be determined based on the preset time period and the frame rate of the video to be processed. For example, the length of time represented by each video frame depends on the frame rate of the video. The frame rate refers to the number of frames displayed per second, and common frame rates include 24 frames / second, 30 frames / second, 60 frames / second, etc. The preset time period may be 0.5 seconds, and a corresponding group of video blocks may include 12 frames, 15 frames, 30 frames, etc. Among them, the preset time period may be determined based on the processing capability of the model for processing the video to be processed.

[0069] Step 212: Segment the video to be processed based on the preset frame number to obtain at least one video segment that meets the preset frame number;

[0070] like Figure 3 The time segmentation 31 shown is to divide the frames of the processed video into video blocks, each of which contains t frames (preset frame number). When the total frame number T cannot be divided by t, the remaining part is separately divided into a video segment. In this way, multiple non-overlapping video segments that meet the preset frame number t can be obtained.

[0071] Step 213: Concatenate the video frames in each of the video segments to form the corresponding video block.

[0072] Here, the entire video is divided into non-overlapping video segments of fixed length t according to the time dimension. Assuming that there are T frames in the video, the total number of frames is Finally, the video frames of each video clip are concatenated to form Figure 3 A video block 311 is shown.

[0073] In the embodiment of the present application, the preset number of frames is first determined based on the preset time period and the frame rate of the video to be processed; then the video to be processed is segmented based on the preset number of frames to obtain at least one video segment that meets the preset number of frames; finally, the video frames in each video segment are cascaded to form a corresponding video block. In this way, after segmenting the video to be processed, multiple video blocks that meet the preset time period can be obtained.

[0074] In some embodiments, the above step S120 "while inputting the video sub-block into the converter model for super-resolution processing, performing offset processing by at least one of the temporal offset layer and the spatial offset layer to obtain a super-resolution video sub-block" can be implemented by the following steps:

[0075] Step 121, setting at least one of the time offset layer and the space offset layer between preset layers of the converter model to obtain a target converter model;

[0076] Here, the temporal shift layer (Temporal Shift) is used to implement the temporal shift of the video sub-blocks input to the target converter model. The spatial shift layer (Spatial Shift) is used to implement the spatial shift of the video sub-blocks input to the target converter model.

[0077] The time offset layer can realize the process of moving or adjusting data in the time dimension. This movement can be linear or nonlinear, depending on the application requirements and data processing methods. In video processing, the time offset layer can move video frames or feature vectors forward or backward on the time axis. In the video processing process, the time offset layer can be implemented by directly moving video frames. This operation can be linearly or nonlinearly transformed based on the timestamp or index of the frame. The time offset layer can be used for tasks such as interpolation, denoising, and super-resolution reconstruction of video frames. And by adjusting the time order or position of video frames, the quality and smoothness of the video can be improved.

[0078] The spatial offset layer can move data in the spatial dimension, introduce local receptive fields, and help capture local features in the image. This movement can be linear, that is, moving in a fixed step size and direction; it can also be nonlinear, that is, moving according to a specific algorithm or model. In the field of image processing, the spatial offset layer can be used for tasks such as image enhancement, image restoration, and image transformation. By moving pixels or features in an image, new visual effects can be created, such as simulating parallax effects, image distortion, or repairing defects in an image. The spatial offset layer can be implemented by directly moving pixels in the image. This operation can be linearly or nonlinearly transformed based on the pixel position to create the desired visual effect. The spatial offset layer can introduce a local receptive field and enhance the model's ability to learn local features in the image. By adjusting the parameters of the spatial offset, the model's learning range of image features can be flexibly controlled. And the spatial offset layer helps to improve the generalization ability of the model and reduce the risk of overfitting.

[0079] The transformer model consists of an input part (input-output embedding and position encoding), a multi-layer encoder, a multi-layer decoder, and an output part (output linear layer and normalization layer).

[0080] In the implementation process, a time offset layer or a space offset layer can be set between the layers of the multi-layer encoder and the multi-layer decoder. The offset of the time and space coordinates is introduced in a specific layer of the converter model, and the amount of calculation required is very small.

[0081] The video sub-block is input into the target converter model to perform at least one of the following steps to obtain the super-resolution video sub-block:

[0082] Step 122, while inputting the video sub-block into the target converter model for super-resolution processing, use the time offset layer to perform time offset;

[0083] During the implementation, a time offset layer is inserted into the converter model, and the frames in all video sub-blocks will be offset from back to front or from front to back according to their frame numbers in the original video.

[0084] Step 123: while inputting the video sub-block into the target converter model for super-resolution processing, use the spatial offset layer to perform a spatial offset operation to obtain the super-resolution video sub-block.

[0085] Here, the spatial position of the video frame is determined by its horizontal coordinate and vertical coordinate. Therefore, in order to ensure that the spatial relationship of the image content is not destroyed while performing the spatial offset, the offsets in these two dimensions can be performed separately.

[0086] During the implementation process, a spatial offset layer is inserted into the converter model. At the position where the spatial offset is performed, the content in the video sub-block will be offset once in the vertical or horizontal direction according to its spatial position. The picture block at the starting position will be offset to the corresponding position at the end according to the offset, or the picture block at the end position will be offset to the corresponding starting position according to the offset.

[0087] In some embodiments, a time offset layer may be set in the converter model to implement time offset while super-resolution is performed on the video sub-block to be processed, wherein the time offset amount may be set according to actual needs.

[0088] In some embodiments, a spatial offset layer may be set in the converter model to implement spatial offset while super-resolution processing is performed on the video sub-block to be processed. The spatial offset may be set according to actual needs. For example, the offset in the X-axis direction, the offset in the Y-axis direction, etc. may be set.

[0089] In some embodiments, the time offset layer and the space offset layer may be set in the converter model at the same time. Since the time offset layer and the space offset layer are set at different positions, in the implementation process, step 122 may be performed first and then step 123, or step 123 may be performed first and then step 122, so as to realize the time offset and the space offset.

[0090] In the embodiment of the present application, at least one of a time offset layer and a space offset layer is first set between the preset layers of the converter model to obtain a target converter model; then the video sub-block is input into the target converter model to perform at least one of the following to obtain a super-resolution video sub-block: while the video sub-block is input into the target converter model for super-resolution processing, the time offset is performed using the time offset layer; while the video sub-block is input into the target converter model for super-resolution processing, the space offset layer is used to perform a space offset operation to obtain a super-resolution video sub-block. In this way, by setting the time offset layer, the time sequence or position of the video frame can be adjusted to improve the quality and smoothness of the video. By setting the space offset layer, the local receptive field can be introduced to enhance the model's learning ability for local features in the image. By adjusting the parameters of the space offset, the model's learning range for image features can be flexibly controlled. And the space offset layer helps to improve the generalization ability of the model and reduce the risk of overfitting. Through the time offset and space offset operations, the information processed by different layers of the converter model can come from different time and space positions of the video. Since the converter model multi-frame image quality enhancement model includes a residual structure, the information of the current video sub-block can also be well preserved. This allows all video sub-blocks to obtain sufficient reference information when performing image quality enhancement.

[0091] Figure 4A A block diagram of a solution for processing a video to be processed provided in an embodiment of the present application, such as Figure 4A As shown, the scheme frame includes: a video to be processed 41, a low-quality video sub-block 42, parallel processing 43, a temporal shift layer (Temporal Shift) 431, a horizontal spatial shift layer (Spatial Horizontal Shift) 432, a vertical spatial shift layer 433 (SpatialVertical Shift) and a high-quality video sub-block 44, wherein, in the process of processing the video to be processed:

[0092] First, the video to be processed 41 is divided into time and space to obtain low-quality video sub-blocks 42;

[0093] Here, the low-quality video sub-block 42 is a video sub-block with poor picture quality, blurry, shaky, and distorted colors. The resolution of the video determines the clarity of the picture. If the resolution is too low, the picture will appear blurry and details will be seriously lost. In some embodiments, a resolution threshold can be set to determine that the video below the resolution threshold is a low-quality video sub-block.

[0094] Then, the low-quality video sub-block 42 is input into a target converter model including a temporal offset layer 431 , a horizontal spatial offset layer 432 , and a vertical spatial offset layer 433 for parallel processing 42 ;

[0095] Finally, a high quality video sub-block 44 is obtained from the output of the target transformer model.

[0096] Here, the high-quality video sub-block 44 is a video sub-block with good video quality and clear video picture. The resolution of the video determines the clarity of the picture. If the resolution is too low, the picture will appear blurry and the details will be seriously lost. In some embodiments, a resolution threshold can be set to determine that the video above the resolution threshold is a high-quality video sub-block.

[0097] In the embodiment of the present application, the scheme is simple to implement, and the size of the feature map inside the model is reduced, thereby significantly reducing the computing power requirements of the converter architecture model, and using the interactive operations between the various video sub-blocks, the converter model can obtain enough information to enhance the image quality of the current frame and ensure performance. In addition, the scheme is very flexible, and the spatial size, time length, and time-space offset of the video sub-block can all be adjusted to make it more in line with the needs of business scenarios.

[0098] In some embodiments, the spatial offset layer includes at least one of the following: a horizontal spatial offset layer and a vertical spatial offset layer; the above step 121 "setting at least one of the time offset layer and the spatial offset layer between the preset layers of the converter model to obtain the target converter model" can be implemented by the following process:

[0099] At least one of the time offset layer, the horizontal space offset layer and the vertical space offset layer is set between different preset layers in the converter model to obtain the target converter model.

[0100] Here, the horizontal spatial shift layer (Spatial Horizontal Shift) can realize the process of moving or adjusting data in the spatial dimension, especially in the horizontal direction. This movement can be linear or nonlinear, depending on the application requirements and data processing methods. In the field of image processing, setting the horizontal spatial shift layer can be used for tasks such as image enhancement, image restoration, and image transformation. For example, by horizontally moving pixels in an image, new visual effects can be created or defects in the image can be repaired. In the implementation process, the horizontal spatial shift layer can achieve the horizontal shift of the image by directly moving the pixels in the image in the horizontal direction.

[0101] The vertical spatial shift layer (Spatial Vertical Shift) can realize the process of moving or adjusting data in the vertical direction of the spatial dimension. This movement can be linear or nonlinear, depending on the application requirements and data processing methods. In the field of image processing, the vertical spatial shift layer can be used for tasks such as image enhancement, image restoration, and image transformation. For example, by vertically shifting pixels in an image, new visual effects can be created, such as simulating parallax effects or repairing defects in the vertical direction of an image. In the implementation process, the vertical spatial shift layer can be achieved by directly shifting pixels in the image in the vertical direction. This operation can perform linear or nonlinear transformations based on the pixel position to create the desired visual effect.

[0102] like Figure 4A As shown, a time offset layer 431, a horizontal space offset layer 432 and a vertical space offset layer 433 may be set between different preset layers in the converter model.

[0103] During implementation, the time offset layer 431, the horizontal space offset layer 432 and the vertical space offset layer 433 are respectively performed at different layers, which can effectively avoid data confusion during the execution of time offset, horizontal offset and vertical offset.

[0104] In the embodiment of the present application, at least one of a time offset layer, a horizontal space offset layer, and a vertical space offset layer is set between different preset layers in the converter model to obtain a target converter model. In this way, the time and space offsets are performed in different layers, respectively, which can effectively avoid data confusion during the execution of time offset, horizontal offset, and vertical offset.

[0105] In some embodiments, the above step 122 "while inputting the video sub-block into the target converter model for super-resolution processing, using the time offset layer to perform time offset" can be implemented by the following process:

[0106] The time offset layer is used to offset the first k frames of the Nth group of video sub-blocks arranged in time order to the last k frames of the N-1th group of video sub-blocks, and at the same time, the first k frames of the first group of video sub-blocks are offset to the last k frames of the last group of video sub-blocks, so as to realize the target converter model performing super-resolution processing and performing time offset at the same time, wherein N is an integer greater than or equal to 2, and k is an integer greater than or equal to 1.

[0107] Here, where the temporal offset layer is inserted into the converter model, the frames in all the video sub-blocks will be offset from back to front by a certain amount according to their corresponding frame numbers in the original video.

[0108] The first frame of the video will be offset to the last position of the video. The size and direction of the offset Δt can be set as required. In addition, the position of the time offset layer inserted in the converter model can also be adjusted according to the application scenario. After adding the time offset layer to perform the time offset operation, different layers of the model can receive frames from different time ranges, and more information can be obtained to help the current video sub-block enhance the image quality.

[0109] Here, a time offset of 1 is taken as an example. Figure 4B A schematic diagram of a time offset provided in an embodiment of the present application, such as Figure 4B As shown, the schematic diagram includes three groups of video sub-blocks to be processed: video sub-block 41, video sub-block 42 and video sub-block 43, wherein each group of video sub-blocks includes 5 sub-images arranged in chronological order. Figure 4B For the sake of illustration, only the case where the offset is 1 is drawn.

[0110] In the process of executing time shift, the first frame image in the video sub-block 42 is shifted to the last frame of the video sub-block 41, and each frame in the video sub-block 41 is shifted forward by one frame in turn; similarly, the first frame image in the video sub-block 43 is shifted to the last frame of the video sub-block 42, and each frame in the video sub-block 42 is shifted forward by one frame in turn; each frame in the video sub-block 43 is shifted forward by one frame in turn, and the first frame image in the video sub-block 41 is shifted to the last frame of the video sub-block 43.

[0111] In the embodiment of the present application, the time offset layer is used to offset the first k frames of the Nth group of video sub-blocks arranged in time order to the last k frames of the N-1th group of video sub-blocks, and at the same time, the first k frames of the first group of video sub-blocks are offset to the last k frames of the last group of video sub-blocks, so as to realize the target converter model to perform super-resolution processing and time offset at the same time. In this way, the value of k can be set according to actual needs. After adding the time offset layer to perform the preset time offset operation, different layers of the model can receive frames from different time ranges, and more information can be obtained to help the current video sub-block enhance the image quality.

[0112] In some embodiments, the above step 123 "while inputting the video sub-block into the target converter model for super-resolution processing, using the spatial offset layer to perform a spatial offset operation" can be implemented by at least one of the following steps:

[0113] Step A, using the horizontal spatial offset layer to offset the Mth group of video sub-blocks arranged in the horizontal direction to the position of the M-1th group of video sub-blocks, and at the same time offset the first group of video sub-blocks to the position of the last group of video sub-blocks, so as to realize the target converter model to perform super-resolution processing and perform horizontal offset operation at the same time, wherein M is a positive integer greater than or equal to 2;

[0114] At the position of horizontal spatial offset, the content in the video sub-block will be offset once in the horizontal direction according to its horizontal spatial position, and the picture block at the starting position will be offset to the corresponding position at the end according to the offset amount. The horizontal spatial offset can flexibly set the offset amount Δw.

[0115] Here, a horizontal space offset of 1 is taken as an example. Figure 4C A schematic diagram of a horizontal space offset provided in an embodiment of the present application is shown in FIG. Figure 4C As shown, the schematic diagram includes: three rows of video sub-block groups, wherein the first row includes four groups of video sub-blocks to be processed, namely, video sub-block 44, video sub-block 45, video sub-block 46 and video sub-block 47. Figure 4C For the sake of illustration, only the case where the offset is 1 is drawn.

[0116] In the process of performing horizontal spatial shifting, video sub-block 47 is shifted to the position of video sub-block 46, video sub-block 46 is shifted to the position of video sub-block 45, video sub-block 45 is shifted to the position of video sub-block 44, and video sub-block 44 is shifted to the position of video sub-block 47.

[0117] Step B, using the vertical spatial offset layer to offset the Pth group of video sub-blocks arranged in the vertical direction to the position of the P-1th group of video sub-blocks, and at the same time offset the 1st group of video sub-blocks to the position of the last group of video sub-blocks, so as to realize the target converter model to perform super-resolution processing and perform vertical offset operation at the same time, wherein P is a positive integer greater than or equal to 2.

[0118] At the position of vertical spatial offset, the content in the video sub-block will be offset once in the vertical direction according to its vertical spatial position, and the picture block at the starting position will be offset to the corresponding position at the end according to the offset amount. The vertical spatial offset can flexibly set the offset amount Δh.

[0119] Here, a vertical space offset of 1 is taken as an example. Figure 4D A schematic diagram of a vertical space offset provided in an embodiment of the present application, such as Figure 4D As shown, the schematic diagram includes: four columns of video sub-block groups, wherein the fourth column includes three groups of video sub-blocks to be processed, namely, video sub-block 47, video sub-block 48 and video sub-block 48. Figure 4D For the sake of illustration, only the case where the offset is 1 is drawn.

[0120] In the process of performing the vertical spatial shift, video sub-block 49 is shifted to the position of video sub-block 48 , video sub-block 48 is shifted to the position of video sub-block 47 , and video sub-block 47 is shifted to the position of video sub-block 49 .

[0121] Here, the spatial position of the video frame is determined by its horizontal coordinate and vertical coordinate. Therefore, in order to ensure that the spatial relationship of the image content is not destroyed while performing the spatial offset, the offsets of these two dimensions are performed separately. Figure 4C and Figure 4D As shown in the figure, at the position of spatial offset, the content in the video sub-block will be offset once in the vertical or horizontal direction according to its spatial position, and the picture block at the starting position will be offset to the corresponding position at the end according to the offset amount. Unlike time offset, in addition to the offset amounts Δw and Δh that can be flexibly adjusted according to scene needs, the direction of spatial offset can also be adjusted as needed to ensure that the sub-block can receive all the required information about the surrounding area when performing inference.

[0122] In the embodiment of the present application, the horizontal space offset layer is used to offset the Mth group of video sub-blocks arranged in the horizontal direction to the position of the M-1th group of video sub-blocks, and the 1st group of video sub-blocks is offset to the position of the last group of video sub-blocks, so as to realize the target converter model to perform the super-resolution processing and the horizontal offset operation at the same time. In this way, after adding the horizontal space offset layer to perform the preset horizontal offset operation, different layers of the model can receive frames from different horizontal space ranges, and more information can be obtained to help the current video sub-block enhance the image quality.

[0123] The vertical spatial offset layer is used to offset the Pth group of video sub-blocks arranged in the vertical direction to the position of the P-1th group of video sub-blocks, and the 1st group of video sub-blocks is offset to the position of the last group of video sub-blocks, so as to realize the vertical offset operation while the target converter model performs super-resolution processing. In this way, after adding the vertical spatial offset layer to perform the preset vertical spatial offset operation, different layers of the model can receive frames from different vertical spatial ranges, and can obtain more information to help the current video sub-block enhance the image quality.

[0124] Based on the foregoing embodiments, the embodiments of the present application provide a video quality enhancement device, which includes the modules included, each module includes each sub-module, each sub-module includes a unit, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0125] Figure 5A schematic diagram of the structure of the video quality enhancement device provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the device 500 includes:

[0126] A segmentation module 510 is used to segment the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks;

[0127] The video processing module 520 is used for inputting the video sub-block into the converter model for super-resolution processing, and performing offset processing by at least one of the temporal offset layer and the spatial offset layer to obtain a super-resolution video sub-block;

[0128] The combining module 530 is used to combine the super-resolution video sub-blocks to obtain a target video with enhanced image quality.

[0129] In some embodiments, the segmentation module 510 includes a first segmentation submodule and a second segmentation submodule, wherein the first segmentation submodule is used to segment the video to be processed in the time dimension based on a preset number of frames to obtain at least one group of video blocks that meet the preset number of frames; the second segmentation submodule is used to divide each of the video blocks in the spatial dimension based on a preset length and a preset width to obtain at least one of the video sub-blocks.

[0130] In some embodiments, the first segmentation submodule includes a determination unit, a segmentation unit and a cascade unit, wherein the determination unit is used to determine the preset number of frames based on a preset time period and a frame rate length of the video to be processed; the segmentation unit is used to segment the video to be processed based on the preset number of frames to obtain at least one video segment that meets the preset number of frames; and the cascade unit is used to cascade the video frames in each of the video segments to form the corresponding video block.

[0131] In some embodiments, the video processing module 520 includes a setting submodule, and at least one of the following submodules: a first video processing submodule and a second video processing submodule, wherein the setting submodule is used to set at least one of the time offset layer and the spatial offset layer between preset layers of the converter model to obtain a target converter model; the first video processing submodule is used to input the video subblock into the target converter model for super-resolution processing while using the spatial offset layer to perform time offset; the second video processing submodule is used to input the video subblock into the target converter model for super-resolution processing while using the spatial offset layer to perform spatial offset operation to obtain the super-resolution video subblock.

[0132] In some embodiments, the spatial offset layer includes at least one of the following: a horizontal spatial offset layer and a vertical spatial offset layer; the setting submodule is used to set at least one of the time offset layer, the horizontal spatial offset layer and the vertical spatial offset layer between different preset layers in the converter model to obtain the target converter model.

[0133] In some embodiments, the first video processing submodule is used to use the time offset layer to offset the first k frames of the Nth group of video sub-blocks arranged in time order to the last k frames of the N-1th group of video sub-blocks, and at the same time offset the first k frames of the first group of video sub-blocks to the last k frames of the last group of video sub-blocks, so as to realize the target converter model performing super-resolution processing and performing time offset at the same time, wherein N is an integer greater than or equal to 2, and k is an integer greater than or equal to 1.

[0134] In some embodiments, the second video processing submodule includes at least one of the following subunits: a first video processing subunit and a second video processing subunit, wherein the first video processing subunit is used to use the horizontal spatial offset layer to offset the Mth group of video subblocks arranged in the horizontal direction to the position of the M-1th group of video subblocks, and at the same time offset the 1st group of video subblocks to the position of the last group of video subblocks, so as to realize the target converter model to perform super-resolution processing and perform horizontal offset operation at the same time, wherein M is a positive integer greater than or equal to 2; the second video processing subunit is used to use the vertical spatial offset layer to offset the Pth group of video subblocks arranged in the vertical direction to the position of the P-1th group of video subblocks, and at the same time offset the 1st group of video subblocks to the position of the last group of video subblocks, so as to realize the target converter model to perform super-resolution processing and perform vertical offset operation at the same time, wherein P is a positive integer greater than or equal to 2.

[0135] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0136] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0137] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in a video quality enhancement method provided in the above embodiment are implemented.

[0138] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and when the processor 602 executes the program, the steps in a video quality enhancement method provided in the above embodiment are implemented.

[0139] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0140] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0141] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0142] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0145] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0146] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0147] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0148] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0149] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0150] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0151] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for enhancing video quality, the method comprising: Divide the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks; While inputting the video sub-block into the converter model for super-resolution processing, at least one of the temporal offset layer and the spatial offset layer performs offset processing to obtain a super-resolution video sub-block; The super-resolution video sub-blocks are combined to obtain a target video with enhanced image quality.

2. The method according to claim 1, wherein the video to be processed is divided in the spatial dimension and the temporal dimension to form video sub-blocks, comprising: Dividing the video to be processed in the time dimension based on a preset number of frames to obtain at least one group of video blocks that meet the preset number of frames; Each of the video blocks is divided in a spatial dimension based on a preset length and a preset width to obtain at least one of the video sub-blocks.

3. The method according to claim 2, wherein the video to be processed is segmented in the time dimension based on a preset number of frames to obtain at least one group of video blocks that meet the preset number of frames, comprising: Determining the preset number of frames based on a preset time period and a frame rate of the video to be processed; Segmenting the video to be processed based on the preset number of frames to obtain at least one video segment satisfying the preset number of frames; The video frames in each of the video segments are concatenated to form the corresponding video block.

4. The method of claim 1, wherein while inputting the video sub-block into the converter model for super-resolution processing, performing offset processing by at least one of a temporal offset layer and a spatial offset layer to obtain a super-resolution video sub-block comprises: Setting at least one of the time offset layer and the space offset layer between preset layers of the converter model to obtain a target converter model; The video sub-block is input into the target converter model to perform at least one of the following steps to obtain the super-resolution video sub-block: While inputting the video sub-blocks into the target converter model for super-resolution processing, using the time offset layer to perform time offset; The video sub-block is input into the target converter model for super-resolution processing, and the spatial offset layer is used to perform a spatial offset operation to obtain the super-resolution video sub-block.

5. The method of claim 4, wherein the spatial offset layer comprises at least one of: a horizontal spatial offset layer and a vertical spatial offset layer; The step of setting at least one of the time offset layer and the space offset layer between the preset layers of the converter model to obtain a target converter model comprises: At least one of the time offset layer, the horizontal space offset layer and the vertical space offset layer is set between different preset layers in the converter model to obtain the target converter model.

6. The method of claim 4, wherein the step of inputting the video sub-blocks into the target converter model for super-resolution processing and performing time shifting using the time shift layer comprises: The time offset layer is used to offset the first k frames of the Nth group of video sub-blocks arranged in time order to the last k frames of the N-1th group of video sub-blocks, and at the same time, the first k frames of the first group of video sub-blocks are offset to the last k frames of the last group of video sub-blocks, so as to realize the target converter model performing super-resolution processing and performing time offset at the same time, wherein N is an integer greater than or equal to 2, and k is an integer greater than or equal to 1.

7. The method of claim 5, wherein the step of inputting the video sub-blocks into the target converter model for super-resolution processing and performing a spatial offset operation using the spatial offset layer comprises at least one of the following steps: The horizontal spatial offset layer is used to offset the Mth group of video sub-blocks arranged in the horizontal direction to the position of the M-1th group of video sub-blocks, and the first group of video sub-blocks is offset to the position of the last group of video sub-blocks, so as to realize the target converter model to perform super-resolution processing and perform horizontal offset operation at the same time, wherein, M is a positive integer greater than or equal to 2; The vertical spatial offset layer is used to offset the Pth group of video sub-blocks arranged based on the vertical direction to the position of the P-1th group of video sub-blocks, and at the same time, the 1st group of video sub-blocks is offset to the position of the last group of video sub-blocks, so as to realize the vertical offset operation while the target converter model performs super-resolution processing, wherein P is a positive integer greater than or equal to 2.

8. A video quality enhancement device, comprising: A segmentation module, used for segmenting the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks; A video processing module, configured to input the video sub-block into a converter model for super-resolution processing, and at the same time, perform offset processing by at least one of a temporal offset layer and a spatial offset layer to obtain a super-resolution video sub-block; The combination module is used to combine the super-resolution video sub-blocks to obtain a target video with enhanced image quality.

9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it divides the video to be processed in the spatial dimension and the temporal dimension to form video sub-blocks; while the video sub-blocks are input into a converter model for super-resolution processing, at least one of the temporal offset layer and the spatial offset layer performs offset processing to obtain super-resolution video sub-blocks; and the super-resolution video sub-blocks are combined to obtain a target video with enhanced image quality.

10. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements segmentation of a video to be processed in spatial and temporal dimensions to form video sub-blocks; while inputting the video sub-blocks into a converter model for super-resolution processing, at least one of a temporal offset layer and a spatial offset layer performs offset processing to obtain super-resolution video sub-blocks; and combining the super-resolution video sub-blocks to obtain a target video with enhanced image quality.

Images