Adaptive resolution sub-pixel interpolation method based on multi-scale network

Abstract

The invention belongs to the technical field of image / video compression, and discloses a multi-scale network-based adaptive resolution sub-pixel interpolation method, which comprises the following steps of: 1, constructing a model selection module; the model selection module selects a corresponding neural network model according to the video resolution, the model selection module is located in an inter-frame prediction module of the codec, and an output network model of the model selection module is used for subsequently generating a sub-pixel prediction block by a sub-pixel interpolation method based on deep learning; 2, constructing a multi-scale feature fusion network based on cavity convolution; step 3, training a multi-scale feature fusion network based on cavity convolution; and step 4, constructing a sub-pixel prediction module based on deep learning. According to the invention, the corresponding neural network model is selected according to the video resolution, the network model is used for a subsequent sub-pixel interpolation task to obtain the sub-pixel prediction block, and the sub-pixel prediction accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image / video compression, and in particular relates to an adaptive resolution pixel interpolation method based on a multi-scale network. Background Art

[0002] In the current era of explosive information production, digital video has become the largest form of media. Video coding and decoding technology has enabled digital video to appear widely in people's daily lives by improving the effectiveness of information. Focusing on the specific technologies in video coding and decoding technology, pixel-by-pixel motion compensation prediction uses the principle of fractional precision motion to effectively remove temporal redundancy. In video coding standards, a method based on DCT fixed coefficient filters is usually used to generate fractional position pixel values, which has the advantage of simple calculation, but also has the problem of unsatisfactory interpolation effect.

[0003] In recent years, many deep learning-based methods have been applied to video encoding and decoding, and the pixel-by-pixel interpolation method based on deep learning has achieved a significant improvement in coding efficiency. Compared with the simple linear interpolation method designed manually, the neural network has a powerful representation ability and can learn more complex and effective interpolation methods in a nonlinear form. Among them, the pixel-by-pixel motion compensation based on the inter-frame regression model directly obtains the pixel-by-pixel prediction result through neural network inference. It defines the input and output of the neural network as the reference image block B and the output of the neural network as B. ref and the current coding block B cur , that is, the neural network needs to learn B ref To B cur The mapping F ref→cur The training set of the model is extracted from the video encoder. The coded blocks using fractional motion vectors in the coded video are selected as labels, and the reference blocks with integer motion vectors are used as inputs to construct sample pairs. A model is trained for each fractional position to generate the prediction block for the corresponding fractional position.

[0004] In the existing technology, there are two problems: 1. A single model cannot be fully applicable to videos of all resolutions. For videos of different resolutions, the current technology uses a model for various videos. However, the models trained from encoded videos of different resolutions have different learned mappings. A single model only learns the mapping at a single resolution and cannot adapt well to other resolutions, which leads to inaccurate pixel prediction results. 2. The performance and complexity of the model cannot be well balanced. As a key module for motion compensation prediction, pixel interpolation is particularly critical in terms of prediction effect and computational complexity. The pixel interpolation task based on the inter-frame regression model has its particularity. Increasing the complexity of the neural network does not result in a proportional improvement in prediction effect, but will greatly reduce the running speed of the codec. Summary of the invention

[0005] The purpose of the present invention is to provide an adaptive resolution pixel interpolation method based on a multi-scale network to solve the above-mentioned technical problems.

[0006] In order to solve the above technical problems, a specific technical solution of a multi-scale network-based adaptive resolution pixel interpolation method of the present invention is as follows:

[0007] An adaptive resolution pixel interpolation method based on a multi-scale network comprises the following steps:

[0008] Step 1: Construct a model selection module; the model selection module selects the corresponding neural network model according to the video resolution. The model selection module is located in the inter-frame prediction module of the codec, and the network model output by the model selection module is used for the subsequent pixel interpolation method based on deep learning to generate pixel prediction blocks;

[0009] Step 2: Construct a multi-scale feature fusion network based on dilated convolution;

[0010] Step 3: Train a multi-scale feature fusion network based on dilated convolution.

[0011] Step 4: Build a deep learning-based pixel prediction module.

[0012] Furthermore, the model selection module input is the video resolution, and the output is the neural network model, and the construction method is as follows:

[0013] Three network models based on multi-scale feature fusion of void convolution are trained using YUV videos with resolutions of 1080p, 720p and 240p respectively: A, B and C. The selection process is as follows: when the width of the video is greater than or equal to 1920 or the height of the video is greater than or equal to 1080, the output of the module is model A; when the width of the video is greater than or equal to 1280 or the height of the video is greater than or equal to 720, the output of the module is model B; for other video resolutions, the output of the module is model C.

[0014] Furthermore, on the encoding side, the input of the model selection module is determined by the resolution of the target video, and based on the HEVC reference software HM, the video resolution is obtained through the SourceWidth and SourceHeight parameters in the video configuration file under the project cfg directory; on the decoding side, the input of the model selection module is determined by the syntactic elements related to the resolution in the bitstream, and the video resolution is obtained using two syntactic elements pic_width_in_luma_samples and pic_height_in_luma_samples in the HEVC sequence parameter set.

[0015] Furthermore, the multi-scale feature fusion network based on dilated convolution in step 2 consists of three stages: shallow feature extraction, deep feature extraction and reconstruction. in represents the input of the network, B out Represents the output of the network. The nonlinear mapping used by the model is LeakyReLU, and its parameters are the default values.

[0016] Furthermore, the shallow feature extraction maps the input image block to a high-dimensional feature space through 3x3 convolution and nonlinear mapping, which is expressed as:

[0017] F s =H SF (B in ), (1)

[0018] Among them, H SF (·) indicates a shallow feature extraction module, where the number of input channels of the 3x3 convolution operation is 1, the number of output channels is 32, the convolution kernel size is 3, and there is no padding.

[0019] Furthermore, in the deep feature extraction, a convolution and nonlinear mapping operation is first performed, which is expressed as:

[0020] F h1 =H h1 (F s ), (2)

[0021] Among them, H h1 (·) represents the convolution and nonlinear mapping process. The number of input channels of the 5x5 convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0022] Convolution kernels of 1x1, 3x3, 5x5, and 7x7 are used to extract features of different scales. Before the 1x1 convolution operation, average pooling is used to align the spatial dimensions. Before the 3x3, 5x5, and 7x7 convolution operations, 1x1 convolution is used to reduce the feature dimension and the amount of calculation. The 5x5 and 7x7 convolution operations are completed by 3x3 convolutions with expansion rates of 2 and 3, respectively. A large receptive field is obtained with fewer parameters. Through feature-level splicing, multi-scale feature maps are directly spliced ​​together to complete the fusion of multi-scale features. The above process is expressed as:

[0023] F Conv1 =Conv1(AvgPool(F h1 )), (3)

[0024] F conv3 =Conv3(Conv1(F h1 )), (4)

[0025] F conv5 =DConv5(Conv1(F h1 )), (5)

[0026] F conv7 =DConv7(Conv1(F h1 )), (6)

[0027] F fused =Concat(F conv1 ,F conv3 ,F conv5 ,F conv7 ), (7)

[0028] Among them, AvgPool(·) represents the average pooling operation, its convolution kernel size is 3, and the convolution step size is 1. Conv1(·) represents the 1x1 convolution operation, its input channel number is 32, the output channel number is 8, and the convolution kernel size is 1. Conv3(·) represents the 3x3 convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, and there is no padding. DConv5(·) represents the 5x5 dilated convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 1, and the dilation rate is 2. DConv7(·) represents the 7x7 dilated convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 2, and the dilation rate is 3. Concat(·) represents the splicing operation along the channel dimension.

[0029] Finally, nonlinear mapping, convolution and nonlinear mapping operations are performed, which can be expressed as:

[0030] F h2 =H h2 (F fused ), (8)

[0031] Among them, H h2 (·) represents the process of nonlinear mapping, convolution and nonlinear mapping. The number of input channels of the convolution operation is 32.

[0032] The number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0033] Furthermore, a 1x1 convolution is used as the reconstruction stage, expressed as:

[0034] B out =Conv1(F h2 ), (9)

[0035] The number of input channels of the convolution operation is 32, the number of output channels is 1, and the convolution kernel size is 1.

[0036] Furthermore, the training steps of step 3 are as follows:

[0037] Three 1080p, 720p, and 240p resolution models: A, B, and C, trained using Cactus_1920x1080_50.yuv, FourPeople_1280x720_60.yuv, and BlowingBubbles_416x240_50.yuv videos, respectively;

[0038] The HEVC reference software HM is used to generate training data, and its encoding configuration uses lowdelay_P_main.

[0039] And change the following options: turn off Merge Mode, MaxCUWidth = 16, MaxCUHeight = 16, MaxPartitionDepth = 2, QuadtreeTULog2MaxSize = 4, QPoffset = 0, after the encoder compresses the video,

[0040] Select the coding blocks whose motion vectors are fractional precision, and extract the following data: POC of the coding block corresponding to the image, the position of the coding block in the image, the size of the coding block, POC of the reference block corresponding to the image, the position of the reference block in the image, the size of the reference block, the integer part and the fractional part of the motion vector. Based on the above information, extract the integer pixel block from the reconstructed video and fill it with 6 pixels around it as B ref , extract the coded block from the target video as B cur , forming a training sample pair (X i ,Y i )=(B ref ,B cur ), make a set of training sets for each sub-pixel position and a set of training sets for different resolution models.

[0041] Furthermore, the step 3 uses the PyTorch deep learning framework, the loss function is SAD, and the optimizer is Adam.

[0042] The batch size is set to 64, the learning rate is set to 0.00028, and the training is performed for 50 rounds.

[0043] Furthermore, the construction steps of step 4 are as follows:

[0044] Using the LibTorch framework, a multi-scale feature fusion network based on dilated convolution is integrated in the reference software HM, which serves as the main component of the deep learning-based pixel-by-pixel prediction module.

[0045] The adaptive resolution pixel interpolation method based on a multi-scale network of the present invention has the following advantages: the present invention selects the corresponding neural network model according to the video resolution, and the output network model is used for subsequent pixel interpolation tasks to generate pixel prediction blocks, thereby improving the accuracy of pixel prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram for explaining the principle;

[0047] Figure 2 It is a schematic diagram of the function and location of the model selection module;

[0048] Figure 3 It is the flow chart of the model selection module;

[0049] Figure 4 This is a schematic diagram of the multi-scale feature fusion network architecture based on dilated convolution; DETAILED DESCRIPTION

[0050] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of an adaptive resolution pixel interpolation method based on a multi-scale network of the present invention in conjunction with the accompanying drawings.

[0051] First, the principle of the method of the present invention is described:

[0052] like Figure 1 As shown, use B ref represents the integer pixel reference block, B cur Indicates the current coding block, F ref→cur Indicates that from B ref To B cur The mapping of S HR→LR Represents the high resolution (HR) block B HR To low resolution (LR) block B LR downsampling process.

[0053] For two videos with different resolutions, assuming that the low-resolution video is downsampled from the high-resolution video, equation 1 holds true:

[0054] B LR =S HR→LR (B HR ), (1)

[0055] According to Equation 1 and Figure 1 The 1 and 2 paths in the equation 2 are obtained as follows:

[0056]

[0057] Since the present invention adopts a convolutional neural network as a network model, its output is related to the input in the receptive field, and downsampling will change the input in the network receptive field, that is, inequality 3 holds:

[0058]

[0059] From equation 2 and inequality 3, we can get

[0060]

[0061] The mapping learned by the neural network model trained by a video of a certain resolution cannot be fully applied to videos of other resolutions, so designing separate models for videos of different resolutions is a good way to improve the accuracy of pixel-by-pixel prediction.

[0062] The present invention provides a multi-scale network-based adaptive resolution pixel interpolation method, comprising the following steps:

[0063] Step 1: Build a model selection module

[0064] like Figure 2 As shown in the figure, the input of the model selection module is the video resolution, and the output is the neural network model. The function of the model selection module is to select the corresponding neural network model according to the video resolution. The model selection module is located in the inter-frame prediction module of the codec, and its output network model is used for the subsequent pixel interpolation method based on deep learning to generate pixel prediction blocks. Three network models based on multi-scale feature fusion of dilated convolution are trained using YUV videos with resolutions of 1080p, 720p, and 240p: A, B, and C. Figure 3 As shown in the figure, the selection process of the module is as follows: when the width of the video is greater than or equal to 1920 or the height of the video is greater than or equal to 1080, the output of the module is model A; when the width of the video is greater than or equal to 1280 or the height of the video is greater than or equal to 720, the output of the module is model B; for other video resolutions, the output of the module is model C.

[0065] On the encoding side, the input of this module is determined by the resolution of the target video. The present invention is based on the HEVC reference software HM, and obtains the video resolution through the SourceWidth and SourceHeight parameters in the video configuration file under the project cfg directory. On the decoding side, the input of this module is determined by the syntactic elements related to the resolution in the bitstream. The present invention uses two syntactic elements pic_width_in_luma_samples and pic_height_in_luma_samples in the HEVC sequence parameter set to obtain the video resolution.

[0066] Step 2: Construct a multi-scale feature fusion network based on dilated convolution

[0067] like Figure 4 As shown in Figure 2, the network consists of three stages: shallow feature extraction, deep feature extraction, and reconstruction. in represents the input of the network, B out Represents the output of the network. The nonlinear mapping used in the model is LeakyReLU, and its parameters are the default values.

[0068] like Figure 4 As shown in shallow feature extraction, shallow feature extraction maps the input image block to a high-dimensional feature space through 3x3 convolution and nonlinear mapping, which is expressed as:

[0069] F s =H SF (B in ), (1)

[0070] Among them, H SF (·) indicates a shallow feature extraction module, where the number of input channels of the 3x3 convolution operation is 1, the number of output channels is 32, the convolution kernel size is 3, and there is no padding.

[0071] like Figure 4 As shown in deep feature extraction, in deep feature extraction, it first undergoes a convolution and nonlinear mapping operation, which is expressed as:

[0072] F h1 =H h1 (F s ), (2)

[0073] Among them, H h1 (·) represents the convolution and nonlinear mapping process. The number of input channels of the 5x5 convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0074] Then, in order to adapt to objects of various scales in the video and more useful feature information, convolution kernels of 1x1, 3x3, 5x5 and 7x7 are used to extract features of different scales. Before the 1x1 convolution operation, average pooling is used to align the spatial dimensions. Before the 3x3, 5x5 and 7x7 convolution operations, 1x1 convolution is used to reduce the feature dimension and reduce the amount of calculation. In particular, the 5x5 and 7x7 convolution operations are completed by 3x3 convolutions with expansion rates of 2 and 3, respectively, and a large receptive field is obtained with fewer parameters. Through feature-level splicing, multi-scale feature maps are directly spliced ​​together to complete the fusion of multi-scale features. The above process is expressed as:

[0075] F Conv1 =Conv1(AvgPool(Fh1 )), (3)

[0076] F conv3 =Conv3(Conv1(F h1 )), (4)

[0077] F conv5 =DConv5(Conv1(F h1 )), (5)

[0078] F conv7 =DConv7(Conv1(F h1 )), (6)

[0079] F fused =Concat(F conv1 ,F conv3 ,F conv5 ,F conv7 ), (7)

[0080] Avg Pool (·) represents an average pooling operation, with a kernel size of 3 and a convolution stride of 1. Conv1 (·) represents a 1x1 convolution operation, with 32 input channels, 8 output channels, and a kernel size of 1. Conv3 (·) represents a 3x3 convolution operation, with 8 input channels, 8 output channels, a kernel size of 3, and no padding. DConv5 (·) represents a 5x5 dilated convolution operation, with 8 input channels, 8 output channels, a kernel size of 3, a padding size of 1, and a dilation rate of 2. DConv7 (·) represents a 7x7 dilated convolution operation, with 8 input channels, 8 output channels, a kernel size of 3, a padding size of 2, and a dilation rate of 3. Concat (·) represents a concatenation operation along the channel dimension.

[0081] Finally, nonlinear mapping, convolution and nonlinear mapping operations are performed, which can be expressed as:

[0082] F h2 =H h2 (F fused ), (8)

[0083] Among them, H h2 (·) represents the process of nonlinear mapping, convolution, and nonlinear mapping. The number of input channels of the convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0084] like Figure 4 As shown in reconstruction, a 1x1 convolution is used as the reconstruction stage, expressed as:

[0085] B out=Conv1(F h2 ), (9)

[0086] The number of input channels of the convolution operation is 32, the number of output channels is 1, and the convolution kernel size is 1.

[0087] Step 3: Train a multi-scale feature fusion network based on dilated convolution

[0088] The three 1080p, 720p and 240p resolution models designed by the present invention: A, B, C, are trained using Cactus_1920x1080_50.yuv, FourPeople_1280x720_60.yuv, and BlowingBubbles_416x240_50.yuv videos respectively.

[0089] The HEVC reference software HM is used to generate the training set data. Its encoding configuration uses lowdelay_P_main and changes the following options: turn off Merge Mode, MaxCUWidth=16, MaxCUHeight=16, MaxPartitionDepth=2, QuadtreeTULog2MaxSize=4, QPoffset=0. After the encoder compresses the video, it selects the coding blocks in which the motion vector is fractional precision and extracts the following data: the POC of the image corresponding to the coding block, the position of the coding block in the image, the size of the coding block, the POC of the image corresponding to the reference block, the position of the reference block in the image, the size of the reference block, the integer part and the fractional part of the motion vector. Based on the above information, the whole pixel block is extracted from the reconstructed video and padded with 6 pixels on all sides as B ref , extract the coded block from the target video as B cur , forming a training sample pair (X i ,Y i )=(B ref ,B cur ). Create a training set for each sub-pixel position and a set of training sets for different resolution models.

[0090] The PyTorch deep learning framework was used, the loss function was SAD, the optimizer was Adam, the batch size was set to 64, the learning rate was set to 0.00028, and 50 rounds of training were performed.

[0091] Step 4: Build a pixel-wise prediction module based on a neural network

[0092] Using the LibTorch framework, a multi-scale feature fusion network based on dilated convolution is integrated in the reference software HM, which serves as the main component of the neural network-based pixel-wise prediction module.

[0093] The present invention is tested according to the general test conditions of JVET, and the test results are as follows:

[0094]

[0095]

[0096] The comparison results of the technical effects of the present invention based on the general test conditions of JCT-VC and other technical solutions are as follows:

[0097]

[0098]

[0099] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. An adaptive resolution pixel interpolation method based on a multi-scale network, characterized in that: The steps include: Step 1: Construct a model selection module; the model selection module selects the corresponding neural network model according to the video resolution. The model selection module is located in the inter-frame prediction module of the codec, and the network model output by the model selection module is used for the subsequent pixel interpolation method based on deep learning to generate pixel prediction blocks; Step 2: Construct a multi-scale feature fusion network based on dilated convolution; Step 3: Train a multi-scale feature fusion network based on dilated convolution; Step 4: Build a deep learning-based pixel prediction module.

2. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 1, characterized in that: The model selection module input is the video resolution, and the output is the neural network model. The construction method is as follows: Three network models based on multi-scale feature fusion of void convolution are trained using YUV videos with resolutions of 1080p, 720p and 240p respectively: A, B and C. The selection process is as follows: when the width of the video is greater than or equal to 1920 or the height of the video is greater than or equal to 1080, the output of the module is model A; when the width of the video is greater than or equal to 1280 or the height of the video is greater than or equal to 720, the output of the module is model B; for other video resolutions, the output of the module is model C.

3. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 1, characterized in that: On the encoding side, the model selection module input is determined by the resolution of the target video. Based on the HEVC reference software HM, the video resolution is obtained through the SourceWidth and SourceHeight parameters in the video configuration file in the project cfg directory; On the decoding side, the input of the model selection module is determined by the resolution-related syntax elements in the bitstream, and the video resolution is obtained using two syntax elements pic_width_in_luma_samples and pic_height_in_luma_samples in the HEVC sequence parameter set.

4. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 1, characterized in that: The multi-scale feature fusion network based on dilated convolution in step 2 consists of three stages: shallow feature extraction, deep feature extraction and reconstruction. in represents the input of the network, B out Represents the output of the network. The nonlinear mapping used by the model is LeakyReLU, and its parameters are the default values.

5. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 4, characterized in that: The shallow feature extraction maps the input image block to a high-dimensional feature space through 3x3 convolution and nonlinear mapping, which is expressed as: F s =H SF (B in ), (1) Among them, H SF (·) indicates a shallow feature extraction module, where the number of input channels of the 3x3 convolution operation is 1, the number of output channels is 32, the convolution kernel size is 3, and there is no padding.

6. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 4, characterized in that: In the deep feature extraction, a convolution and nonlinear mapping operation is first performed, which is expressed as: F h1 =H h1 (F s ), (2) Among them, H h1 (·) represents the convolution and nonlinear mapping process. The number of input channels of the 5x5 convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding. Convolution kernels of 1x1, 3x3, 5x5, and 7x7 are used to extract features of different scales. Before the 1x1 convolution operation, average pooling is used to align the spatial dimensions. Before the 3x3, 5x5, and 7x7 convolution operations, 1x1 convolution is used to reduce the feature dimension and the amount of calculation. The 5x5 and 7x7 convolution operations are completed by 3x3 convolutions with expansion rates of 2 and 3, respectively. A large receptive field is obtained with fewer parameters. Through feature-level splicing, multi-scale feature maps are directly spliced ​​together to complete the fusion of multi-scale features. The above process is expressed as: F Conv1 =Conv1(AvgPool(F h1 )), (3) F Conv3 =Conv3(Conv1(F h1 )), (4) F Conv5 =DConv5(Conv1(F h1 )), (5) F Conv7 =DConv7(Conv1(F h1 )), (6) F fused =Concat(F conv1 ,F conv3 ,F conv5 ,F conv7 ), (7) Among them, AvgPool(·) represents the average pooling operation, its convolution kernel size is 3, and the convolution step size is 1. Conv1(·) represents the 1x1 convolution operation, its input channel number is 32, the output channel number is 8, and the convolution kernel size is 1. Conv3(·) represents the 3x3 convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, and there is no padding. DConv5(·) represents the 5x5 dilated convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 1, and the dilation rate is 2. DConv7(·) represents the 7x7 dilated convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 2, and the dilation rate is 3. Concat(·) represents the splicing operation along the channel dimension. Finally, nonlinear mapping, convolution and nonlinear mapping operations are performed, which can be expressed as: F h2 =H h2 (F fused ), (8) Among them, H h2 (·) represents the process of nonlinear mapping, convolution, and nonlinear mapping. The number of input channels of the convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

7. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 4, characterized in that: Using 1x1 convolution as the reconstruction stage, it is expressed as: B out =Conv1(F h2 ), (9) The number of input channels of the convolution operation is 32, the number of output channels is 1, and the convolution kernel size is 1.

8. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 1, characterized in that: The training steps of step 3 are as follows: Three 1080p, 720p, and 240p resolution models: A, B, and C, trained using Cactus_1920x1080_50.yuv, FourPeople_1280x720_60.yuv, and BlowingBubbles_416x240_50.yuv videos, respectively; The HEVC reference software HM is used to generate the training set data. Its encoding configuration adopts lowdelay_P_main and changes the following options: turn off Merge Mode, MaxCUWidth=16, MaxCUHeight=16, MaxPartitionDepth=2, QuadtreeTULog2MaxSize=4, QPoffset=0. After the encoder compresses the video, it selects the coding blocks whose motion vectors are fractional precision, and extracts the following data: POC of the coding block corresponding to the image, the position of the coding block in the image, the size of the coding block, POC of the reference block corresponding to the image, the position of the reference block in the image, the size of the reference block, the integer part and the fractional part of the motion vector. Based on the above information, the whole pixel block is extracted from the reconstructed video and padded with 6 pixels around it as B ref , extract the coded block from the target video as B cur , forming a training sample pair (X i ,Y i )=(B ref ,B cur ), make a set of training sets for each sub-pixel position and a set of training sets for different resolution models.

9. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 1, characterized in that: In step 3, the PyTorch deep learning framework is used, the loss function is SAD, the optimizer is Adam, the batch size is set to 64, the learning rate is set to 0.00028, and 50 rounds of training are performed.

10. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 1, characterized in that: The step 4 uses the LibTorch framework to integrate a multi-scale feature fusion network based on dilated convolution in the reference software HM.

Images