An adaptive resolution pixel-by-pixel interpolation method based on multi-scale network

Abstract

The present invention belongs to the field of image / video compression technology and discloses a multi-scale network-based adaptive resolution pixel interpolation method, comprising the following steps: Step 1: Constructing a model selection module; the model selection module selects a corresponding neural network model based on the video resolution. The model selection module is located in the inter-frame prediction module of the codec, and the network model output by the model selection module is used in a subsequent deep learning-based pixel interpolation method to generate pixel prediction blocks; Step 2: Constructing a multi-scale feature fusion network based on dilated convolution; Step 3: Training the multi-scale feature fusion network based on dilated convolution; Step 4: Constructing a deep learning-based pixel prediction module. The present invention selects a corresponding neural network model based on the video resolution, and the network model is used in subsequent pixel interpolation tasks to obtain pixel prediction blocks, thereby improving the accuracy of pixel prediction.

Description

Technical Field

[0001] The present invention belongs to the field of image / video compression technology, and in particular relates to an adaptive resolution pixel interpolation method based on a multi-scale network. Background Art

[0002] In today's era of explosive information production, digital video has become the largest media format. Video coding and decoding technologies, by improving information efficiency, have enabled the widespread use of digital video in our daily lives. Focusing on specific technologies within video coding and decoding, pixel-by-pixel motion compensation prediction utilizes the principle of fractional motion precision to effectively remove temporal redundancy. Video coding standards typically use a DCT-based fixed-coefficient filter to generate fractional pixel values. This method offers the advantage of computational simplicity, but also suffers from unsatisfactory interpolation performance.

[0003] In recent years, many deep learning-based methods have been applied to video encoding and decoding, and the pixel-by-pixel interpolation method based on deep learning has achieved a significant improvement in coding efficiency. Compared with the simple linear interpolation method designed manually, neural networks have powerful representation capabilities and can learn more complex and effective interpolation methods in a nonlinear form. Among them, the pixel-by-pixel motion compensation based on the inter-frame regression model directly obtains the pixel-by-pixel prediction results through neural network inference. It defines the input and output of the neural network as the reference image block B, ref and the current coding block B cur , that is, the neural network needs to learn B ref to B cur The mapping F ref→cur The model's training set is extracted from a video encoder. Coded blocks using fractional motion vectors are selected as labels, and reference blocks using integer motion vectors are used as input. Sample pairs are constructed from these two. A model is trained for each fractional position to generate a prediction block for that fractional position.

[0004] In the existing technology, there are two problems: 1. A single model cannot be fully applicable to videos of all resolutions. For videos of different resolutions, the current technology adopts a model for various videos. However, the models trained from encoded videos of different resolutions learn different mappings. A single model only learns the mapping at a single resolution and cannot adapt well to situations with other resolutions, which leads to inaccurate pixel prediction results. 2. The performance and complexity of the model cannot be well balanced. As a key module of motion compensation prediction, pixel interpolation is particularly critical in terms of prediction effect and computational complexity. The pixel interpolation task based on the inter-frame regression model has its own particularity. Increasing the complexity of the neural network does not achieve a proportional improvement in prediction effect, but will greatly reduce the running speed of the codec. Summary of the Invention

[0005] The purpose of the present invention is to provide an adaptive resolution pixel-by-pixel interpolation method based on a multi-scale network to solve the above technical problems.

[0006] To solve the above technical problems, the specific technical solution of the adaptive resolution pixel interpolation method based on a multi-scale network of the present invention is as follows:

[0007] A multi-scale network-based adaptive resolution pixel interpolation method includes the following steps:

[0008] Step 1: Construct a model selection module; the model selection module selects the corresponding neural network model according to the video resolution. The model selection module is located in the inter-frame prediction module of the codec, and the network model it outputs is used to generate pixel-by-pixel prediction blocks using a subsequent deep learning-based pixel-by-pixel interpolation method.

[0009] Step 2: Construct a multi-scale feature fusion network based on dilated convolution;

[0010] Step 3: Train a multi-scale feature fusion network based on dilated convolution.

[0011] Step 4: Build a deep learning-based pixel prediction module.

[0012] Furthermore, the model selection module input is the video resolution, and the output is the neural network model. The construction method is as follows:

[0013] Three network models based on multi-scale feature fusion of dilated convolution are trained using YUV videos with resolutions of 1080p, 720p, and 240p: A, B, and C. The selection process is as follows: when the width of the video is greater than or equal to 1920 or the height of the video is greater than or equal to 1080, the output of the module is model A; when the width of the video is greater than or equal to 1280 or the height of the video is greater than or equal to 720, the output of the module is model B; for other video resolutions, the output of the module is model C.

[0014] Furthermore, on the encoding side, the input of the model selection module is determined by the resolution of the target video, and based on the HEVC reference software HM, the video resolution is obtained through the SourceWidth and SourceHeight parameters in the video configuration file under the project cfg directory; on the decoding side, the input of the model selection module is determined by the resolution-related syntactic elements in the bitstream, and the video resolution is obtained using the two syntactic elements pic_width_in_luma_samples and pic_height_in_luma_samples in the HEVC sequence parameter set.

[0015] Furthermore, the multi-scale feature fusion network based on dilated convolution in step 2 consists of three stages: shallow feature extraction, deep feature extraction and reconstruction. in represents the input of the network, B out Represents the output of the network. The nonlinear mapping used by the model is LeakyReLU, and its parameters are the default values.

[0016] Furthermore, the shallow feature extraction maps the input image block to a high-dimensional feature space through 3x3 convolution and nonlinear mapping, which is expressed as:

[0017] F s =H SF (B in ), (1)

[0018] Among them H SF (·) represents the shallow feature extraction module, the number of input channels of the 3x3 convolution operation is 1, the number of output channels is 32, the convolution kernel size is 3, and there is no padding.

[0019] Furthermore, in the deep feature extraction, a convolution and nonlinear mapping operation is first performed, which can be expressed as:

[0020] F h1 =H h1 (F s ), (2)

[0021] Among them H h1 (·) represents the convolution and nonlinear mapping process. The number of input channels of the 5x5 convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0022] Use 1x1, 3x3, 5x5, and 7x7 convolution kernels to extract features of different scales. Before the 1x1 convolution operation, use average pooling to align the spatial size. Before the 3x3, 5x5, and 7x7 convolution operations, use 1x1 convolution to reduce the feature dimension and reduce the amount of calculation. The 5x5 and 7x7 convolution operations are completed by 3x3 convolution with expansion rates of 2 and 3, respectively. Fewer parameters are used to obtain a large receptive field. Through feature-level splicing, the multi-scale feature maps are directly spliced ​​together to complete the fusion of multi-scale features. The above process is expressed as:

[0023] F Conv1 =Conv1(AvgPool(F h1 )), (3)

[0024] F conv3 =Conv3(Conv1(F h1 )), (4)

[0025] F conv5 =DConv5(Conv1(F h1 )), (5)

[0026] F conv7 =DConv7(Conv1(F h1 )), (6)

[0027] F fused =Concat(F conv1 ,F conv3 ,F conv5 ,F conv7 ), (7)

[0028] Among them, AvgPool(·) represents the average pooling operation, its convolution kernel size is 3, and the convolution stride is 1. Conv1(·) represents the 1x1 convolution operation, its input channel number is 32, the output channel number is 8, and the convolution kernel size is 1. Conv3(·) represents the 3x3 convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, and there is no padding. DConv5(·) represents the 5x5 void convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 1, and the dilation rate is 2. DConv7(·) represents the 7x7 void convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 2, and the dilation rate is 3. Concat(·) represents the splicing operation along the channel dimension;

[0029] Finally, the operations of nonlinear mapping, convolution and nonlinear mapping are performed, which can be expressed as:

[0030] F h2 =H h2 (F fused ), (8)

[0031] Among them H h2 (·) represents the process of nonlinear mapping, convolution and nonlinear mapping. The number of input channels of the convolution operation is 32.

[0032] The number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0033] Furthermore, 1x1 convolution is used as the reconstruction stage, which is expressed as:

[0034] B out =Conv1(F h2 ), (9)

[0035] The number of input channels of the convolution operation is 32, the number of output channels is 1, and the convolution kernel size is 1.

[0036] Furthermore, the training steps of step 3 are as follows:

[0037] Three models with 1080p, 720p, and 240p resolutions: A, B, and C, trained using the Cactus_1920x1080_50.yuv, FourPeople_1280x720_60.yuv, and BlowingBubbles_416x240_50.yuv videos, respectively;

[0038] The HEVC reference software HM is used to generate training data, and its encoding configuration adopts lowdelay_P_main.

[0039] And change the following options: turn off Merge Mode, MaxCUWidth=16, MaxCUHeight=16, MaxPartitionDepth=2, QuadtreeTULog2MaxSize=4, QPoffset=0, after the encoder compresses the video,

[0040] Select the coding blocks whose motion vectors are fractional precision, and extract the following data: POC of the coding block corresponding to the image, the position of the coding block in the image, the size of the coding block, POC of the reference block corresponding to the image, the position of the reference block in the image, the size of the reference block, the integer part and the fractional part of the motion vector. Based on the above information, extract the whole pixel block from the reconstructed video and fill it with 6 pixels around it as B ref , extract the coding block from the target video as B cur , forming a training sample pair (X i ,Y i )=(B ref ,B cur ), make a set of training sets for each sub-pixel position, and make a set of training sets for different resolution models.

[0041] Furthermore, the step 3 uses the PyTorch deep learning framework, the loss function is SAD, and the optimizer is Adam.

[0042] The batch size is set to 64, the learning rate is set to 0.00028, and the training is performed for 50 rounds.

[0043] Furthermore, the construction steps of step 4 are as follows:

[0044] Using the LibTorch framework, a multi-scale feature fusion network based on dilated convolution is integrated into the reference software HM. This network serves as the main component of the deep learning-based pixel-wise prediction module.

[0045] The adaptive resolution pixel interpolation method based on a multi-scale network of the present invention has the following advantages: the present invention selects the corresponding neural network model according to the video resolution, and the output network model is used for subsequent pixel interpolation tasks to generate pixel prediction blocks, thereby improving the accuracy of pixel prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram illustrating the principle;

[0047] Figure 2 It is a schematic diagram of the function and location of the model selection module;

[0048] Figure 3 It is the flow chart of the model selection module;

[0049] Figure 4 This is a schematic diagram of the multi-scale feature fusion network architecture based on dilated convolution; DETAILED DESCRIPTION

[0050] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of the adaptive resolution pixel interpolation method based on a multi-scale network of the present invention in conjunction with the accompanying drawings.

[0051] First, the principle of the method of the present invention is described:

[0052] like Figure 1 As shown, use B ref represents the integer pixel reference block, B cur Indicates the current coding block, F ref→cur Indicates that from B ref To B cur The mapping of S HR→LR Indicates that the high resolution (HR) block B HR To low resolution (LR) block B LR downsampling process.

[0053] For two videos with different resolutions, assuming that the low-resolution video is downsampled from the high-resolution video, equation 1 holds true:

[0054] B LR =S HR→LR (B HR ), (1)

[0055] According to Equation 1 and Figure 1 Paths 1 and 2 in the equation 2 are obtained as follows:

[0056]

[0057] Since the present invention adopts a convolutional neural network as the network model, its output is related to the input within the receptive field, and downsampling will change the input within the network receptive field, that is, inequality 3 holds:

[0058]

[0059] From Equation 2 and Inequality 3, we can get

[0060]

[0061] The mapping learned by the neural network model trained on a video of a certain resolution cannot be fully applied to videos of other resolutions. Therefore, designing separate models for videos of different resolutions is a good way to improve the accuracy of pixel-by-pixel prediction.

[0062] The present invention provides a multi-scale network-based adaptive resolution pixel interpolation method, comprising the following steps:

[0063] Step 1: Build a model selection module

[0064] like Figure 2 As shown in the figure, the input of the model selection module is the video resolution, and the output is the neural network model. The function of the model selection module is to select the corresponding neural network model according to the video resolution. The model selection module is located in the inter-frame prediction module of the codec. The network model it outputs is used for the subsequent pixel-by-pixel interpolation method based on deep learning to generate pixel-by-pixel prediction blocks. Three network models based on multi-scale feature fusion of dilated convolution are trained using YUV videos with resolutions of 1080p, 720p, and 240p: A, B, and C. Figure 3 As shown in the figure, the selection process of the module is as follows: when the width of the video is greater than or equal to 1920 or the height of the video is greater than or equal to 1080, the output of the module is model A; when the width of the video is greater than or equal to 1280 or the height of the video is greater than or equal to 720, the output of the module is model B; for other video resolutions, the output of the module is model C.

[0065] On the encoding side, the module's input is determined by the target video's resolution. This paper, based on the HEVC reference software HM, obtains the video resolution through the SourceWidth and SourceHeight parameters in the video configuration file under the project's cfg directory. On the decoding side, the module's input is determined by resolution-related syntax elements in the bitstream. This paper uses two syntax elements in the HEVC sequence parameter set, pic_width_in_luma_samples and pic_height_in_luma_samples, to obtain the video resolution.

[0066] Step 2: Construct a multi-scale feature fusion network based on dilated convolution

[0067] like Figure 4 As shown, the network consists of three stages: shallow feature extraction, deep feature extraction and reconstruction. in represents the input of the network, B out Represents the output of the network. The nonlinear mapping used in the model is LeakyReLU, and its parameters are the default values.

[0068] like Figure 4 As shown in the shallow feature extraction, shallow feature extraction maps the input image block to a high-dimensional feature space through 3x3 convolution and nonlinear mapping, which is expressed as:

[0069] F s =H SF (B in ), (1)

[0070] Among them H SF (·) indicates a shallow feature extraction module, where the number of input channels of the 3x3 convolution operation is 1, the number of output channels is 32, the convolution kernel size is 3, and there is no padding.

[0071] like Figure 4 As shown in the deep feature extraction, in deep feature extraction, it first undergoes a convolution and nonlinear mapping operation, which is expressed as:

[0072] F h1 =H h1 (F s ), (2)

[0073] Among them H h1 (·) represents the convolution and nonlinear mapping process. The number of input channels of the 5x5 convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0074] Then, in order to adapt to objects of various scales in the video and more useful feature information, convolution kernels of 1x1, 3x3, 5x5 and 7x7 are used to extract features of different scales. Before the 1x1 convolution operation, average pooling is used to align the spatial dimensions. Before the 3x3, 5x5 and 7x7 convolution operations, 1x1 convolution is used to reduce the feature dimension and reduce the amount of calculation. In particular, the 5x5 and 7x7 convolution operations are completed by 3x3 convolution with expansion rates of 2 and 3, respectively, and a large receptive field is obtained with fewer parameters. Through feature-level splicing, the multi-scale feature maps are directly spliced ​​together to complete the fusion of multi-scale features. The above process is expressed as:

[0075] F Conv1 =Conv1(AvgPool(Fh1 )), (3)

[0076] F conv3 =Conv3(Conv1(F h1 )), (4)

[0077] F conv5 =DConv5(Conv1(F h1 )), (5)

[0078] F conv7 =DConv7(Conv1(F h1 )), (6)

[0079] F fused =Concat(F conv1 ,F conv3 ,F conv5 ,F conv7 ), (7)

[0080] Avg Pool (·) represents an average pooling operation with a kernel size of 3 and a stride of 1. Conv1 (·) represents a 1x1 convolution operation with 32 input channels, 8 output channels, and a kernel size of 1. Conv3 (·) represents a 3x3 convolution operation with 8 input channels, 8 output channels, a kernel size of 3, and no padding. DConv5 (·) represents a 5x5 atrous convolution operation with 8 input channels, 8 output channels, a kernel size of 3, a padding size of 1, and a dilation rate of 2. DConv7 (·) represents a 7x7 atrous convolution operation with 8 input channels, 8 output channels, a kernel size of 3, a padding size of 2, and a dilation rate of 3. Concat (·) represents a concatenation operation along the channel dimension.

[0081] Finally, the operations of nonlinear mapping, convolution and nonlinear mapping are performed, which can be expressed as:

[0082] F h2 =H h2 (F fused ), (8)

[0083] Among them H h2 (·) represents the process of nonlinear mapping, convolution, and nonlinear mapping. The number of input channels of the convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

[0084] like Figure 4 As shown in reconstruction, 1x1 convolution is used as the reconstruction stage, which is expressed as:

[0085] B out=Conv1(F h2 ), (9)

[0086] The number of input channels of the convolution operation is 32, the number of output channels is 1, and the convolution kernel size is 1.

[0087] Step 3: Train a multi-scale feature fusion network based on dilated convolution

[0088] The three 1080p, 720p and 240p resolution models designed in this invention: A, B, and C are trained using the Cactus_1920x1080_50.yuv, FourPeople_1280x720_60.yuv, and BlowingBubbles_416x240_50.yuv videos, respectively.

[0089] The HEVC reference software HM is used to generate the training set data. Its encoding configuration adopts lowdelay_P_main and changes the following options: turn off Merge Mode, MaxCUWidth=16, MaxCUHeight=16, MaxPartitionDepth=2, QuadtreeTULog2MaxSize=4, QPoffset=0. After the encoder compresses the video, it selects the coding blocks whose motion vectors are fractional precision and extracts the following data: the POC of the coding block corresponding to the image, the position of the coding block in the image, the size of the coding block, the POC of the reference block corresponding to the image, the position of the reference block in the image, the size of the reference block, the integer part and the fractional part of the motion vector. Based on the above information, the whole pixel block is extracted from the reconstructed video and padded with 6 pixels on all sides as B ref , extract the coding block from the target video as B cur , forming a training sample pair (X i ,Y i )=(B ref ,B cur ). Create a training set for each sub-pixel position and a set of training sets for different resolution models.

[0090] The PyTorch deep learning framework is used, the loss function is SAD, the optimizer is Adam, the batch size is set to 64, the learning rate is set to 0.00028, and training is performed for 50 rounds.

[0091] Step 4: Build a pixel-wise prediction module based on a neural network

[0092] Using the LibTorch framework, a multi-scale feature fusion network based on dilated convolution is integrated into the reference software HM, which serves as the main component of the neural network-based pixel-wise prediction module.

[0093] The present invention was tested according to the general test conditions of JVET, and the test results are as follows:

[0094]

[0095]

[0096] The comparison results of the technical effects of the present invention based on the general test conditions of JCT-VC and other technical solutions are as follows:

[0097]

[0098]

[0099] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A multi-scale network-based adaptive resolution pixel interpolation method, characterized in that: The steps include: Step 1: Construct a model selection module; the model selection module selects the corresponding neural network model according to the video resolution. The model selection module is located in the inter-frame prediction module of the codec. The network model it outputs is used for the subsequent pixel-by-pixel interpolation method based on deep learning to generate pixel-by-pixel prediction blocks. Three network models based on multi-scale feature fusion based on dilated convolution are trained using YUV videos with 1080p, 720p, and 240p resolutions: A, B, and C. Step 2: Construct a multi-scale feature fusion network based on dilated convolution; The multi-scale feature fusion network based on dilated convolution in step 2 consists of three stages: shallow feature extraction, deep feature extraction and reconstruction. in represents the input of the network, B out Represents the output of the network. The nonlinear mapping used in the model is LeakyReLU, and its parameters are default values. The shallow feature extraction maps the input image block to a high-dimensional feature space through 3x3 convolution and nonlinear mapping. It first undergoes a convolution and nonlinear mapping operation, and finally performs nonlinear mapping, convolution, and nonlinear mapping operations. Using 1x1 convolution as the reconstruction stage, it is expressed as: B out =Conv1(F h2 ), (9) The number of input channels of the convolution operation is 32, the number of output channels is 1, and the convolution kernel size is 1; Step 3: Train a multi-scale feature fusion network based on dilated convolution; Three models with 1080p, 720p, and 240p resolutions: A, B, and C, trained using the Cactus_1920x1080_50.yuv, FourPeople_1280x720_60.yuv, and BlowingBubbles_416x240_50.yuv videos, respectively; The HEVC reference software HM is used to generate the training set data. Its encoding configuration adopts lowdelay_P_main and changes the following options: turn off Merge Mode, MaxCUWidth=16, MaxCUHeight=16, MaxPartitionDepth=2, QuadtreeTULog2MaxSize=4, QPoffset=0. After the encoder compresses the video, it selects the coding blocks whose motion vectors are fractional precision and extracts the following data: POC of the coding block corresponding to the image, the position of the coding block in the image, the size of the coding block, POC of the reference block corresponding to the image, the position of the reference block in the image, the size of the reference block, the integer part and the fractional part of the motion vector. Based on the above information, the whole pixel block is extracted from the reconstructed video and padded with 6 pixels around it as B ref , extract the coding block from the target video as B cur , forming a training sample pair (X i ,Y i )=(B ref ,B cur ), create a set of training sets for each sub-pixel position and a set of training sets for different resolution models; Step 4: Build a deep learning-based pixel prediction module.

2. The multi-scale network-based adaptive resolution pixel interpolation method according to claim 1, characterized in that: The model selection module input is video resolution, and output is a neural network model. The construction method is as follows: The selection process is as follows: when the video width is greater than or equal to 1920 or the video height is greater than or equal to 1080, the module output is model A; when the video width is greater than or equal to 1280 or the video height is greater than or equal to 720, the module output is model B; for other video resolutions, the module output is model C.

3. The multi-scale network-based adaptive resolution pixel interpolation method according to claim 1, characterized in that: On the encoding side, the model selection module input is determined by the resolution of the target video. Based on the HEVC reference software HM, the video resolution is obtained through the SourceWidth and SourceHeight parameters in the video configuration file in the project cfg directory; On the decoding side, the input of the model selection module is determined by the resolution-related syntax elements in the bitstream, and the video resolution is obtained by using the two syntax elements pic_width_in_luma_samples and pic_height_in_luma_samples in the HEVC sequence parameter set.

4. The multi-scale network-based adaptive resolution pixel interpolation method according to claim 3, characterized in that: The shallow feature extraction maps the input image block to a high-dimensional feature space through 3x3 convolution and nonlinear mapping, which is expressed as: F s =H SF (B in ), (1) Among them H SF (·) indicates a shallow feature extraction module, where the number of input channels of the 3x3 convolution operation is 1, the number of output channels is 32, the convolution kernel size is 3, and there is no padding.

5. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 3, wherein: In the deep feature extraction, a convolution and nonlinear mapping operation is first performed, which is expressed as: F h1 =H h1 (F s ), (2) Among them H h1 (·) represents the convolution and nonlinear mapping process. The number of input channels of the 5x5 convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding. Use 1x1, 3x3, 5x5, and 7x7 convolution kernels to extract features of different scales. Before the 1x1 convolution operation, use average pooling to align the spatial size. Before the 3x3, 5x5, and 7x7 convolution operations, use 1x1 convolution to reduce the feature dimension and reduce the amount of calculation. The 5x5 and 7x7 convolution operations are completed by 3x3 convolution with expansion rates of 2 and 3, respectively. Fewer parameters are used to obtain a large receptive field. Through feature-level splicing, the multi-scale feature maps are directly spliced ​​together to complete the fusion of multi-scale features. The above process is expressed as: F Conv1 =Conv1(AvgPool(F h1 )), (3) F Conv3 =Conv3(Conv1(Fh1)), (4) F Conv5 =DConv5(Conv1(Fh1)), (5) F Conv7 =DConv7(Conv1(F h1 )), (6) F fused =Concat(F conv1 ,F conv3 ,F conv5 ,F conv7 ), (7) Among them, AvgPool(·) represents the average pooling operation, its convolution kernel size is 3, the convolution stride is 1, Conv1(·) represents the 1x1 convolution operation, its input channel number is 32, the output channel number is 8, the convolution kernel size is 1, Conv3(·) represents the 3x3 convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, and there is no padding, DConv5(·) represents the 5x5 void convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 1, and the expansion rate is 2, DConv7(·) represents the 7x7 void convolution operation, its input channel number is 8, the output channel number is 8, the convolution kernel size is 3, the padding size is 2, and the expansion rate is 3, Concat(·) represents the splicing operation along the channel dimension; finally, the nonlinear mapping, convolution and nonlinear mapping operations are performed, which are expressed as: F h2 =H h2 (F fused ), (8) Among them H h2 (·) represents the process of nonlinear mapping, convolution, and nonlinear mapping. The number of input channels of the convolution operation is 32, the number of output channels is 32, the convolution kernel size is 5, and there is no padding.

6. The multi-scale network-based adaptive resolution pixel interpolation method according to claim 1, characterized in that: In step 3, the PyTorch deep learning framework is used, the loss function is SAD, the optimizer is Adam, the batch size is set to 64, the learning rate is set to 0.00028, and 50 rounds of training are performed.

7. The method for adaptive resolution pixel interpolation based on a multi-scale network according to claim 1, wherein: In step 4, the LibTorch framework is used to integrate a multi-scale feature fusion network based on dilated convolution in the reference software HM.

Images