Image processing network, image resolution expansion method, device, equipment and medium

Abstract

The application discloses an image processing network, an image resolution expansion method, a display device, equipment and a medium. The image processing network of one embodiment comprises: an image input end for acquiring an original image to be processed; a first convolution module connected in series with the image input end, configured to perform convolution processing on the original image with a preset first convolution size to output a first image; a feature extraction module connected in series with the first convolution module, configured to perform residual and pooling attention extraction on the first image to output a second image; a first feature summation module configured to perform feature summation processing on the second image input by the first input end and the first image input by the second input end; a sampling module connected in series with the first feature summation module, configured to perform sampling processing on the image output by the first feature summation module to generate a super-resolution image; and an image output end configured to output the super-resolution image generated by the sampling module.

Description

Technical Field

[0001] This invention relates to the field of display technology. More specifically, it relates to an image processing network, an image resolution expansion method, apparatus, device, and medium. Background Technology

[0002] As display devices upgrade and iterate, users are experiencing increasingly higher image and video resolutions, such as transitioning from 480p to 720p, 2K, and 4K. However, because higher resolutions require greater computational power, most existing image super-resolution models are complex, large models, ranging from hundreds of megabytes in size, making them difficult to deploy on terminals. While smaller image super-resolution models can be deployed on terminals, their computational power increases exponentially with the resolution of the input video source, as the super-resolution model remains fixed. Therefore, integrating image super-resolution models into terminal devices and rationally configuring the terminal's image processing computing power is crucial. Summary of the Invention

[0003] The purpose of this invention is to provide an image processing network, an image resolution expansion method, an apparatus, a device, and a medium to solve at least one of the problems existing in the prior art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] The first aspect of this invention provides an image processing network based on a deep learning network, characterized in that the image processing network comprises:

[0006] Image input terminal, used to acquire the raw image to be processed;

[0007] The first convolution module is connected in series with the image input terminal and is used to perform convolution processing on the original image with a preset first convolution size to output the first image;

[0008] The feature extraction module, connected in series with the first convolution module, is used to perform residual and pooling attention extraction on the first image to output a second image.

[0009] The first feature summation module includes a first input terminal connected to the output terminal of the feature extraction module and a second input terminal connected to the first convolution module, and is used to perform feature summation processing on the second image input to the first input terminal and the first image input to the second input terminal;

[0010] The sampling module, connected in series with the first feature summation module, is used to sample the image output by the first feature summation module to generate a super-resolution image.

[0011] The image output terminal is used to output the super-resolution image generated by the sampling module.

[0012] Furthermore, the feature extraction module includes:

[0013] Multiple cascaded feature extraction units, with the feature extraction unit located at one end of the cascaded sequence connected to the first convolutional module, output a pooled attention image after feature extraction; and

[0014] The second convolutional unit is connected at one end to the feature extraction unit located at the other end of the series connection, and at the other end to the second input end of the first feature summing module. It is used to convolve the pooling attention image input to the connected feature extraction unit to output the second image.

[0015] The feature extraction unit includes:

[0016] A residual extraction unit is configured to extract residuals from an image input to the residual extraction unit; and

[0017] A pooling attention unit connected in series with the residual extraction unit is used to perform pooling and feature extraction on the image generated by the residual extraction unit to output a pooled attention image.

[0018] Furthermore, the residual extraction unit includes:

[0019] The first residual unit includes a third input terminal, which serves as the residual extraction unit, for extracting and normalizing the input image using a 1*1 convolution kernel.

[0020] The second residual unit, connected in series with the first residual unit, is used to extract and normalize the input image using a 3*3 convolution kernel;

[0021] The first residual summation unit connected in series with the second residual unit includes a fourth input terminal connected to the second residual unit and a fifth input terminal connected to the first residual unit;

[0022] The third residual unit, connected to the first residual summation unit, is used to extract and normalize the input image using a 3*3 convolution kernel; and

[0023] The second residual summing unit, connected to the third residual unit, includes a sixth input terminal connected to the third residual unit and a seventh input terminal connected to the third input terminal. It is used to sum the image input from the sixth input terminal and the graphic input from the seventh input terminal, thereby outputting the generated image to the pooling attention unit.

[0024] Furthermore, the pooling attention unit includes:

[0025] The first pooling unit is used to perform pooling processing on the image input to the residual extraction unit connected to the pooling attention unit, so as to output the first pooled image.

[0026] Multiple block self-attention units are connected in parallel. Each block self-attention unit is connected to the output of the first pooling unit. Each block self-attention unit is used to generate self-attention images with different preset window lengths.

[0027] A second pooling unit connected to the output of each of the segmented self-attention units is used to merge features of self-attention images with different preset window lengths, thereby outputting a second pooled image.

[0028] The first feature product unit is connected at one end to the second pooling unit and at the other end to the output of the second residual summation unit. It is used to generate a pooled attention image by multiplying the second pooling image and the first pooling image.

[0029] Furthermore, the segmented self-attention unit includes:

[0030] The segmentation unit is used to segment the first pooled image output by the first pooling unit into blocks with a preset window length to generate a segmented image containing multiple sub-images.

[0031] A first shape reconstruction unit connected to a block unit is used to resize the image using a first matrix to generate a first shape reconstruction image.

[0032] A second shape reconstruction unit, connected to the block unit and in parallel with the first shape reconstruction unit, is used to resize the image using a second matrix to generate a second shape reconstruction image.

[0033] A self-attention feature merging unit connected to the output of the first shape reconstruction unit and the second shape reconstruction unit is used to perform feature merging based on the first shape reconstruction image and the second shape reconstruction image to generate a self-attention image corresponding to the preset window length.

[0034] Furthermore, the self-attention feature merging unit includes:

[0035] The second feature multiplication unit is connected to the first shape reconstruction unit and the second shape reconstruction unit respectively, and is used to perform matrix multiplication on the first shape reconstruction image and the second shape reconstruction image;

[0036] The third shape reconstruction unit, connected to the second feature product unit, is used to reconstruct the shape of the image output by the second feature product unit through the fourth sub-reconstruction layer;

[0037] The merging unit connected to the third shape reconstruction unit is used to merge features of the image output by the third shape reconstruction unit to generate a self-attention image corresponding to the preset window length.

[0038] Furthermore, the sampling module is any one of the following: a pixel-based upsampling module, a bilinear interpolation sampling module, a bicubic interpolation sampling module, and a transposed convolution sampling module.

[0039] A second aspect of the present invention provides a method for image resolution expansion, the image resolution expansion method comprising:

[0040] Detect the resolution of the original image to be processed;

[0041] Set the preset window length according to the resolution;

[0042] The preset window length and the original image are input into a trained image processing network, wherein the image processing network is any one of the embodiments of the first aspect of the present invention, and

[0043] Output the super-resolution image generated after processing by the image processing network.

[0044] A third aspect of the present invention provides an image processing method based on a deep learning network, the image processing method comprising:

[0045] Obtain the original image to be processed;

[0046] The original image is convolved with a preset first convolution size to output a first image;

[0047] The first image is subjected to residual and pooling attention extraction to output the second image.

[0048] Perform feature summation processing on the second image input from the first input terminal and the first image input from the second input terminal;

[0049] The image output by the first feature summation module is sampled to generate a super-resolution image.

[0050] Output the super-resolution image generated by the sampling module.

[0051] A fourth aspect of the present invention provides a method for image resolution expansion, the image resolution expansion method comprising:

[0052] Detect the resolution of the original image to be processed;

[0053] Set the preset window length according to the resolution;

[0054] The method described in the third aspect of the present invention is executed based on the preset window length and the original image.

[0055] The super-resolution image generated after processing by the image processing method described in the third aspect of the present invention is output.

[0056] A fifth aspect of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the method as described in any one of the fourth aspects of the present invention.

[0057] The sixth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in the second, third, and fourth aspects of the present invention.

[0058] A seventh aspect of the present invention provides a display device comprising a controller, wherein the controller, when executing the program, implements the methods described in the second, third, and fourth aspects of the present invention.

[0059] The beneficial effects of this invention are as follows:

[0060] The image processing network of this invention compresses features through a first convolution module, extracts features through residual processing and pooling attention processing through a feature extraction module, and merges features through a first feature summation module and a sampling module, thereby outputting a super-resolution image with the depth features of the original image. This network has fast image processing speed and good image super-resolution effect. Attached Figure Description

[0061] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0062] Figure 1 This diagram illustrates the network architecture of an image processing network based on a deep learning network according to an embodiment of the present invention.

[0063] Figure 2 This diagram illustrates the network architecture of an image processing network based on a deep learning network according to an embodiment of the present invention.

[0064] Figure 3 A schematic diagram of the residual extraction unit according to an embodiment of the present invention is shown;

[0065] Figure 4 A schematic diagram of the pooling attention unit according to an embodiment of the present invention is shown;

[0066] Figure 5 This diagram illustrates a schematic structural diagram of a block-based self-attention unit according to an embodiment of the present invention.

[0067] Figure 6 This diagram illustrates another schematic structural diagram of a block-based self-attention unit according to an embodiment of the present invention;

[0068] Figure 7 A schematic diagram of the principle structure of the upsampling unit based on pixel arrangement according to an embodiment of the present invention is shown;

[0069] Figure 8 This diagram illustrates the steps of an image resolution expansion method according to an embodiment of the present invention.

[0070] Figure 9 This diagram illustrates the steps of an image processing method based on a deep learning network according to an embodiment of the present invention.

[0071] Figure 10 A schematic diagram of a computer device is shown, illustrating another embodiment of the present invention. Detailed Implementation

[0072] To more clearly illustrate the present invention, the following description, in conjunction with embodiments and accompanying drawings, further explains the invention. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0073] This invention proposes an image processing network, processing method, extension method, apparatus, device, and medium to solve the above-mentioned problems.

[0074] The first embodiment of this invention proposes an image processing network based on deep learning networks, such as... Figure 1 As shown, the image processing network includes:

[0075] Image input terminal 10 is used to acquire the original image to be processed;

[0076] The first convolution module 20 is connected in series with the image input terminal 10 and is used to perform convolution processing on the original image with a preset first convolution size to output the first image.

[0077] Feature extraction module 30, connected in series with the first convolution module 20, is used to perform residual and pooling attention extraction on the first image to output a second image.

[0078] The first feature summation module 40 includes a first input terminal connected to the output terminal of the feature extraction module and a second input terminal connected to the first convolution module 20, and is used to perform feature summation processing on the second image input to the first input terminal and the first image input to the second input terminal;

[0079] The sampling module 50, connected in series with the first feature summation module 40, is used to sample the image output by the first feature summation module 40 to generate a super-resolution image.

[0080] Image output terminal 60 is used to output the super-resolution image generated by the sampling module 50.

[0081] The image processing network of this invention compresses features through a first convolution module 20, extracts features through residual processing and pooling attention processing through a feature extraction module, and merges features through a first feature summation module 40 and a sampling module 50, thereby outputting a super-resolution image with the depth features of the original image. This network has fast image processing speed and good image super-resolution effect.

[0082] For example, if the original image resolution is 960x540, the resolution of the super-resolution image generated by the image processing network of this embodiment can be expanded to 1920x1080. The resolution expansion factor can be designed by those skilled in the art and will not be described in detail here.

[0083] Furthermore, the image processing network of this embodiment can be applied to large terminal devices with high operating capabilities, such as TVs, as well as small terminal devices with low operating capabilities, such as small mobile phones, electronic watches, and other terminal devices with limited processing capabilities. It can reduce the memory size to the KB level and has broad application prospects.

[0084] In a specific example, such as Figure 2 As shown, the first convolutional module 20's convolutional layer conv-3x3 uses a pre-defined first convolutional kernel with a size of 3*3, which can compress the image features of the original image;

[0085] In this embodiment, the height of the original image is H and the width is W. The size of the super-resolution image generated based on the image processing network is H*k and W*k. That is, the size of the super-resolution image is the size of the original image with a preset super-resolution multiple, thereby realizing the design of image super-resolution.

[0086] In an optional embodiment, such as Figure 2 As shown, the feature extraction module 30 includes:

[0087] Multiple cascaded feature extraction units 31, with the feature extraction unit 31 located at one end of the cascaded sequence connected to the first convolution module 20, output a pooling attention image after feature extraction; and

[0088] The second convolutional unit 32 is connected at one end to the feature extraction unit 31 located at the other end of the series connection, and at the other end to the second input end of the first feature summing module 40. It is used to convolve the pooling attention image input to the connected feature extraction unit 31 to output the second image.

[0089] In this embodiment of the invention, the number of feature extraction units 31 is determined according to the operating capability of the device using the image processing network. For example, a smartphone may be equipped with 3 to 5 feature extraction units 31, and a large-size display device may be equipped with more feature extraction units 31. Those skilled in the art can design according to actual applications, and will not elaborate further here.

[0090] Furthermore, in an optional embodiment, such as Figure 2 As shown, the feature extraction unit 31 includes:

[0091] Residual extraction unit 311 is used to extract residuals from the image input to the residual extraction unit 311 to output a residual feature image; and

[0092] The pooling attention unit 312, which is connected in series with the residual extraction unit 311, is used to perform pooling and feature extraction on the image generated by the residual extraction unit 311 to output a pooled attention image.

[0093] The embodiments of the present invention utilize a combined structure of a residual extraction unit DRB (Double Residual Block) and a pooling attention unit PAB (Pooling Attention Block) to effectively extract deep high-frequency features from the original image, ensuring the effect of image resolution expansion.

[0094] In an optional embodiment, such as Figure 3 As shown, the residual extraction unit 311 includes:

[0095] The first residual unit 311a includes a third input terminal, which serves as the residual extraction unit 311, for extracting and normalizing the input image using a 1*1 convolution kernel.

[0096] The second residual unit 311b, which is connected in series with the first residual unit 311a, is used to extract and normalize the input image with a 3*3 convolution kernel;

[0097] The first residual summing unit 311c, which is connected in series with the second residual unit 311b, includes a fourth input terminal connected to the second residual unit 311b and a fifth input terminal connected to the first residual unit 311a.

[0098] The third residual unit 311d, connected to the first residual summing unit 311c, is used to extract and normalize the input image using a 3*3 convolution kernel; and

[0099] The second residual summing unit 311e, which is connected to the third residual unit 311d, includes a sixth input terminal connected to the third residual unit 311d and a seventh input terminal connected to the third input terminal. It is used to sum the image input by the sixth input terminal and the graphic input by the seventh input terminal, thereby outputting the generated output residual feature image to the pooling attention unit 312.

[0100] based on Figure 2 and Figure 3 As shown, the residual extraction unit 311 in this embodiment is a two-layer residual structure, using two 1x1 convolutions and one 3x3 convolution. The 1x1 convolution can effectively reduce the number of model parameters and improve the model inference speed. The 3x3 convolution can obtain the image features of adjacent regions in the image. With the above settings, the residual extraction unit in this embodiment can extract and compress image features, making the overall image processing network faster and improving the image processing effect.

[0101] like Figure 2 and Figure 3 As shown, when there are multiple feature extraction units 31, the previous feature extraction unit 31 and the next feature extraction unit 31 are connected. The feature extraction unit 31 located at the left end of the series connection is connected to the output end of the first convolution module 20 and serves as the input end of the feature extraction module 30. The feature extraction unit 31 located at the right end of the series connection is connected to the first feature summation module 40 and serves as the output end of the feature extraction module 30.

[0102] Therefore, when there are multiple feature extraction units 31, the input of the residual extraction unit 311 in the feature extraction unit 31 located at the left end of the series connection is the second image output by the first convolution module 20. For two adjacent feature extraction units 31, the residual extraction unit 311 of the latter feature extraction unit 31 is the pooled attention image output by the pooling attention unit 312 of the former feature extraction unit 31. In other words, the input of the residual extraction unit 311 at different locations is different, which will not be elaborated further here.

[0103] In a specific example, such as Figure 3 As shown, the residual extraction unit 311 includes:

[0104] The first residual unit 311a includes a first residual convolution unit 311a1 (conv-1*1) with a 1*1 convolution size and a first normalization unit 311a2 (ReLU).

[0105] The second residual unit 311b includes a second residual convolution unit 311b1 (conv-3*3) with a 3*3 convolution size and a second normalization unit 311b2 (ReLU).

[0106] The third residual unit 311d includes a third residual convolution unit 311d1 (conv-1*1) with a convolution size of 1*1 and a third normalization unit 311d2 (Relu), thereby realizing the extraction of depth features of the original image.

[0107] In an optional embodiment, such as Figure 4 As shown, the pooling attention unit 312 includes: a first pooling unit 3121, multiple parallel block self-attention units 3122, a second pooling unit 3123, and a first feature product unit 3124.

[0108] The first pooling unit 3121 is used to perform pooling processing on the residual feature image input to the residual extraction unit 311 connected to the pooling attention unit 312, so as to output a first pooled image. For example, as shown... Figure 3 and Figure 4 As shown, the first pooling unit 3121 is connected to the second residual summing unit 311e of the residual extraction unit 311. In this embodiment, the first pooling unit 3121 is a first average pooling layer or a max pooling layer, which can compress the input multi-channel residual feature image features into single-channel image features, thereby greatly reducing the amount of computation and inference time and improving processing efficiency.

[0109] Each of the multiple parallel block self-attention units 3122 is connected to the output of the first pooling unit 3121, and each block self-attention unit 3122 is used to generate self-attention images with different preset window lengths. Figure 4 As shown, in this embodiment, three parallel block self-attention units 3122 are set. Each block self-attention unit 3122 has a different preset window length, and the number of blocks in the generated self-attention image is different. For example, the block self-attention unit 3122 at the top has 4 blocks and a window length of 2; the block self-attention unit 3122 in the middle has 16 blocks and a window length of 4; and the block self-attention unit 3122 at the bottom has 64 blocks and a window length of 8. Through this setting, the image features can be divided into several different small sizes, making great use of the local features of the image and ensuring the accuracy of image processing.

[0110] The second pooling unit 3123 is connected to the output of each of the block-based self-attention units 3122, and is used to merge the features of self-attention images with different preset window lengths to output a second pooled image. In this embodiment of the invention, the second pooling unit 3123 merges the image features of three self-attention images into a single feature layer. For example, the second pooling unit 3123 can use an average pooling layer or a max pooling layer.

[0111] One end of the first feature product unit 3124 is connected to the second pooling unit 3123, and the other end is connected to the output of the second residual summation unit 311e. It is used to generate a pooled attention image by multiplying the second pooled image and the first pooled image. In this embodiment, the product of the first feature product unit 3124 is a dot product, which involves multiplying the image data at the same position in the first and second pooled images.

[0112] In an optional embodiment, the block self-attention unit 3122 determines the preset window length of each block self-attention unit 3122 based on the resolution of the original image. In a specific example, when the resolution of the original image X to be processed is less than or equal to 960x540, the preset window length M is set to M = 8, 10, 12, and self-attention images corresponding to different preset window lengths are generated by the three block self-attention units 3122 respectively; when the resolution of the original image X to be processed is greater than 960x540 and less than or equal to 1920x1080, the preset window length M is set to the window lengths of the three blocks M = 10, 12, 14; when the resolution of the original image X to be processed is greater than 1920x1080, the preset window length M is set to the window lengths of the three blocks M = 12, 14, 16. Through this setting, the preset window length can be set according to the original image with different input data sizes to generate a super-resolution image with a larger resolution, thereby ensuring the image processing effect while magnifying the original image.

[0113] It is worth noting that the specific value of the preset window length in the embodiments of the present invention is not limited, and those skilled in the art can design it according to actual applications, which will not be elaborated here.

[0114] In an optional embodiment, such as Figure 5 and Figure 6 As shown, the segmented self-attention unit 3122 includes:

[0115] The segmentation unit 3122a is used to segment the first pooled image output by the first pooling unit 3121 into blocks with a preset window length to generate a segmented image containing multiple sub-images.

[0116] The first shape reconstruction unit 3122b, connected to the block unit 3122a, is used to resize the image using a first matrix to generate a first shape reconstruction image.

[0117] The second shape reconstruction unit 3122c, connected to the block unit 3122a and in parallel with the first shape reconstruction unit 3122b, is used to resize the image using a second matrix to generate a second shape reconstruction image.

[0118] The self-attention feature merging unit 3122d, which is connected to the output of the first shape reconstruction unit 3122b and the second shape reconstruction unit 3122c, is used to perform feature merging based on the first shape reconstruction image and the second shape reconstruction image to generate a self-attention image corresponding to the preset window length.

[0119] In this embodiment, as Figure 5 and Figure 6 The diagram shows a network architecture for a block-based self-attention unit 3122, namely... Figure 4 One of the parallel block self-attention units 3122, Figure 5 and Figure 6 The input of the block self-attention unit 3122 shown is the first pooling image output by the first pooling unit 3121, and the self-attention image output by it is used as the input of the second pooling unit 3123.

[0120] In this embodiment, the input of the block unit 3122a is the first pooling image. After pooling by the first pooling unit 3121, the first pooling unit 3121 is a single-channel image. The size of the first pooling image F of the single-channel image features is B×1×H×W, where B is the batch size, which is the batch value or batch size. The B×1×H×W of the first pooling image F means that B batch values ​​of 1 and size of H×W are input. H and W are the height and width of the first pooling image F, respectively.

[0121] Segmentation unit 3122a divides the first pooling image F into N M×M sub-images F' with a preset window length M, wherein the size of the segmented sub-images F' is B×M. 2 ×h×w, where h=H / / M, w=W / / M.

[0122] Furthermore, considering that the size of the sub-image F' is enlarged after rounding up, making it impossible to stitch it back into the first pooled image F before splitting, in an optional embodiment, if the height H of the first pooled image cannot be divided evenly by the length of the preset window, the height of each sub-image after segmentation is rounded up. That is, if h = H / M is 3.1, which cannot be divided evenly, then h is taken as an integer 4, or the height H of the first pooled image is expanded to an integer multiple of the divisibility value of the window length M after rounding up. For example, the H of the original first pooled image F is expanded to 4M to achieve divisibility.

[0123] Furthermore, the data of the sub-image at the height corresponding to the extended position is filled with preset data. For example, for the sub-image with no image data in the length range of 3.1 to 4, the image data has a grayscale value of 0. Alternatively, the data of the first pooled image at the height corresponding to the extended position is filled with preset data. For example, the H of the first pooled image is extended to 16. If the original size of the first pooled image is 12.4, then the image data in the height direction from 12.4 to 16 is filled with a grayscale value of 0 to achieve the completion setting of the sub-image or the first pooled image.

[0124] Similarly, the same design is used for the width of the first pooling image or the width of each sub-image after the first pooling image is divided. In an optional embodiment, if the width W of the first pooling image cannot be divided evenly by the length of the preset window, the width of each sub-image after being divided is rounded up, or the width W of the first pooling image is expanded to an integer multiple of the integer divisibility value of the window length M after rounding up.

[0125] Furthermore, the data of the sub-image at the width corresponding to the extended position is filled with preset data. For example, for the sub-image with a width of 3.1 to 4, which has no image data, it is image data with a grayscale value of 0. Alternatively, the data of the first pooling image at the width corresponding to the extended position is filled with preset data. For example, the H of the first pooling image is extended to 16. If the original size of the first pooling image is 12.4, then the image data in the width direction from 12.4 to 16 is filled with a grayscale value of 0 to achieve the completion setting of the sub-image or the first pooling image.

[0126] After being divided into blocks by the segmentation unit 3122a, multiple blocks are obtained. The first shape reconstruction unit 3122b and the second shape reconstruction unit 3122c are used to reconstruct the shape of the block image to extract the relationship between image features at different locations in the block image, thereby extracting the depth features of the block image and improving the image processing accuracy.

[0127] In an optional embodiment, such as Figure 6 As shown,

[0128] The first shape reconstruction unit 3122b includes:

[0129] The first convolutional normalization layer 610 includes a 1×1 convolutional layer 611 and a normalization layer 612 (BatchNorm), and

[0130] The first sub-reconstruction layer 620 (Reshape1) and the second sub-reconstruction layer 630 (Reshape2) are connected in parallel. The inputs of both the first sub-reconstruction layer 620 and the second sub-reconstruction layer 630 are connected to the first convolutional normalization layer 610, and they reshape the output of the first convolutional normalization layer 610 in different ways; and

[0131] The third feature product unit 640 is used to multiply the image output by the first sub-reconstruction layer 620 and the image output by the second sub-reconstruction layer 630 to output the first shape reconstruction image.

[0132] The second shape reconstruction unit 3122c includes:

[0133] The second convolutional normalization layer 650 includes a 1×1 convolutional layer 651 and a normalization layer (BatchNorm) 652, and

[0134] The third sub-reconstruction layer 660 (Reshape3), which is connected to the second convolutional normalization layer 650, performs shape reconstruction on the image output by the second convolutional normalization layer 650 in the same reconstruction method as the first sub-reconstruction layer 620.

[0135] In a specific example, the input to the first shape reconstruction unit 3122b is a block image, the size of which is B×M. 2 ×h×w, the size output after the first sub-reconstruction layer 620 is B×h×M. 2 ×w, the size output after the second sub-reconstruction layer 630 is B×h×w×M. 2 The size of the output after the third feature product unit is B×h×M 2 ×M 2 Therefore, the size of the first shape reconstruction image output by the first shape reconstruction unit 3122b is B×h×M. 2 ×M 2 .

[0136] Furthermore, the input to the second shape reconstruction unit 3122c is a block image, the size of which is also B×M. 2 ×h×w, the size output by the third sub-reconstruction layer 660 is B×h×M. 2 Therefore, the output of the second shape reconstruction unit 3122c is B×h×M. 2 ×w.

[0137] By processing the block image through the first shape reconstruction unit 3122b and the second shape reconstruction unit 3122c, the correlation between image features at a certain location in the block image and image features at other locations can be extracted, thereby improving the image processing accuracy.

[0138] Furthermore, in the image processing network, the first shape reconstruction image and the second shape reconstruction image are input to the self-attention feature merging unit 3122d for feature merging. In an optional embodiment, such as... Figure 6 As shown, the self-attention feature merging unit 3122d includes a second feature product unit 670, a third shape reconstruction unit 680, and a merging unit 690.

[0139] like Figure 6 As shown, the second feature product unit 670 is connected to both the first shape reconstruction unit 3122b and the second shape reconstruction unit 3122c, and is used to perform matrix multiplication on the first shape reconstruction image and the second shape reconstruction image. In this embodiment, in a specific example, the matrix multiplication performed by the second feature product unit 670 is the overall multiplication of the two reconstructed images. After matrix multiplication, the size of the image output by the second feature product unit 670 is B×h×M. 2 ×w.

[0140] The third shape reconstruction unit 680, connected to the second feature product unit 670, is used to reconstruct the shape of the image output by the second feature product unit 670 through the fourth sub-reconstruction layer (Reshape4), and the size of the output image is B×M. 2 ×h×w, that is, after the above shape reconstruction processing, the batch value, quantity, height and width of the image output by the third shape reconstruction unit 680 have not changed compared with the input block image, but the feature association in the image has been extracted.

[0141] The merging unit 690 is connected to the third shape reconstruction unit 680 and is used to merge features of the image output by the third shape reconstruction unit 680 to generate a self-attention image corresponding to the preset window length. In this embodiment, the size of the self-attention image is the same as the size of the input first pooling image, both being B×1×h×w. That is, the merging unit 690 merges the M×M smaller block images output by the previous third shape reconstruction unit into a single self-attention image of the complete size.

[0142] In a specific example, the principle of the block operation performed by block unit 3122a is as follows: Figure 6As shown, taking a block size of N=4 as an example, the first pooling image F is divided into 4 sub-images, each with a size of B×1×h×w, i.e., each sub-image is B×1×H / 2×W / 2. The resulting 4 small sub-images are arranged and combined in order to obtain a block image F' containing multiple sub-images. The image feature of the block image F' is of size B×4×H / 2×W / 2, see... Figure 6 The image is divided into blocks, F'.

[0143] It is worth noting that this embodiment does not limit the arrangement order of sub-images to form block images. For example, the arrangement can be set from left to right and from top to bottom in the cross grid. Those skilled in the art can design it according to actual application.

[0144] Based on the block division principle of the block division unit 3122a in the aforementioned embodiment, the merging unit 690 merges the image output by the third shape reconstruction unit 680. The merging rule is the reverse of the arrangement rule of the block division unit 3122a, merging the four sub-image blocks into a single self-attention image, i.e. Figure 6 As shown, after the sub-image in the upper left corner of the image is processed, it is still merged into the upper left corner of the self-attention image. After the sub-image in the upper right corner of the image is processed, it is still merged into the upper right corner of the self-attention image, thus ensuring the consistency of image features.

[0145] In an optional embodiment, the sampling module 50 is any one of a pixel-based upsampling module (PiexlShuffle), a bilinear interpolation sampling module, a bicubic interpolation sampling module, and a transposed convolution sampling module.

[0146] In this embodiment, the sampling module 50 is used to reconstruct the final image, thereby expanding the resolution of the original image. In a specific example, the sampling module 50 is taken as a pixel-based upsampling module. Figure 2 As shown, the sampling module 50 includes a 3*3 sampling convolutional layer 51 and a pixel-arranged upsampling (PiexlShuffle) unit 52 connected in series with the sampling convolutional layer 51.

[0147] In a specific example, the principle of the upsampling unit 52 is as follows: Figure 7 As shown, the output of the sampling convolutional layer is a multi-channel image, and the image features of each channel are as follows: Figure 7 As shown on the left, there are four channels: a, b, c, and d. The image data in each channel is... Figure 7For example, the sampling unit uses the data selector Mux to sequentially select image data at the same location in different channel images according to channel order, and arranges them according to the arrangement rules. Image data at the same location in multiple different channel images are located at the same location in the generated super-resolution image.

[0148] In a specific example, the top-left corners a11, b11, c11, and d11 of the first, second, third, and fourth channel images are all located at the top-left corner of the super-resolution image. The top-right corners a12, b12, c12, and d12 of the first, second, third, and fourth channel images are all located at the top-right corner of the super-resolution image, thus forming the entire top-right corner design. Through this super-sampling design, the resolution of the resulting super-resolution image is improved.

[0149] like Figure 8 As shown, another embodiment of the present invention proposes a method for image resolution expansion, the image resolution expansion method comprising:

[0150] S81. Detect the resolution of the original image to be processed;

[0151] S82. Detect the preset window length set according to the resolution;

[0152] S83. The preset window length and the original image are input into a trained image processing network, wherein the image processing network is the image processing network of the above embodiments of the present invention, and

[0153] S84. Detect and output the super-resolution image generated after processing by the image processing network.

[0154] The image resolution expansion method of this invention will now be described with specific examples:

[0155] S81. Detect the resolution of the original image to be processed. For example, the original image to be processed may be a video stream or an image frame.

[0156] S82. Detect the preset window length set according to the resolution;

[0157] In this step, five different window lengths (M = 8, 10, 12, 14, 16) can be preset in practical applications, based on the resolution of the video stream or image frame to be processed on the current application device. When the resolution of the original image X to be processed is less than or equal to 960x540, the preset window length M is set to M = 8, 10, 12; when the resolution of the original image X to be processed is greater than 960x540 but less than or equal to 1920x1080, the preset window length M is set to M = 10, 12, 14 for these three blocks; when the resolution of the original image X to be processed is greater than 1920x1080, the preset window length M is set to M = 12, 14, 16 for these three blocks. Through this setting, the preset window length can be set according to the original image with different input data sizes, thereby achieving adaptive selection of the image processing network for image resolution expansion based on the resolution of the video source, in order to generate a higher resolution super-resolution image, and ensuring image processing effect while expanding the original image.

[0158] S83. The preset window length and the original image are input into a trained image processing network, wherein the image processing network is the image processing network of the above embodiments of the present invention, and

[0159] S84. Output the super-resolution image generated after processing by the image processing network.

[0160] Based on the above method, the embodiments of the present invention can adjust the preset window length in real time according to the resolution of the original image to be processed, so as to achieve the best display effect after the image is expanded.

[0161] Another embodiment of the present invention proposes an image processing method based on deep learning networks, such as... Figure 9 As shown, the image processing method includes:

[0162] S91. Obtain the original image to be processed;

[0163] S92. Perform convolution processing on the original image with a preset first convolution size to output a first image;

[0164] S93. Perform residual and pooling attention extraction on the first image to output the second image.

[0165] S94. Perform feature summation processing on the second image input from the first input terminal and the first image input from the second input terminal;

[0166] S95. The image output by the first feature summation module 40 is sampled to generate a super-resolution image;

[0167] S96. Output the super-resolution image generated by the sampling module.

[0168] The image processing method based on the embodiments of the present invention achieves feature compression through a first convolution module, feature extraction through residual processing and pooling attention processing through a feature extraction module, and feature merging through a first feature summation module and a sampling module, thereby outputting a processed super-resolution image with the depth features of the original image. This network has fast image processing speed and good image super-resolution effect.

[0169] The method in this embodiment is similar to the principle of the aforementioned image processing network, and will not be described again here.

[0170] Another embodiment of the present invention provides a method for image resolution expansion, the image resolution expansion method comprising:

[0171] Detect the resolution of the original image to be processed;

[0172] Set the preset window length according to the resolution;

[0173] Based on the preset window length and based on the original image, the following is executed. Figure 9 The method described,

[0174] Output Figure 9 The super-resolution image generated after processing by the image processing method described above.

[0175] Based on the above method, the embodiments of the present invention can adjust the preset window length in real time according to the resolution of the original image to be processed, so as to achieve the best display effect after the image is expanded.

[0176] Another embodiment of the present invention provides a display device including a controller. When the controller executes the program, it implements the image resolution expansion method based on image processing networks, the image processing method based on deep learning networks, and the image resolution expansion method based on image processing methods as described above. The display device can be any product or component with display function, such as electronic paper, mobile phone, tablet computer, television, monitor, laptop computer, digital photo frame, or navigator. This embodiment does not limit this.

[0177] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements: the aforementioned method for image resolution expansion based on image processing networks, the image processing method based on deep learning networks, and the method for image resolution expansion based on image processing methods.

[0178] In practical applications, the computer-readable storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0179] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0180] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0181] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0182] like Figure 10As shown, another embodiment of the present invention provides a structural schematic diagram of a computer device. Figure 10 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0183] like Figure 10 As shown, the computer device 12 is presented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 78, and bus 18 connecting different system components (including system memory 78 and processing unit 16).

[0184] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0185] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0186] System memory 78 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 80 and / or cache memory 782. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 84 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 78 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0187] A program / utility 90 having a set (at least one) of program modules 92 may be stored, for example, in memory 78. Such program modules 92 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 92 typically perform the functions and / or methods described in the embodiments of the present invention.

[0188] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 74, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 72. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 70. Figure 10 As shown, network adapter 70 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 10 As not shown, it can be used in conjunction with computer device 12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0189] The processor unit 16 executes various functional applications and data processing by running programs stored in the system memory 78, such as implementing the aforementioned image resolution expansion method based on image processing networks, image processing method based on deep learning networks, and image resolution expansion method based on image processing methods.

[0190] In the description of this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0191] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. An image processing network based on deep learning networks, characterized in that, The image processing network includes: Image input terminal, used to acquire the raw image to be processed; The first convolution module is connected in series with the image input terminal and is used to perform convolution processing on the original image with a preset first convolution size to output the first image; The feature extraction module, connected in series with the first convolution module, is used to perform residual and pooling attention extraction on the first image to output a second image. The first feature summation module includes a first input terminal connected to the output terminal of the feature extraction module and a second input terminal connected to the first convolution module, and is used to perform feature summation processing on the second image input to the first input terminal and the first image input to the second input terminal; The sampling module, connected in series with the first feature summation module, is used to sample the image output by the first feature summation module to generate a super-resolution image. The image output terminal is used to output the super-resolution image generated by the sampling module; The feature extraction module includes: Multiple cascaded feature extraction units, with the feature extraction unit located at one end of the cascaded sequence connected to the first convolutional module, output a pooled attention image after feature extraction; and The second convolutional unit is connected at one end to the feature extraction unit located at the other end of the series connection, and at the other end to the second input end of the first feature summing module. It is used to convolve the pooling attention image input to the connected feature extraction unit to output the second image. The feature extraction unit includes: A residual extraction unit is configured to extract residuals from an image input to the residual extraction unit; and A pooling attention unit connected in series with the residual extraction unit is used to perform pooling and feature extraction on the image generated by the residual extraction unit to output a pooled attention image. The residual extraction unit includes: The first residual unit includes a third input terminal, which serves as the residual extraction unit, for extracting and normalizing the input image using a 1*1 convolution kernel. The second residual unit, connected in series with the first residual unit, is used to extract and normalize the input image using a 3*3 convolution kernel; The first residual summation unit connected in series with the second residual unit includes a fourth input terminal connected to the second residual unit and a fifth input terminal connected to the first residual unit; The third residual unit, connected to the first residual summation unit, is used to extract and normalize the input image using a 3*3 convolution kernel; and The second residual summing unit, connected to the third residual unit, includes a sixth input terminal connected to the third residual unit and a seventh input terminal connected to the third input terminal. It is used to sum the image input from the sixth input terminal and the graphic input from the seventh input terminal, thereby outputting the generated image to the pooling attention unit.

2. The image processing network according to claim 1, characterized in that, The pooling attention unit includes: The first pooling unit is used to perform pooling processing on the image input to the residual extraction unit connected to the pooling attention unit, so as to output the first pooled image. Multiple block self-attention units are connected in parallel. Each block self-attention unit is connected to the output of the first pooling unit. Each block self-attention unit is used to generate self-attention images with different preset window lengths. A second pooling unit connected to the output of each of the segmented self-attention units is used to merge features of self-attention images with different preset window lengths, thereby outputting a second pooled image. The first feature product unit is connected at one end to the second pooling unit and at the other end to the output of the second residual summation unit. It is used to generate a pooled attention image by multiplying the second pooling image and the first pooling image.

3. The image processing network according to claim 2, characterized in that, The segmented self-attention unit includes: The segmentation unit is used to segment the first pooled image output by the first pooling unit into blocks with a preset window length to generate a segmented image containing multiple sub-images. A first shape reconstruction unit connected to a block unit is used to resize the image using a first matrix to generate a first shape reconstruction image. A second shape reconstruction unit, connected to the block unit and in parallel with the first shape reconstruction unit, is used to resize the image using a second matrix to generate a second shape reconstruction image. A self-attention feature merging unit connected to the outputs of the first shape reconstruction unit and the second shape reconstruction unit is used to perform feature merging based on the first shape reconstruction image and the second shape reconstruction image to generate a self-attention image corresponding to the preset window length.

4. The image processing network according to claim 3, characterized in that, The self-attention feature merging unit includes: The second feature multiplication unit is connected to the first shape reconstruction unit and the second shape reconstruction unit respectively, and is used to perform matrix multiplication on the first shape reconstruction image and the second shape reconstruction image; The third shape reconstruction unit, connected to the second feature product unit, is used to reconstruct the shape of the image output by the second feature product unit through the fourth sub-reconstruction layer; The merging unit connected to the third shape reconstruction unit is used to merge features of the image output by the third shape reconstruction unit to generate a self-attention image corresponding to the preset window length.

5. The image processing network according to claim 1, characterized in that, The sampling module is any one of the following: pixel-based upsampling module, bilinear interpolation sampling module, bicubic interpolation sampling module, and transposed convolution sampling module.

6. A method for image resolution expansion, characterized in that, The image resolution expansion method includes: Detect the resolution of the original image to be processed; Set the preset window length according to the resolution; The preset window length and the original image are input into a trained image processing network, wherein the image processing network is any one of claims 1 to 5, and Output the super-resolution image generated after processing by the image processing network.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in claim 6.

9. A display device, comprising a controller, characterized in that, The controller executes the program to implement the method as described in claim 6.

Images