A method, system, terminal and storage medium for image super-resolution processing

Abstract

This invention discloses a method, system, terminal, and storage medium for image super-resolution processing. The method includes: acquiring an original image, its pixel values, and the size of available video memory; dividing the original image into blocks to obtain the number of blocks, the pixel values ​​of the blocks, and the step size pixel values ​​between adjacent blocks, wherein the step size pixel values ​​are less than the pixel values ​​of the blocks; performing padding on the original image to obtain a padded image and its pixel values; creating a blank image; sequentially cropping multiple first screenshots from the padded image, wherein adjacent first screenshots have overlapping pixels; performing model inference on all first screenshots to obtain multiple second screenshots; removing padding from each of the multiple second screenshots to obtain multiple third screenshots; and sequentially pasting the multiple third screenshots into the blank image to obtain the target image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method, system, terminal, and storage medium for image super-resolution processing. Background Technology

[0002] Super-resolution reconstruction algorithms process low-resolution images to obtain high-resolution images, meeting the huge demand for high-quality images. Although the performance of models has improved with the introduction of deep learning into the field of image super-resolution, the following problems still exist: large number of model parameters, high computational resource requirements, and most deep learning models focus only on performance, making them difficult to implement in practical applications.

[0003] To enable practical applications, people have begun to lightweight models. However, typical methods for model lightweighting rely solely on reducing the number of network layers, which leads to a decline in the performance of the lightweight model, making it difficult to meet requirements.

[0004] Therefore, it is necessary to dynamically segment the input image to limit computing resources, meet the needs of most hardware application models, and improve the applicability of deep learning-based models. Summary of the Invention

[0005] The main objective of this invention is to provide an image super-resolution processing method that addresses the problems of large model parameter count and high computational resource requirements in existing image super-resolution processing methods.

[0006] The first aspect of this invention provides a method for image super-resolution processing, characterized by comprising the following steps:

[0007] Get the original image, the pixel values ​​of the original image, and the size of the available video memory;

[0008] The original image is divided into blocks based on the pixel values ​​of the original image, the size of the available video memory, the preset magnification factor, and the preset overlap pixel value to obtain the number of region blocks, the pixel values ​​of the region blocks, and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block.

[0009] The original image is filled according to the number of region blocks and the step size pixel value to obtain the filled image and the pixel values ​​of the filled image;

[0010] A blank image is created based on the pixel values ​​of the filled image and the preset overlapping pixel values;

[0011] Multiple first screenshots are sequentially cropped from the filled image based on the pixel values ​​of the region block and the step size pixel values, wherein there are overlapping pixels between two adjacent first screenshots.

[0012] Based on the preset magnification factor, all the first screenshots are subjected to model inference to obtain multiple second screenshots;

[0013] Based on the preset magnification factor and the preset overlapping pixel value, the multiple second screenshots are respectively subjected to defilling processing to obtain multiple third screenshots;

[0014] Multiple third screenshots are pasted sequentially into the blank image to obtain the target image.

[0015] In an optional embodiment of the first aspect of the present invention, the step of dividing the original image into blocks based on the pixel values ​​of the original image, the size of the available video memory, a preset magnification factor, and a preset overlap pixel value to obtain the number of region blocks, the pixel values ​​of the region blocks, and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block, includes the following steps:

[0016] Calculate the required video memory size, which is equal to the original image width pixel value multiplied by the original image height pixel value multiplied by the preset magnification factor multiplied by the preset magnification factor multiplied by 2 divided by 1024;

[0017] Calculate the scaling factor, which is equal to the required video memory size divided by the available video memory size, and then rounded up to the nearest integer.

[0018] Calculate the first quantity and the second quantity respectively. The first quantity is equal to the square root of the proportional coefficient. If the first quantity is not an integer, round it down. The second quantity is equal to the proportional coefficient divided by the first quantity. If the second quantity is not an integer, add 1 to the first quantity and continue to calculate the second quantity until the calculated second quantity is an integer. Then, substitute the value of the first quantity back into the first quantity.

[0019] The number of region blocks is calculated, and the number of region blocks is equal to the number in the width direction multiplied by the number in the height direction. When the original image width pixel value is greater than the original image height pixel value, the number in the width direction is equal to the first number, and the number in the height direction is equal to the second number. When the original image width pixel value is less than the original image height pixel value, the number in the width direction is equal to the second number, and the number in the height direction is equal to the first number.

[0020] The step size pixel value is calculated, which includes a step size width pixel value and a step size height pixel value. The step size width pixel value is equal to the original image width pixel value divided by the number of pixels in the width direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4. The step size height pixel value is equal to the original image height pixel value divided by the number of pixels in the height direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4.

[0021] Calculate the pixel value of the region block, wherein the width pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size width pixel value, and the height pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size height pixel value.

[0022] In an optional embodiment of the first aspect of the present invention, in the step of filling the original image according to the number of region blocks and the step size pixel value to obtain a filled image and the pixel values ​​of the filled image;

[0023] The four sides of the original image are filled respectively. The width pixel value of the filled image is equal to the preset overlap pixel value multiplied by 2 plus the width pixel value of the region block multiplied by the number of pixels in the width direction. The height pixel value of the filled image is equal to the preset overlap pixel value multiplied by 2 plus the height pixel value of the region block multiplied by the number of pixels in the height direction.

[0024] In an optional embodiment of the first aspect of the present invention, creating a blank image based on the pixel values ​​of the filled image and the preset overlapping pixel values ​​includes the following steps:

[0025] Calculate the pixel values ​​of the blank image, where the pixel values ​​of the blank image are equal to the pixel values ​​of the filled image multiplied by a preset magnification factor;

[0026] Create a blank image based on the pixel values ​​of the blank image.

[0027] In an optional embodiment of the first aspect of the present invention, a plurality of first screenshots are sequentially cropped on the filled image according to the pixel value of the region block and the step size pixel value, wherein there are overlapping pixels between two adjacent first screenshots, including the following steps:

[0028] A first screenshot is taken from the filled image, wherein the first screenshot is located at one corner of the filled image, and the pixel value of the first screenshot is equal to the pixel value of the region block;

[0029] Based on the position of the first screenshot, shift the step width pixel value along the width direction to take a screenshot. Repeat this step until the number of first screenshots taken in the width direction is the same as the number in the width direction.

[0030] Based on the positions of all the first screenshots in the previous row, take a screenshot by shifting the step height pixel value along the height direction. Repeat this step until the number of first screenshots taken in the height direction is the same as the number taken in the height direction.

[0031] In an optional embodiment of the first aspect of the present invention, in the step of performing model inference on all the first screenshots according to the preset magnification factor to obtain a plurality of second screenshots,

[0032] The pixel value of the second screenshot is equal to the pixel value of the first screenshot multiplied by a preset magnification factor.

[0033] In an optional embodiment of the first aspect of the present invention, in the step of performing fill removal processing on a plurality of second screenshots according to the preset magnification factor and the preset overlapping pixel value to obtain a plurality of third screenshots,

[0034] The fill removal process is performed on all four sides of the second screenshot, wherein the pixel value removed from each side is equal to the preset magnification factor multiplied by the preset overlap pixel value.

[0035] A second aspect of the present invention provides a system for image super-resolution processing, comprising:

[0036] The acquisition module is used to acquire the original image, the pixel values ​​of the original image, and the size of the available video memory;

[0037] The calculation module is used to perform block calculation on the original image based on the pixel value of the original image, the size of the available video memory, the preset magnification factor and the preset overlapping pixel value, to obtain the number of region blocks, the pixel value of the region block and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block.

[0038] A filling module is used to fill the original image according to the number of region blocks and the step size pixel value to obtain a filled image and the pixel values ​​of the filled image.

[0039] A creation module is used to create a blank image based on the pixel values ​​of the filled image and the preset overlapping pixel values;

[0040] The cropping module is used to sequentially crop multiple first screenshots on the filled image according to the pixel value of the region block and the step size pixel value, wherein there are overlapping pixels between two adjacent first screenshots;

[0041] The model inference module is used to perform model inference on all the first screenshots according to the preset magnification factor to obtain multiple second screenshots;

[0042] The fill removal module is used to perform fill removal processing on multiple second screenshots according to the preset magnification factor and the preset overlapping pixel value to obtain multiple third screenshots;

[0043] The pasting module is used to paste multiple third screenshots sequentially into the blank image to obtain the target image.

[0044] A third aspect of the present invention provides a terminal for image super-resolution processing, the terminal comprising: a memory and at least one processor, the memory storing instructions, and the memory and the at least one processor being interconnected via a circuit;

[0045] The at least one processor invokes the instructions in the memory to cause the terminal to perform the image super-resolution processing method as described in any of the preceding claims.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image super-resolution processing method as described in any of the preceding claims.

[0047] Beneficial Effects: This invention provides a method, system, terminal, and storage medium for image super-resolution processing. The method includes acquiring an original image, its pixel values, and the available video memory size; performing block calculations on the original image; performing fill processing on the original image to obtain a filled image and its pixel values; creating a blank image; sequentially cropping multiple first screenshots from the filled image, wherein adjacent first screenshots have overlapping pixels; performing model inference on all first screenshots to obtain multiple second screenshots; performing fill removal processing on the multiple second screenshots to obtain multiple third screenshots; and sequentially pasting the multiple third screenshots into the blank image to obtain a target image. By performing block calculations on the original image, the original image is divided into multiple regions, and then model inference is performed on each region. The smaller the available video memory size, the more regions are divided, and the smaller the size of each region, so that the video memory required for model inference on the region is less than the available video memory size. Therefore, regardless of how small the available video memory size is, image super-resolution processing can be performed. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of an embodiment of an image super-resolution processing method according to the present invention.

[0050] Figure 2 This is a schematic diagram of an embodiment of an image super-resolution processing system according to the present invention.

[0051] Figure 3 This is a schematic diagram of an embodiment of an image super-resolution processing terminal according to the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0054] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0055] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0056] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0057] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0058] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first aspect of the present invention provides a method for image super-resolution processing, comprising the following steps:

[0059] S100: Obtain the original image, the pixel values ​​of the original image, and the size of the available video memory;

[0060] S200: The original image is divided into blocks according to the pixel values ​​of the original image, the size of the available video memory, the preset magnification factor and the preset overlapping pixel value to obtain the number of region blocks, the pixel values ​​of the region blocks and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block.

[0061] S300: The original image is filled according to the number of region blocks and the step size pixel value to obtain the filled image and the pixel values ​​of the filled image;

[0062] S400. Create a blank image based on the pixel values ​​of the filled image and the preset overlapping pixel values;

[0063] S500: Based on the pixel value of the region block and the step size pixel value, a plurality of first screenshots are sequentially cropped on the filled image, wherein there are overlapping pixels between two adjacent first screenshots.

[0064] S600: Perform model inference on all the first screenshots according to the preset magnification factor to obtain multiple second screenshots;

[0065] S700: Perform fill removal processing on the multiple second screenshots according to the preset magnification factor and the preset overlapping pixel value to obtain multiple third screenshots;

[0066] S800: Paste multiple third screenshots sequentially into the blank image to obtain the target image.

[0067] In this embodiment, since the available video memory is small, it is impossible to perform model inference on the original image. Therefore, the original image needs to be divided into blocks. By performing block calculation on the original image, the original image is split into multiple region blocks. Then, model inference is performed on each region block. The smaller the available video memory, the more region blocks are split. At the same time, the size of each region block is also smaller, so that the video memory required for model inference on the region block is smaller than the available video memory. Therefore, no matter how small the available video memory is, super-resolution processing of the image can be performed. Since the step size pixel value is smaller than the pixel value of the region block, there will be overlapping parts between two adjacent region blocks. The purpose is to prevent obvious dividing lines between two adjacent region blocks when the third screenshot is pasted into the blank image.

[0068] In steps S500-S800, the method employed is to first capture all first screenshots, then process all the first screenshots uniformly, and finally paste them together into a blank image. In other embodiments, it can also be that each first screenshot is captured, processed, and then pasted into the blank image. In step S800, the multiple third screenshots are pasted into the blank image in the order in which the first screenshots were captured in step S500, with adjacent third screenshots placed close together. Furthermore, if there are blank areas in the target image, these blank areas are captured and deleted.

[0069] In an optional embodiment of the first aspect of the present invention, the step of dividing the original image into blocks based on the pixel values ​​of the original image, the size of the available video memory, a preset magnification factor, and a preset overlap pixel value to obtain the number of region blocks, the pixel values ​​of the region blocks, and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block, includes the following steps:

[0070] The required video memory size is calculated as follows: The required video memory size is equal to the original image width in pixels multiplied by the original image height in pixels multiplied by the preset magnification factor multiplied by 2 divided by 1024. The calculation method for the required video memory size is shown below:

[0071] required_mem=w*h*mul*mul*2 / 1024;

[0072] In the formula: required_mem represents the required video memory size, w represents the original image width in pixels, h represents the original image height in pixels, and mul represents the preset magnification factor.

[0073] Calculate the scaling factor, which is equal to the required video memory size divided by the available video memory size, then rounded up to the nearest integer. The formula for calculating the scaling factor is as follows:

[0074] n=ceil(required_mem / available_mem);

[0075] In the formula, n represents the scaling factor, ceil represents rounding up to the nearest integer, required_mem represents the required video memory size, and available_mem represents the available video memory size;

[0076] Calculate the first quantity and the second quantity separately. The first quantity is equal to the square root of the proportionality coefficient. If the first quantity is not an integer, round it down. The second quantity is equal to the proportionality coefficient divided by the first quantity. If the second quantity is not an integer, add 1 to the first quantity and continue calculating the second quantity until the calculated second quantity is an integer. Then, substitute the value of the first quantity back into the first quantity. The formulas for calculating the first quantity and the second quantity are as follows:

[0077]

[0078] In the formula, f1 represents the first quantity, f0 represents the second quantity, floor represents rounding down to the nearest integer, sqrt represents taking the square root, and n represents the proportionality coefficient;

[0079] The number of region blocks is calculated, and the number of region blocks is equal to the number in the width direction multiplied by the number in the height direction. Specifically, when the original image width pixel value is greater than the original image height pixel value, the number in the width direction equals the first number, and the number in the height direction equals the second number. When the original image width pixel value is less than the original image height pixel value, the number in the width direction equals the second number, and the number in the height direction equals the first number. The formula for calculating the number of region blocks is as follows:

[0080]

[0081] In the formula, N represents the number of regions, w_n represents the number in the width direction, h_n represents the number in the height direction, f1 represents the first quantity, and f0 represents the second quantity;

[0082] The step size pixel value is calculated, which includes a step size width pixel value and a step size height pixel value. The step size width pixel value is equal to the original image width pixel value divided by the number of pixels in the width direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4. The step size height pixel value is equal to the original image height pixel value divided by the number of pixels in the height direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4. The formula for calculating the step size pixel value is as follows:

[0083]

[0084] In the formula, stride_x represents the step width in pixels, stride_y represents the step height in pixels, w represents the original image width in pixels, h represents the original image height in pixels, w_n represents the number of pixels in the width direction, h_n represents the number of pixels in the height direction, and ceil represents rounding up to the nearest integer.

[0085] Calculate the pixel values ​​of the region block, wherein the width pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size width pixel value, and the height pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size height pixel value; the formula for calculating the pixel values ​​of the region block is as follows:

[0086]

[0087] In the formula, block_x represents the width of the region block in pixels, block_y represents the height of the region block in pixels, overlap represents the preset overlap in pixels, stride_x represents the stride width in pixels, and stride_y represents the stride height in pixels.

[0088] In an optional embodiment of the first aspect of the present invention, in the step of filling the original image according to the number of region blocks and the step size pixel value to obtain a filled image and the pixel values ​​of the filled image;

[0089] The four sides of the original image are filled respectively. The width pixel value of the filled image is equal to the preset overlap pixel value multiplied by 2, plus the width pixel value of the region block multiplied by the number of pixels in the width direction. The height pixel value of the filled image is equal to the preset overlap pixel value multiplied by 2, plus the height pixel value of the region block multiplied by the number of pixels in the height direction. The formula for calculating the pixel value of the filled image is as follows:

[0090]

[0091] In the formula, w_p represents the width pixel value of the filled image, h_p represents the height pixel value of the filled image, stride_x represents the stride width pixel value, stride_y represents the stride height pixel value, w_n represents the number of pixels in the width direction, h_n represents the number of pixels in the height direction, and overlap represents the preset overlap pixel value.

[0092] In an optional embodiment of the first aspect of the present invention, creating a blank image based on the pixel values ​​of the filled image and the preset overlapping pixel values ​​includes the following steps:

[0093] Calculate the pixel values ​​of the blank image, where the pixel values ​​of the blank image are equal to the pixel values ​​of the filled image multiplied by a preset magnification factor;

[0094] A blank image is created based on the pixel values ​​of a blank image; the formula for calculating the pixel values ​​of the blank image is shown below:

[0095]

[0096] In the formula, w_o represents the width pixel value of the blank image, h_o represents the height pixel value of the blank image, and mul represents the preset magnification factor.

[0097] In an optional embodiment of the first aspect of the present invention, a plurality of first screenshots are sequentially cropped on the filled image according to the pixel value of the region block and the step size pixel value, wherein there are overlapping pixels between two adjacent first screenshots, including the following steps:

[0098] A first screenshot is taken from the filled image, wherein the first screenshot is located at one corner of the filled image, and the pixel value of the first screenshot is equal to the pixel value of the region block;

[0099] Based on the position of the first screenshot, shift the step width pixel value along the width direction to take a screenshot. Repeat this step until the number of first screenshots taken in the width direction is the same as the number in the width direction.

[0100] Based on the positions of all the first screenshots in the previous row, take a screenshot by shifting the step height pixel value along the height direction. Repeat this step until the number of first screenshots taken in the height direction is the same as the number taken in the height direction.

[0101] In this embodiment, a first screenshot can be taken from the top left corner. Then, a second screenshot can be taken by shifting the position of the first screenshot to the right by a step width pixel value. This process continues, with the number of screenshots taken directly in this row equal to the number of screenshots in the width direction. A second screenshot can be taken by shifting the position of the first screenshot downwards by a step height pixel value. This first screenshot is the first screenshot of the second row. The second row is then screenshotted using the same method. The other areas are then screenshotted using the same method, with the total number of rows equal to the number of screenshots in the height direction.

[0102] In an optional embodiment of the first aspect of the present invention, in the step of performing model inference on all the first screenshots according to the preset magnification factor to obtain a plurality of second screenshots, the pixel value of the second screenshot is equal to the pixel value of the first screenshot multiplied by the preset magnification factor. The formula for calculating the second screenshot is as follows:

[0103]

[0104] In the formula, w_2 represents the width pixel value of the second screenshot, h_2 represents the height pixel value of the second screenshot, block_x represents the width pixel value of the region block, block_y represents the height pixel value of the region block, and mul represents the preset magnification factor.

[0105] In an optional embodiment of the first aspect of the present invention, in the step of performing fill removal processing on a plurality of second screenshots according to the preset magnification factor and the preset overlapping pixel value to obtain a plurality of third screenshots, fill removal processing is performed on all four sides of the second screenshots, wherein the pixel value removed from each side is equal to the preset magnification factor multiplied by the preset overlapping pixel value; the formula for calculating the pixel value of the third screenshot is as follows:

[0106]

[0107] In the formula, w_3 represents the width pixel value of the third screenshot, h_3 represents the height pixel value of the third screenshot, w_2 represents the width pixel value of the second screenshot, h_2 represents the height pixel value of the second screenshot, mul represents the preset magnification factor, and overlap represents the preset overlap pixel value.

[0108] See Figure 2 A second aspect of the present invention provides a system for image super-resolution processing, comprising:

[0109] The acquisition module 10 is used to acquire the original image, the pixel values ​​of the original image, and the size of the available video memory;

[0110] The calculation module 20 is used to perform block calculation on the original image based on the pixel value of the original image, the size of the available video memory, the preset magnification factor and the preset overlapping pixel value, to obtain the number of region blocks, the pixel value of the region block and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block.

[0111] The filling module 30 is used to fill the original image according to the number of region blocks and the step size pixel value to obtain the filled image and the pixel values ​​of the filled image.

[0112] The creation module 40 is used to create a blank image based on the pixel values ​​of the filled image and the preset overlapping pixel values;

[0113] The cropping module 50 is used to sequentially crop multiple first screenshots on the filled image according to the pixel value of the region block and the step size pixel value, wherein there are overlapping pixels between two adjacent first screenshots.

[0114] The model inference module 60 is used to perform model inference on all the first screenshots according to the preset magnification factor to obtain multiple second screenshots;

[0115] The fill removal module 70 is used to perform fill removal processing on multiple second screenshots according to the preset magnification factor and the preset overlapping pixel value to obtain multiple third screenshots;

[0116] The pasting module 80 is used to paste multiple third screenshots sequentially into the blank image to obtain the target image.

[0117] In an optional embodiment of the second aspect of the present invention, the computing module 20 includes:

[0118] The first calculation unit is used to calculate the required video memory size, which is equal to the original image width pixel value multiplied by the original image height pixel value multiplied by the preset magnification factor multiplied by the preset magnification factor multiplied by 2 divided by 1024.

[0119] The second calculation unit is used to calculate the scaling factor, which is equal to the required video memory size divided by the available video memory size, and then rounded up to the integer.

[0120] The third calculation unit is used to calculate the first quantity and the second quantity respectively. The first quantity is equal to the square root of the proportional coefficient. When the first quantity is not an integer, it is rounded down. The second quantity is equal to the proportional coefficient divided by the first quantity. When the second quantity is not an integer, the first quantity is added by 1, and the calculation of the second quantity continues until the calculated second quantity is an integer. Then, the value of the first quantity is substituted back into the first quantity.

[0121] The fourth calculation unit is used to calculate the number of region blocks, wherein the number of region blocks is equal to the number in the width direction multiplied by the number in the height direction. When the original image width pixel value is greater than the original image height pixel value, the number in the width direction is equal to the first number and the number in the height direction is equal to the second number. When the original image width pixel value is less than the original image height pixel value, the number in the width direction is equal to the second number and the number in the height direction is equal to the first number.

[0122] The fifth calculation unit is used to calculate the step size pixel value, which includes a step size width pixel value and a step size height pixel value. The step size width pixel value is equal to the original image width pixel value divided by the number of pixels in the width direction, divided by 4, then rounded up to the nearest integer, and then multiplied by 4. The step size height pixel value is equal to the original image height pixel value divided by the number of pixels in the height direction, divided by 4, then rounded up to the nearest integer, and then multiplied by 4.

[0123] The sixth calculation unit is used to calculate the pixel value of the region block, wherein the width pixel value of the region block is equal to the preset overlapping pixel value multiplied by 2 plus the step size width pixel value, and the height pixel value of the region block is equal to the preset overlapping pixel value multiplied by 2 plus the step size height pixel value.

[0124] In an optional embodiment of the second aspect of the invention, the creation module 40 includes:

[0125] The seventh calculation unit is used to calculate the pixel value of the blank image, wherein the pixel value of the blank image is equal to the pixel value of the filled image multiplied by a preset magnification factor;

[0126] Create a cell to create a blank image based on the pixel values ​​of a blank image.

[0127] In an optional embodiment of the second aspect of the present invention, the interception module 50 includes:

[0128] The first cropping unit is used to crop a first screenshot on the filled image, wherein the first screenshot is located at one corner of the filled image, and the pixel value of the first screenshot is equal to the pixel value of the region block.

[0129] The second cropping unit is used to crop by translating the step width pixel value along the width direction based on the position of the previous first cropping image, and repeating this step until the number of first cropping images cropped in the width direction is the same as the number of first cropping images in the width direction.

[0130] The third cropping unit is used to crop by translating the step height pixel value along the height direction based on the positions of all the first cropping images in the previous row, and repeating this step until the number of first cropping images cropped in the height direction is the same as the number of first cropping images in the height direction.

[0131] Figure 3 This is a schematic diagram of the structure of an image super-resolution processing terminal provided in an embodiment of the present invention. The image super-resolution processing terminal can vary significantly due to differences in configuration or performance, and may include one or more processors 1 (central processing units, CPUs) (e.g., one or more processors) and memory 2, and one or more storage media 3 (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media can be temporary or persistent storage. The program stored in the storage media may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the image super-resolution processing terminal. Furthermore, the processor may be configured to communicate with the storage media and execute the series of instruction operations stored in the storage media on the image super-resolution processing terminal.

[0132] The image super-resolution processing terminal of the present invention may further include one or more power supplies 4, one or more wired or wireless network interfaces 5, one or more input / output interfaces 6, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The terminal structure for image super-resolution processing shown does not constitute a limitation on the terminal for image super-resolution processing. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0133] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the image super-resolution processing method.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image super-resolution processing, characterized in that, Includes the following steps: Get the original image, the pixel values ​​of the original image, and the available video memory size; The original image is divided into blocks based on the pixel values ​​of the original image, the size of the available video memory, the preset magnification factor, and the preset overlap pixel value to obtain the number of region blocks, the pixel values ​​of the region blocks, and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block. The original image is filled according to the number of region blocks and the step size pixel value to obtain the filled image and the pixel values ​​of the filled image; A blank image is created based on the pixel values ​​of the filled image and the preset overlapping pixel values; Multiple first screenshots are sequentially cropped from the filled image based on the pixel values ​​of the region block and the step size pixel values, wherein there are overlapping pixels between two adjacent first screenshots. Based on the preset magnification factor, all the first screenshots are subjected to model inference to obtain multiple second screenshots; Based on the preset magnification factor and the preset overlapping pixel value, the multiple second screenshots are respectively subjected to defilling processing to obtain multiple third screenshots; Multiple third screenshots are pasted sequentially into the blank image to obtain the target image; The step of dividing the original image into blocks based on the pixel values ​​of the original image, the size of the available video memory, a preset magnification factor, and a preset overlap pixel value to obtain the number of region blocks, the pixel values ​​of the region blocks, and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is less than the pixel value of the region block, includes the following steps: Calculate the required video memory size, which is equal to the original image width pixel value multiplied by the original image height pixel value multiplied by the preset magnification factor multiplied by the preset magnification factor multiplied by 2 divided by 1024; Calculate the scaling factor, which is equal to the required video memory size divided by the available video memory size, and then rounded up to the nearest integer. Calculate the first quantity and the second quantity respectively. The first quantity is equal to the square root of the proportional coefficient. If the first quantity is not an integer, round it down. The second quantity is equal to the proportional coefficient divided by the first quantity. If the second quantity is not an integer, add 1 to the first quantity and continue to calculate the second quantity until the calculated second quantity is an integer. Then, substitute the value of the first quantity back into the first quantity. The number of region blocks is calculated, and the number of region blocks is equal to the number in the width direction multiplied by the number in the height direction. When the original image width pixel value is greater than the original image height pixel value, the number in the width direction is equal to the first number, and the number in the height direction is equal to the second number. When the original image width pixel value is less than the original image height pixel value, the number in the width direction is equal to the second number, and the number in the height direction is equal to the first number. The step size pixel value is calculated, which includes a step size width pixel value and a step size height pixel value. The step size width pixel value is equal to the original image width pixel value divided by the number of pixels in the width direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4. The step size height pixel value is equal to the original image height pixel value divided by the number of pixels in the height direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4. Calculate the pixel value of the region block, wherein the width pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size width pixel value, and the height pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size height pixel value.

2. The image super-resolution processing method according to claim 1, characterized in that, In the step of filling the original image according to the number of region blocks and the step size pixel value to obtain the filled image and the pixel values ​​of the filled image; The four sides of the original image are filled respectively. The width pixel value of the filled image is equal to the preset overlap pixel value multiplied by 2 plus the width pixel value of the region block multiplied by the number of pixels in the width direction. The height pixel value of the filled image is equal to the preset overlap pixel value multiplied by 2 plus the height pixel value of the region block multiplied by the number of pixels in the height direction.

3. The image super-resolution processing method according to claim 2, characterized in that, Creating a blank image based on the pixel values ​​of the filled image and the preset overlapping pixel values ​​includes the following steps: Calculate the pixel values ​​of the blank image, where the pixel values ​​of the blank image are equal to the pixel values ​​of the filled image multiplied by a preset magnification factor; Create a blank image based on the pixel values ​​of the blank image.

4. The image super-resolution processing method according to claim 3, characterized in that, Multiple first screenshots are sequentially extracted from the filled image based on the pixel values ​​of the region block and the step size pixel values, wherein adjacent first screenshots have overlapping pixels, including the following steps: A first screenshot is taken from the filled image, wherein the first screenshot is located at one corner of the filled image, and the pixel value of the first screenshot is equal to the pixel value of the region block; Based on the position of the first screenshot, shift the step width pixel value along the width direction to take a screenshot. Repeat this step until the number of first screenshots taken in the width direction is the same as the number in the width direction. Based on the positions of all the first screenshots in the previous row, take a screenshot by shifting the step height pixel value along the height direction. Repeat this step until the number of first screenshots taken in the height direction is the same as the number taken in the height direction.

5. The image super-resolution processing method according to claim 4, characterized in that, In the step of performing model inference on all the first screenshots according to the preset magnification factor to obtain multiple second screenshots... The pixel value of the second screenshot is equal to the pixel value of the first screenshot multiplied by a preset magnification factor.

6. The image super-resolution processing method according to claim 5, characterized in that, In the step of removing fill from multiple second screenshots according to the preset magnification factor and the preset overlapping pixel value to obtain multiple third screenshots... The fill removal process is performed on all four sides of the second screenshot, wherein the pixel value removed from each side is equal to the preset magnification factor multiplied by the preset overlap pixel value.

7. A system for image super-resolution processing, characterized in that, include: The acquisition module is used to acquire the original image, the pixel values ​​of the original image, and the size of the available video memory; The calculation module is used to perform block calculation on the original image based on the pixel value of the original image, the size of the available video memory, the preset magnification factor and the preset overlapping pixel value, to obtain the number of region blocks, the pixel value of the region block and the step size pixel value between two adjacent region blocks, wherein the step size pixel value is smaller than the pixel value of the region block. A filling module is used to fill the original image according to the number of region blocks and the step size pixel value to obtain a filled image and the pixel values ​​of the filled image. A creation module is used to create a blank image based on the pixel values ​​of the filled image and the preset overlapping pixel values; The cropping module is used to sequentially crop multiple first screenshots on the filled image according to the pixel value of the region block and the step size pixel value, wherein there are overlapping pixels between two adjacent first screenshots; The model inference module is used to perform model inference on all the first screenshots according to the preset magnification factor to obtain multiple second screenshots; The fill removal module is used to perform fill removal processing on multiple second screenshots according to the preset magnification factor and the preset overlapping pixel value to obtain multiple third screenshots; The pasting module is used to paste multiple third screenshots sequentially into the blank image to obtain the target image; Specifically, the filling module is used for: Calculate the required video memory size, which is equal to the original image width pixel value multiplied by the original image height pixel value multiplied by the preset magnification factor multiplied by the preset magnification factor multiplied by 2 divided by 1024; Calculate the scaling factor, which is equal to the required video memory size divided by the available video memory size, and then rounded up to the nearest integer. Calculate the first quantity and the second quantity respectively. The first quantity is equal to the square root of the proportional coefficient. If the first quantity is not an integer, round it down. The second quantity is equal to the proportional coefficient divided by the first quantity. If the second quantity is not an integer, add 1 to the first quantity and continue to calculate the second quantity until the calculated second quantity is an integer. Then, substitute the value of the first quantity back into the first quantity. The number of region blocks is calculated, and the number of region blocks is equal to the number in the width direction multiplied by the number in the height direction. When the original image width pixel value is greater than the original image height pixel value, the number in the width direction is equal to the first number, and the number in the height direction is equal to the second number. When the original image width pixel value is less than the original image height pixel value, the number in the width direction is equal to the second number, and the number in the height direction is equal to the first number. The step size pixel value is calculated, which includes a step size width pixel value and a step size height pixel value. The step size width pixel value is equal to the original image width pixel value divided by the number of pixels in the width direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4. The step size height pixel value is equal to the original image height pixel value divided by the number of pixels in the height direction, divided by 4, rounded up to the nearest integer, and then multiplied by 4. Calculate the pixel value of the region block, wherein the width pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size width pixel value, and the height pixel value of the region block is equal to the preset overlap pixel value multiplied by 2 plus the step size height pixel value.

8. A terminal for image super-resolution processing, characterized in that, The terminal includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the terminal to perform the image super-resolution processing method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the image super-resolution processing method as described in any one of claims 1-6.

Images