A super-resolution image processing method and system with image type adaptability

Through a two-step processing method, combining the 'general' and 'fine' super-resolution model, the problem of poor results of conventional super-resolution methods is solved, and the detailed improvement of image resolution and brightness correction is achieved, adapting to the characteristics of different image types , improves image quality.

CN111724303BActive Publication Date: 2024-11-29HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202010398292.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-05-12
Publication Date
2024-11-29
Estimated Expiration
2040-05-12

AI Technical Summary

Technical Problem

The conventional image-type super-resolution method has poor effect and cannot effectively adapt to the characteristics of different image types, resulting in the image resolution improvement not being detailed enough, and there may be local brightness abnormalities.

Method used

Two-step processing method is adopted: first, the 'generic' super-resolution model based on multi-type images is used to generate preliminary high-resolution images, and then the 'fine' super-resolution model based on single-type images is refined by training depth The construction of network and transformation matrix L and R gradually improves image resolution and corrects brightness abnormalities.

Benefits of technology

It realizes effective improvement and refinement of image resolution, making the image more detailed texture and color that conforms to its own characteristics, and at the same time overcomes the problem of local brightness abnormalities and improves the quality of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111724303B_ABST
    Figure CN111724303B_ABST
Patent Text Reader

Abstract

The present invention provides a super-resolution image processing method with image type adaptability, comprising the following steps: S1, constructing a "universal" super-resolution model based on multiple types of images, taking the image to be processed as the input of the "universal" super-resolution model based on multiple types of images, and running the "universal" super-resolution model based on multiple types of images to obtain a preliminary high-resolution image of the image to be processed; S2, constructing a "fine" super-resolution model based on a single type of image, taking the preliminary high-resolution image as the input of the "fine" super-resolution model based on a single type of image, and running the "fine" super-resolution model based on a single type of image to obtain a final high-resolution image. The present invention also provides a super-resolution image processing system with image type adaptability. The beneficial effect of the present invention is that while improving the image resolution, it can also overcome problems such as local brightness anomalies in the original image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image processing method, and in particular to a super-resolution image processing method and system with image type adaptability. Background Art

[0002] Image super-resolution has a wide range of application needs. Conventional image super-resolution methods use a single model to obtain a super-resolution image of the original image, but the effect is poor. Summary of the invention

[0003] In order to solve the problems in the prior art, the present invention provides a super-resolution image processing method and system with image type adaptability.

[0004] The present invention provides a super-resolution image processing method with image type adaptability, comprising the following steps:

[0005] S1. construct a "universal" super-resolution model based on multiple types of images, use the image to be processed as the input of the "universal" super-resolution model based on multiple types of images, and after running the "universal" super-resolution model based on multiple types of images, obtain a preliminary high-resolution image of the image to be processed;

[0006] S2. Build a "fine" super-resolution model based on a single type of image, use the preliminary high-resolution image as the input of the "fine" super-resolution model based on a single type of image, and after running the "fine" super-resolution model based on a single type of image, obtain the final high-resolution image.

[0007] As a further improvement of the present invention, a method for constructing a "universal" super-resolution model based on multi-type images is as follows: taking low-resolution images of multi-type images as input, and taking the original high-resolution images of multi-type images as output, determining an objective function, and obtaining a "universal" super-resolution model through training.

[0008] As a further improvement of the present invention, a method for constructing a "universal" super-resolution model based on multiple types of images also includes: assuming that there are M original low-resolution images and corresponding M high-resolution images, first all high-resolution images and low-resolution images are scaled to the same image size, and then each original low-resolution image and each high-resolution image matrix are converted into column vectors in a column-concatenated manner, and the matrix composed of the column vectors corresponding to all M original low-resolution images is A, and the matrix composed of the column vectors of the M high-resolution images corresponding to the original low-resolution images is B. It is assumed that A can be finally transformed into B by using transformation matrices L and R in the manner of LA=Z, ZR=B.

[0009] As a further improvement of the present invention, L and R are solved as follows:

[0010] (1) First, we give the initial value of L by specifying any random number between 0 and 1, and then let

[0011]

[0012] (2) Order

[0013] (3) Order

[0014] (4) If the current Value and previous step If the norm of the difference matrix between the values ​​is less than ε, the current and

[0015] as the solution of L and R; otherwise, go to (3) to execute.

[0016] As a further improvement of the present invention, in step S2, the first deep network trained in the corresponding "fine" super-resolution model is applied to a preliminary high-resolution image to obtain the output image of the first deep network, and the second deep network trained in the "fine" super-resolution model is applied to the "inverse image" of the preliminary high-resolution image to obtain the preliminary output image of the second deep network. The image obtained by subtracting each pixel value in the preliminary output image of the second deep network from 255 is the final output result of the second deep network. The final output result of the second deep network is matrix added to the output image of the first deep network and divided by 2 to obtain the final high-resolution image.

[0017] As a further improvement of the present invention, a method for constructing a "fine" super-resolution model based on a single type of image is as follows: taking a low-resolution image of a single type of image as input, and taking the original high-resolution image of the single type of image as output, designing a first deep network and a second deep network, and using the input and output to train the first deep network and the second deep network to obtain a "fine" super-resolution model for a single type of image.

[0018] As a further improvement of the present invention, a method for constructing a "fine" super-resolution model based on a single type of image also includes: the input and output used when training the first deep network are the original low-resolution image and the original high-resolution image of the single type of image, respectively; the input and output used when training the second deep network are the original low-resolution image and the "inverse image" of the original high-resolution image of the single type of image, and the "inverse image" has exactly the same size as the original image; an "inverse image" of the original image is obtained in the following manner: the value of each pixel in the "inverse image" is equal to 255 minus the pixel value of the corresponding original image, and a "fine" super-resolution model is established for each type of image.

[0019] As a further improvement of the present invention, before constructing the "general" super-resolution model and the "fine" super-resolution model, the following data collection and acquisition work for model training is carried out:

[0020] (1) Training data collection and acquisition for building a “general” super-resolution model:

[0021] First, we use the collected high-resolution images to construct low-resolution images. The basic scheme for constructing low-resolution images is pixel extraction.

[0022] (2) Training data collection and acquisition for building a “fine” super-resolution model:

[0023] Firstly, a single type of high-resolution image is collected, and then its low-resolution image is constructed. The basic scheme for constructing the low-resolution image is pixel extraction.

[0024] As a further improvement of the present invention, a low-resolution camera is used to collect images to be processed, a processor is used to run a "general" super-resolution model based on multiple types of images and a "fine" super-resolution model based on a single type of image, and a final high-resolution image is output through a display screen.

[0025] The present invention also provides a super-resolution image processing system with image type adaptability, comprising a readable storage medium, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, they are used to implement any of the methods described above.

[0026] The beneficial effects of the present invention are as follows: through the above scheme, not only can the characteristics of general visible light images be used to perform preliminary image resolution improvement, but also the image resolution can be refined according to the image type so that the image is more specifically in line with its own characteristics of detailed texture and color; while improving the image resolution, problems such as local brightness anomalies in the original image can also be overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The present invention is a flow chart of a super-resolution image processing method with image type adaptability.

[0028] Figure 2 It is a flow chart of the present invention using the "fine" super-resolution model to obtain the final high-resolution image.

[0029] Figure 3 It is a network structure diagram of the deep network of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Aiming at the characteristics that different types of visible light images have partially similar image structures and the same type of visible light images have fine-grained similarities, the present invention proposes a super-resolution image processing method with image type adaptability. First, a "general" super-resolution model is used to obtain a preliminary high-resolution image of the image to be processed; then the preliminary high-resolution image is used as the input of a "fine" super-resolution model, and the final high-resolution image is obtained after running the model.

[0032] like Figure 1 As shown in the figure, after the "general" super-resolution model and the "fine" super-resolution model are built, the super-resolution operation of the image to be processed is performed in the following two steps:

[0033] The first step is to use the image to be processed as the input of the "universal" super-resolution model, and obtain the corresponding high-resolution image by running the model, which is called the preliminary high-resolution image in the present invention.

[0034] Step 2: Use the preliminary high-resolution image as the input of the "fine" super-resolution model, and obtain the corresponding high-resolution image by running the model, which is referred to as the final high-resolution image in the present invention.

[0035] A low-resolution camera is used to collect images to be processed, a processor is used to run a "general" super-resolution model based on multiple types of images and a "fine" super-resolution model based on a single type of image, and a display screen is used to output the final high-resolution image.

[0036] The specific process of using the "fine" super-resolution model to obtain the final high-resolution image is shown in Figure 2. Specifically, after applying the first deep network trained in the corresponding "fine" super-resolution model to a preliminary high-resolution image, an output image can be obtained. After applying the second deep network trained in the "fine" super-resolution model to the "inverse image" of the preliminary high-resolution image, an output image is also obtained. The image obtained by subtracting each pixel value in the output image of the second deep network from 255 is the final output result of the second deep network. The final output result of the second deep network is matrix-added to the output image of the first deep network and divided by 2 to obtain the final high-resolution image. Figure 2 The “image type selection” here means manually selecting the type of image to which super-resolution is being performed. After the “image type selection” is performed, the system automatically uses the trained “fine” super-resolution model for that image type.

[0037] The method for constructing a "universal" super-resolution model based on multiple types of images is as follows: take low-resolution images of multiple types of images as input, and their original high-resolution images as output, determine an objective function, and obtain a "universal" super-resolution model through training. Assuming that there are M original low-resolution images and corresponding M high-resolution images, first scale all high-resolution images and low-resolution images to the same image size (that is, make all image matrices the same size by image scaling), and then convert each original low-resolution image and each high-resolution image matrix into a column vector in a column-concatenated manner. Let the matrix composed of column vectors corresponding to all M original low-resolution images be A, and the matrix composed of column vectors of M high-resolution images corresponding to the original low-resolution images be B. Assume that A can be finally transformed into B by using transformation matrices L and R in the manner of LA=Z and ZR=B. The solution of L and R is as follows:

[0038] (1) First, we give the initial value of L by specifying any random number between 0 and 1, and then let

[0039]

[0040] (2) Order

[0041] (3) Order

[0042] (4) If the current Value and previous step If the norm of the difference matrix between the values ​​is less than ε, the current and

[0043] as the solution of L and R; otherwise, go to (3) to execute.

[0044] The method for constructing a "fine" super-resolution model based on a single type of image is as follows: a low-resolution image of a single type of image is used as input, and its original high-resolution image is used as output. Two deep networks are designed, and they are trained using the input and output to obtain a "fine" super-resolution model for a specific type of image. Among them, the input and output used when training the first deep network are the original low-resolution image and the original high-resolution image of a single type of image, respectively. The input and output used when training the second deep network are the "inverse image" of the original low-resolution image and the original high-resolution image of a single type of image. The "inverse image" has exactly the same size as the original image. The "inverse image" of an original image is obtained in the following way: the value of each pixel in the "inverse image" is equal to 255 minus the pixel value of the corresponding original image. It should be noted that a "fine" super-resolution model needs to be established for each type of image.

[0045] The first deep network has the same network structure as the first deep network, except that the input and output images during training are different. Figure 3 Among them, the product image of adjacent pixels of the input image is conducive to sharpening the original pixels of the input image, thereby highlighting the detail information in the image.

[0046] The construction of the "general" super-resolution model and the "fine" super-resolution model is based on a large number of high-resolution images and corresponding low-resolution images. Before the construction of the "general" super-resolution model and the "fine" super-resolution model, the following data collection and acquisition work for model training is carried out:

[0047] (1) Training data collection and acquisition for building a “general” super-resolution model:

[0048] First, we use the various high-resolution images we have collected to construct low-resolution images. The basic method for constructing low-resolution images is pixel extraction. For example, we perform "pixel extraction" on the pixels in the horizontal and vertical directions of the image, and the original high-resolution image will be transformed into an image with half the resolution in both the horizontal and vertical directions (i.e., a low-resolution image). The image resolution can be further reduced by the same method.

[0049] The specific method of using a high-resolution image to construct a low-resolution image is as follows: using the scheme of "alternate point extraction" of pixels, an image with half the resolution of the original high-resolution image in the horizontal and vertical directions and an image with a quarter of the resolution of the original image in the horizontal and vertical directions are obtained. Both types of images are used as low-resolution image data for the construction of a "general" super-resolution model.

[0050] (2) Training data collection and acquisition for building a “fine” super-resolution model:

[0051] First, use the collected high-resolution images of a single type (for example, images of categories such as flowers, plants, and street scenes can all be regarded as images of a single type) to construct their low-resolution images. The basic scheme for constructing low-resolution images is pixel extraction. Specifically, "alternate point extraction" is performed on the pixels in the horizontal and vertical directions of the image, and the original high-resolution image will be transformed into an image with half the resolution in the horizontal and vertical directions (i.e., a low-resolution image). In addition, for some of the low-resolution images obtained in the above steps, a region of size s×t is randomly selected, and two contrast modifications, contrast stretching and compression, are performed to model the brightness anomalies of local image areas in practice. The low-resolution images obtained by the above two schemes are used as data for the construction of a "fine" super-resolution model.

[0052] It should be noted that, since a "fine" super-resolution model needs to be established for each type of image, the solution proposed in the present invention must first determine which image types' "fine" super-resolution models need to be established, and at the same time collect and acquire the training data required to build the "fine" super-resolution models for these image types respectively.

[0053] The present invention also provides a super-resolution image processing system with image type adaptability, comprising a readable storage medium, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, they are used to implement any of the methods described above.

[0054] In order to make full use of the common characteristics of visible light images and the fine-grained similarity between images of the same type, the present invention proposes a two-step super-resolution method with image type adaptability. The method can not only use the characteristics of general visible light images to perform preliminary image resolution improvement, but also refine the image resolution according to the image type, so that the image is more specific and has fine texture and color that are more in line with its own characteristics. While improving the image resolution, the method can also overcome problems such as local brightness anomalies in the original image.

[0055] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A super-resolution image processing method with image type adaptability, It is characterized in that The following steps are involved: S1. construct a "universal" super-resolution model based on multiple types of images, use the image to be processed as the input of the "universal" super-resolution model based on multiple types of images, and after running the "universal" super-resolution model based on multiple types of images, obtain a preliminary high-resolution image of the image to be processed; S2, constructing a "fine" super-resolution model based on a single type of image, taking the preliminary high-resolution image as the input of the "fine" super-resolution model based on a single type of image, and running the "fine" super-resolution model based on a single type of image to obtain a final high-resolution image; in step S2, applying the first deep network trained in the corresponding "fine" super-resolution model to a preliminary high-resolution image to obtain an output image of the first deep network, applying the second deep network trained in the "fine" super-resolution model to the "inverse image" of the preliminary high-resolution image to obtain a preliminary output image of the second deep network, and subtracting 255 from each pixel value in the preliminary output image of the second deep network to obtain an image that is the final output result of the second deep network, and performing matrix addition on the final output result of the second deep network and the output image of the first deep network and dividing by 2 to obtain a final high-resolution image; The "anti-image" has exactly the same size as the original image. The "anti-image" of an original image is obtained in the following way: the value of each pixel in the "anti-image" is equal to 255 minus the corresponding pixel value of the original image. A "fine" super-resolution model is established for each type of image.

2. The super-resolution image processing method with image type adaptability according to claim 1, Features: The method for constructing a "universal" super-resolution model based on multi-type images is as follows: low-resolution images of multi-type images are taken as input, and original high-resolution images of multi-type images are taken as output, an objective function is determined, and a "universal" super-resolution model is obtained through training.

3. The super-resolution image processing method with image type adaptability according to claim 2, Features: The method for constructing a "universal" super-resolution model based on multiple types of images also includes: assuming that there are M original low-resolution images and corresponding M high-resolution images, first scale all high-resolution images and low-resolution images to the same image size, and then convert each original low-resolution image and each high-resolution image matrix into a column vector in a column-concatenated manner, let the matrix composed of the column vectors corresponding to all M original low-resolution images be A, and the matrix composed of the column vectors of the M high-resolution images corresponding to the original low-resolution images be B, assuming that A can be finally transformed into B by using transformation matrices L and R in the manner of LA=Z, ZR=B.

4. The super-resolution image processing method with image type adaptability according to claim 3, Features: The solutions for L and R are as follows: (1) First, we give the initial value of L by specifying any random number between 0 and 1, and then let (2) Order (3) Order (4) If the current Value and previous step If the norm of the difference matrix between the values ​​is less than ε, the current and as the solution of L and R; otherwise, go to (3) to execute.

5. The super-resolution image processing method with image type adaptability according to claim 1, Features: The method for constructing a "fine" super-resolution model based on a single type of image is as follows: take a low-resolution image of a single type of image as input, and take the original high-resolution image of a single type of image as output, design a first deep network and a second deep network, use the input and output to train the first deep network and the second deep network, and obtain a "fine" super-resolution model for a single type of image.

6. The super-resolution image processing method with image type adaptability according to claim 5, Features: The method for constructing a "fine" super-resolution model based on a single type of image also includes: the input and output used when training the first deep network are the original low-resolution image and the original high-resolution image of the single type of image, respectively; the input and output used when training the second deep network are the original low-resolution image and the "inverse image" of the original high-resolution image of the single type of image.

7. The super-resolution image processing method with image type adaptability according to claim 1, Features: A low-resolution camera is used to capture images to be processed, a processor is used to run a "general" super-resolution model based on multiple types of images and a "fine" super-resolution model based on a single type of image, and a display screen is used to output the final high-resolution image.

8. A super-resolution image processing system with image type adaptability, Features: The method comprises a readable storage medium in which execution instructions are stored. When the execution instructions are executed by a processor, the execution instructions are used to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Image super-resolution reconstruction method based on secondary anchor point neighborhood regression

    CN108416736A

  • Image super-resolution method based on two training sets

    CN108846797A

  • Image classification method based on super-resolution reconstruction

    CN110321939A

  • Depth map super-resolution processing method

    US20160328828A1

  • Image super-resolution enhancement method based on bidirectional recursion convolution neural network

    WO2017219263A1