A method of image super-resolution based on progressive network modeling

Abstract

The present invention discloses an image super-resolution method based on progressive network modeling, belonging to the technical field of image processing. The specific method includes: Step 1: Obtain an image judgment model and a super-resolution image, identify the resolution, pixels, and size of the super-resolution image, and integrate and label the identified resolution, pixels, and size as judgment input data. Input the judgment input data into the judgment model to obtain the maximum super-resolution multiple N of the super-resolution image; Step 2: Obtain an image reconstruction model; Step 3: Perform bicubic interpolation on the super-resolution image to obtain a W1 image, and record the interpolation method of the W1 image; Step 4: Reconstruct the W1 image through the image reconstruction model to obtain an M1 image, and identify the clarity L1 of the M1 image. When the identified clarity L1 < X1, where X1 is a threshold, through gradual interpolation and super-resolution, the obtained image is more convenient for the subsequent reconstruction of the image in the following steps, and the reconstructed image is clearer.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically relates to an image super-resolution method based on progressive network modeling. Background Art

[0002] In a large number of electronic image application fields, people often expect to obtain high-resolution images. High resolution means a high pixel density in the image, which can provide more details, and these details are indispensable in many practical applications. For example, high-resolution medical images are very helpful for doctors to make correct diagnoses; it is easy to distinguish similar objects from similar ones using high-resolution satellite images; if high-resolution images can be provided, the performance of pattern recognition in computer vision will be greatly improved. Since the 1970s, charge-coupled devices and CMOS image sensors have been widely used to capture digital images. Although these sensors are suitable for most image applications, the current resolution level and consumer price cannot meet future needs. Therefore, there is an urgent need for an image super-resolution method based on progressive network modeling to solve the problem of image super-resolution. Summary of the Invention

[0003] To solve the problems existing in the above solutions, the present invention provides an image super-resolution method based on progressive network modeling.

[0004] The object of the present invention can be achieved by the following technical solutions:

[0005] An image super-resolution method based on progressive network modeling, the specific method includes:

[0006] Step 1: Obtain an image judgment model and a super-resolution image, identify the resolution, pixels, and size of the super-resolution image, and integrate and label the identified resolution, pixels, and size as judgment input data, and input the judgment input data into the judgment model to obtain the maximum super-resolution multiple N of the super-resolution image;

[0007] Step 2: Obtain an image reconstruction model;

[0008] Step 3: Perform bicubic interpolation on the super-resolution image to obtain a W1 image, and record the interpolation method of the W1 image;

[0009] Step 4: Reconstruct the W1 image through the image reconstruction model to obtain an M1 image, identify the sharpness L1 of the M1 image, and when the identified sharpness L1 < X1, where X1 is a threshold, return to Step 3, adjust the interpolation method of the super-resolution image, and re-obtain the M1 image; save the M1 image with the sharpness L1 ≥ X1;

[0010] Step 5: loop step 3 and step 4 until the Mv image is obtained. When v=N, stop looping step 3 and step 4 and proceed to the next step;

[0011] Step 6: coordinate correlation positioning of all stored Mv images; and produce a super-resolution image set based on the correlation positioning;

[0012] Step 7: Set the super-resolution scrolling frame and use the super-resolution scrolling frame to super-resolution the super-resolution image.

[0013] Furthermore, the method for obtaining the image reconstruction model includes:

[0014] A large number of bicubic interpolated images and corresponding reconstructed images are obtained from the Internet, and the obtained bicubic interpolated images and corresponding reconstructed images are divided into a training set, a test set, and a verification set; an artificial intelligence model is constructed, and the artificial intelligence model is a convolutional neural network model; the artificial intelligence model is trained, tested, and verified through the training set, the test set, and the verification set; and the trained artificial intelligence model is marked as an image reconstruction model.

[0015] Furthermore, a bicubic interpolation is performed once to magnify once, and the magnification factor is one.

[0016] Furthermore, the method for performing coordinate correlation positioning on all stored Mv images includes:

[0017] Obtain the interpolation methods of all recorded Wv images, identify the Wv images that do not use bicubic interpolation, obtain the Wv′ images that are processed with bicubic interpolation and whose corresponding clarity does not meet the requirements, and mark the coordinates in the Wv′ images to the corresponding positions in the corresponding Wv images;

[0018] All Wv images are arranged from top to bottom according to the image name size with the image center as the origin, and then the super-resolution image is placed above the W1 image. The coordinate points of the same pixel in different Wv images are marked as similar associated points, and all coordinate points within the same associated points are associated according to the arrangement order of the Wv images.

[0019] Furthermore, the method for setting the super-resolution scroll frame includes:

[0020] A self-positioning frame is set, and a positioning node is set at the boundary inflection point of the self-positioning frame. The positioning node is used to identify the corresponding coordinates on the super-resolution image, and the identified coordinate points are connected according to the shape of the self-positioning frame; a scrolling unit is set, and the scrolling unit is used to zoom in or out the image in the self-positioning frame after the identified coordinate points are connected, and the scrolling unit and the self-positioning frame are integrated and marked as a super-resolution scrolling frame.

[0021] Furthermore, the shape of the self-positioning frame is a regular shape.

[0022] Furthermore, the method for the scrolling unit to zoom in or out the image within the self-positioning frame includes: input zoom in or out and drag zoom in or out.

[0023] Furthermore, the method of super-resolving the super-resolved image by using the super-resolved scrolling frame includes:

[0024] Set the shape of the super-resolution scrolling frame, place the super-resolution scrolling frame on the super-resolution image that needs to be super-resolution, set the size of the super-resolution scrolling frame, and after the setting is completed, identify the inflection point coordinates of the super-resolution scrolling frame, connect the identified coordinate points, and fix the super-resolution scrolling frame; mark the border formed by the connection of the coordinate points as a super-resolution frame, project the super-resolution frame on the super-resolution image set, select the super-resolution method, and super-resolve the super-resolution image in the super-resolution frame according to the selected super-resolution method.

[0025] Compared with the prior art, the beneficial effects of the present invention are: by setting an image judgment model, the maximum super-resolution multiple of the super-resolution image is preferentially judged, an iteration upper limit is provided for subsequent data processing, and the amount of data processing is reduced;

[0026] By performing bicubic interpolation on the super-resolution image, one bicubic interpolation is equal to one magnification, and the magnification is one time; by gradually performing interpolation and super-resolution, the obtained image is more convenient for image reconstruction in subsequent steps, and the reconstructed image is clearer, and when the clarity of the obtained image does not meet the requirements, it can be adjusted in time to ensure that the final super-resolution image meets the requirements; through the mutual cooperation of the super-resolution image set and the super-resolution scrolling frame, a variety of super-resolution options are provided to users to ensure that the super-resolution image meets the user's requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 It is a schematic diagram of the bicubic interpolation method of the present invention;

[0030] Figure 3 This is a schematic diagram of similar association points of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] like Figures 1 to 3 As shown, an image super-resolution method based on progressive network modeling, the specific method includes:

[0033] Step 1: Obtain an image judgment model and a super-resolution image, where the super-resolution image is the image that needs to be super-resolution. The image judgment model is used to judge the maximum super-resolution multiple of the super-resolution image; identify the resolution, pixels and size of the super-resolution image, and integrate the identified resolution, pixels and size into judgment input data, input the judgment input data into the judgment model, and obtain the maximum super-resolution multiple N of the super-resolution image; where N is a positive integer;

[0034] Methods for obtaining an image judgment model include:

[0035] Obtain historical judgment input data input; the historical judgment input data input includes resolution, pixels and size; set a corresponding maximum super-resolution multiple for the historical judgment input data input; build an artificial intelligence model, which is a neural network model; divide the historical judgment input data input and the corresponding maximum super-resolution multiple into a training set, a test set and a verification set; train, test and verify the artificial intelligence model through the training set, the test set and the verification set; mark the trained artificial intelligence model as an image judgment model;

[0036] Step 2: Obtain an image reconstruction model; the image reconstruction model is used to reconstruct the image after bicubic interpolation;

[0037] Methods for obtaining an image reconstruction model include:

[0038] A large number of bicubic interpolation images and corresponding reconstructed images are obtained from the Internet, and the obtained bicubic interpolation images and corresponding reconstructed images are divided into a training set, a test set, and a verification set; an artificial intelligence model is constructed, and the artificial intelligence model is a convolutional neural network model; the artificial intelligence model is trained, tested, and verified through the training set, the test set, and the verification set; and the trained artificial intelligence model is marked as an image reconstruction model;

[0039] Step 3: Perform bicubic interpolation on the super-resolution image to obtain the W1 image, and record the interpolation method of the W1 image. The W1 image is subjected to a bicubic interpolation. Performing a bicubic interpolation once means magnification once, and the magnification factor is one times. A magnification factor of one times means changing from n times to n+1 times. For example, if the previous magnification factor is 7 times, then performing a bicubic interpolation once means the magnification factor is 8 times, instead of changing from 7 times to 14 times.

[0040] For example, assuming that the source image A is m*n in size, and the size of the scaled target image B is M*N, then according to the ratio, the corresponding coordinates of B(X, Y) on A are A(x, y) = A(X*(m / M), Y*(n / N)). The nearest 16 pixels are selected as the parameters for calculating the pixel value at the target image B(X, Y);

[0041] like Figure 2 The point P shown is the position of the target image B at (X, Y) corresponding to the position in the source image. There will be a decimal part in the coordinate position of P. Assume that the coordinate of P is P(x+u, y+v), where x and y represent the integer part respectively, and u and v represent the decimal part respectively. Figure 2 The positions of the most recent 16 pixels shown are represented here by a(i, j) (i, j = 0, 1, 2, 3).

[0042] Bicubic interpolation method based on BiCubic basis function, the BiCubic basis function form is as follows:

[0043]

[0044] Where a is -0.5, and the parameter x in the BiCubic function is calculated to obtain the coefficients corresponding to 16 pixels. By describing how to calculate the coefficient k_ij corresponding to a(i, j), assuming that the row coefficient is k_i and the column coefficient is k_j; taking the position a00 as an example: First, calculate the position of the current pixel and point P, for example, the distance between a00 and P(x+u, y+v) is (1+u, 1+v); then we can get: k_i_0 = W(1+u), k_j_0 = W(1+v), and similarly, we can get the coefficients corresponding to all rows and columns:

[0045] k_i_0=W(1+u), k_i_1=W(u), k__i_2=W(1-u), k_i_3=W(2-u); k_j_0=W(1+v), k_j_1=W(v), k_j_2=W(1-v), k_j_3=W(2-v);

[0046] By k_i_j=k_i*k_j, we can get the weight corresponding to each pixel a(i, j);

[0047] Finally, the pixel value corresponding to the target image B(X, Y) can be obtained through the summation formula:

[0048] pixelB(X, Y) = pixelA(0, 0)*k_0_0 + pixelA(0, 1)*k_0_1 + … + pixelA(3, 3)*k_3_3.

[0049] Step Four: Reconstruct the W1 image through the image reconstruction model to obtain the M1 image. The M1 image is the image after the reconstruction of the W1 image. Identify the clarity L1 of the M1 image. When the identified clarity L1 < X1, where X1 is the threshold, return to Step Three, adjust the interpolation method of the super-resolution image. For example, change the bicubic interpolation to an interpolation method based on deep learning, and re-obtain the M1 image. Save the M1 image with clarity L1 ≥ X1.

[0050] Step Five: Loop Step Three and Step Four until the Mv image is obtained, where v is a positive integer and Mv is the M image obtained in the v-th time. When v = N, stop the loop of Step Three and Step Four and proceed to the next step.

[0051] Step Six: Perform coordinate correlation positioning on all stored Mv images; make a super-resolution image set according to the correlation positioning.

[0052] The method for performing coordinate correlation positioning on all stored Mv images and making a super-resolution image set includes:

[0053] Obtain the interpolation methods of all recorded Wv images, identify the Wv images that do not use bicubic interpolation, obtain the Wv′ images that have undergone bicubic interpolation processing with unsatisfactory corresponding clarity, and mark the coordinates in the Wv′ images at the corresponding positions in the corresponding Wv images.

[0054] As Figure 3 shown, arrange all Wv images with the image center as the origin, in the order of the image names, i.e., W1, W2, W3, …, from top to bottom. Then place the super-resolution image corresponding above the W1 image. The super-resolution image is the initial image. Mark the coordinate points of the same pixel points in different Wv images as the same type of associated points, that is, Figure 3 the coordinate points corresponding to the curve that runs through from W1 to W3 in Figure 3 . Associate all the coordinate points within the same type of associated points in the order of the arrangement of the Wv images. As

[0055] Step Seven: Set up a super-resolution scroll box to perform super-resolution on the super-resolution image through the super-resolution scroll box.

[0056] The method for setting up the super-resolution scroll box includes:

[0057] A self-positioning frame is set, and the shape of the self-positioning frame is a regular shape, for example, a circle, a regular polygon, a rectangle and other regular shapes, which can be adjusted and selected by the user. A positioning node is set at the boundary inflection point of the self-positioning frame, and the positioning node is used to identify the corresponding coordinates on the super-resolution image, which is equivalent to the positioning node being located at a certain position on the super-resolution image and identifying the coordinates of this position; the identified coordinate points are connected according to the shape of the self-positioning frame; a scrolling unit is set, and the scrolling unit is used to zoom in or out the image in the self-positioning frame after the identified coordinate points are connected. The method for the scrolling unit to zoom in or out the image in the self-positioning frame includes: input zoom in or out and drag zoom in or out; input zoom in or out refers to directly inputting the zoom in or out multiple, and drag zoom in or out refers to zooming in or out by using the mouse wheel. When using a mobile phone for operation, it is manual zooming in or out; the scrolling unit and the self-positioning frame are integrated and marked as a super-resolution scrolling frame.

[0058] Methods for super-resolving a super-resolved image by super-resolved scrolling frames include:

[0059] Set the shape of the super-resolution scrolling frame, place the super-resolution scrolling frame on the super-resolution image that needs to be super-resolution, set the size of the super-resolution scrolling frame, and after the setting is completed, identify the inflection point coordinates of the super-resolution scrolling frame, connect the identified coordinate points, and fix the super-resolution scrolling frame; at this time, the super-resolution scrolling frame is equivalent to a magnifying glass, which can enlarge or reduce the image in the super-resolution scrolling frame; mark the frame formed by connecting the coordinate points as a super-resolution frame, and project the super-resolution frame on the super-resolution image set, which is equivalent to projecting the super-resolution frame on the super-resolution image to W1, W2, W3,... according to the coordinates; select the super-resolution method, which is the method for the scrolling unit to enlarge or reduce the image in the self-positioning frame, and super-resolve the super-resolution image in the super-resolution frame according to the selected super-resolution method.

[0060] Working principle of the present invention: Obtain an image judgment model and a super-resolution image, identify the resolution, pixels, and size of the super-resolution image, and integrate and label the identified resolution, pixels, and size as judgment input data. Input the judgment input data into the judgment model to obtain the maximum super-resolution multiple N of the super-resolution image; Obtain a large number of images subjected to bicubic interpolation and corresponding reconstructed images from the Internet, and divide the obtained images subjected to bicubic interpolation and corresponding reconstructed images into a training set, a test set, and a verification set; Construct an artificial intelligence model, which is a convolutional neural network model; Train, test, and verify the artificial intelligence model through the training set, the test set, and the verification set; Label the trained artificial intelligence model as an image reconstruction model; Perform bicubic interpolation on the super-resolution image to obtain a W1 image, and record the interpolation method of the W1 image; Reconstruct the W1 image through the image reconstruction model to obtain an M1 image, and identify the clarity L1 of the M1 image. When the identified clarity L1 < X1, where X1 is a threshold, return to step three, adjust the interpolation method of the super-resolution image, and re-obtain the M1 image; Save the M1 images with clarity L1 ≥ X1; Perform iteration until an Mv image is obtained. When v = N, stop the loop of steps three and four and enter the next step; Perform coordinate association positioning on all stored Mv images; Produce a super-resolution image set according to the association positioning; Set a super-resolution scroll box, and perform super-resolution on the super-resolution image through the super-resolution scroll box.

[0061] In the embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation; The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the method of this embodiment.

[0062] Furthermore, for those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

[0063] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to cover all changes within the meaning and scope of the equivalent elements of the claims in the present invention. Any associated drawing marks in the claims should not be regarded as limiting the claimed rights.

[0064] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An image super-resolution method based on progressive network modeling, characterized in that: The specific method includes: Step 1: Obtain an image judgment model and a super-resolution image, identify the resolution, pixels, and size of the super-resolution image, integrate and mark the identified resolution, pixels, and size as judgment input data, input the judgment input data into the judgment model, and obtain the maximum super-resolution multiple N of the super-resolution image; Step 2: Obtain an image reconstruction model; Step 3: Perform bicubic interpolation on the super-resolution image to obtain a W1 image, and record the interpolation method of the W1 image; Step 4: Reconstruct the W1 image through the image reconstruction model to obtain an M1 image, identify the clarity L1 of the M1 image. When the identified clarity L1 < X1 (X1 is a threshold), return to Step 3, adjust the interpolation method of the super-resolution image, and re-obtain the M1 image; save the M1 image with clarity L1 ≥ X1; Step 5: Loop Steps 3 and 4 until an Mv image is obtained. When v = N, stop the loop of Steps 3 and 4 and proceed to the next step; Step 6: Perform coordinate association and positioning on all stored Mv images; make a super-resolution image set according to the association and positioning; Step 7: Set a super-resolution scroll box, and perform super-resolution on the super-resolution image through the super-resolution scroll box; The method for setting the super-resolution scroll box includes: Set a self-positioning border. Positioning nodes are provided at the boundary inflection points of the self-positioning border. The positioning nodes are used to identify the corresponding coordinates on the super-resolution image, and connect the identified coordinate points according to the shape of the self-positioning border; set a scrolling unit. The scrolling unit is used to magnify or reduce the image within the self-positioning border when the identified coordinate points are connected. Integrate and mark the scrolling unit and the self-positioning border as the super-resolution scroll box.

2. The image super-resolution method based on progressive network modeling according to claim 1, characterized in that: The method for obtaining the image reconstruction model includes: Obtain a large number of images that have undergone bicubic interpolation and corresponding reconstructed images from the Internet, divide the obtained images that have undergone bicubic interpolation and corresponding reconstructed images into a training set, a test set, and a verification set; construct an artificial intelligence model, and the artificial intelligence model is a convolutional neural network model; train, test, and verify the artificial intelligence model through the training set, the test set, and the verification set; mark the trained artificial intelligence model as the image reconstruction model.

3. The image super-resolution method based on progressive network modeling according to claim 1, characterized in that: Performing bicubic interpolation once is regarded as magnifying once, and the magnification factor is one time.

4. The image super-resolution method based on progressive network modeling according to claim 1, characterized in that: The method for performing coordinate association and positioning on all stored Mv images includes: Obtain the interpolation methods of all recorded Wv images, identify the Wv images that have not undergone bicubic interpolation, obtain the Wv´ images that have undergone bicubic interpolation and whose corresponding clarity does not meet the requirements, and mark the coordinates in the Wv´ images at the corresponding positions in the corresponding Wv images; Arrange all Wv images with the image center as the origin, in order from top to bottom according to the size of the image name, and then place the super-resolution image corresponding above the W1 image. Mark the coordinate points of the same pixel points in different Wv images as the same type of associated points, and associate all the coordinate points within the same type of associated points according to the arrangement order of the Wv images.

5. The image super-resolution method based on progressive network modeling according to claim 1, characterized in that: The shape of the self-positioning border is a regular shape.

6. The image super-resolution method based on progressive network modeling according to claim 1, characterized in that: The method for the scrolling unit to magnify or reduce the image within the self-positioning border includes: input magnification or reduction and drag magnification or reduction.

7. The image super-resolution method based on progressive network modeling according to claim 1, characterized in that: The method for performing super-resolution on the super-resolution image through the super-resolution scroll box includes: Set the shape of the super-resolution scrolling frame, place the super-resolution scrolling frame on the super-resolution image that needs to be super-resolution, set the size of the super-resolution scrolling frame, and after the setting is completed, identify the inflection point coordinates of the super-resolution scrolling frame, connect the identified coordinate points, and fix the super-resolution scrolling frame; mark the border formed by the connection of the coordinate points as a super-resolution frame, project the super-resolution frame on the super-resolution image set, select the super-resolution method, and super-resolve the super-resolution image in the super-resolution frame according to the selected super-resolution method.

Images