Image edge super-resolution enhancement system and method

Abstract

The present invention discloses an image edge super-resolution enhancement system and method, the image edge super-resolution enhancement system comprises a feature extraction module, a feature fusion module, a nonlinear mapping module and an image super-resolution reconstruction module; the feature extraction module is used to extract image features, edge features and gradient features from low-resolution images; the feature fusion module is used to effectively fuse the extracted image features, edge features and gradient features, so that the fused features cover more information and have better discrimination and reconstruction capabilities; the nonlinear mapping module is used to perform nonlinear transformation on the fused features fused by the feature fusion module, so that the features have stronger expression capabilities; the image super-resolution reconstruction module is used to perform high-resolution reconstruction on the images with the features nonlinearly transformed by the nonlinear mapping module. The present invention can improve the clarity and credibility of reconstructing high-resolution text images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to an image processing system, and in particular to an image edge super-resolution enhancement system and method. Background Art

[0002] As the original document that records the true face of history, archives play an important role in economic and social development. Since archives are inevitably affected by environmental factors such as temperature, humidity, and light during storage and utilization, paper archives and photo archives gradually fade, bleed, distort, yellow, break, mildew, and film failure. In order to rescue and protect the information of these precious historical archives, archive practitioners use equipment such as scanners or digital cameras to digitize archives and convert them into digital images stored on tapes, disks, CDs, etc., which are convenient for retrieval and use, thereby protecting the original archives. However, these digital images have defects such as monochrome, distortion, low pixels, and low resolution due to the low clarity of early scanning and shooting equipment, and the visual effect of digital images is poor. To solve such problems, manual adjustment and interpolation techniques are usually used to improve the resolution of image archives.

[0003] Manual adjustment uses tools such as Photoshop to adjust the brightness, saturation, histogram, mean, clarity, color difference, hue, pixels, etc. of electronic image archives to improve the clarity and visual effects of electronic archive images. However, due to manual adjustment of image parameters and diverse image information, it takes a lot of manpower costs and time cycles, and the efficiency is too low. Interpolation technology is used to improve the clarity of the image, including nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and resampling. The unknown pixels in the new image are calculated for the known pixels of the original image. After repair, some details of the image will be lost, such as high-frequency information loss. In addition, noise may be amplified during the interpolation process, resulting in side effects such as blurry reconstructed images. Interpolation technology is not only computationally complex, but also different interpolation methods will make the results of image reconstruction very different.

[0004] In view of this, there is an urgent need to design a new image processing method to overcome at least part of the above-mentioned defects of the existing image processing method. Summary of the invention

[0005] The present invention provides an image edge super-resolution enhancement system and method, which can improve the clarity and credibility of reconstructed high-resolution text images.

[0006] To solve the above technical problem, according to one aspect of the present invention, the following technical solution is adopted:

[0007] An image edge super-resolution enhancement system, the image edge super-resolution enhancement system comprising:

[0008] A feature extraction module is used to extract image features, edge features and gradient features from low-resolution images;

[0009] Feature fusion module, used to effectively fuse the extracted image features, edge features and gradient features, so that the fused features contain more information and have better discrimination and reconstruction capabilities;

[0010] A nonlinear mapping module, used to perform nonlinear transformation on the fused features after fusion by the feature fusion module, so that the features have stronger expression ability;

[0011] The image super-resolution reconstruction module is used to reconstruct the image with high resolution using the features after nonlinear transformation by the nonlinear mapping module.

[0012] As an implementation mode of the present invention, the feature extraction module includes:

[0013] An image feature extraction unit is used to extract image features; the image feature extraction unit includes a plurality of image feature extraction submodules, each of which includes a first convolution layer, a first activation layer and a first normalization layer; the plurality of image feature extraction submodules included in the image feature extraction unit include a shallow image feature extraction submodule and a deep image feature extraction submodule; the shallow image feature extraction submodule extracts low-level features of the image, including texture and color; the deep image feature extraction submodule extracts high-level features of the image, including image category and target number; the image is continuously feature extracted through a plurality of convolution submodules, so that the extracted convolution features have more discriminative power;

[0014] An edge feature extraction unit is used to extract edge features of an image; the edge feature extraction unit includes a plurality of edge feature extraction submodules; in order to more effectively utilize information between images and make the edge of the reconstructed image more credible, a recurrent neural network module RNN is added to the edge feature extraction unit; the recurrent neural network module includes a first input layer, a first output layer and a first hidden layer, and information is transmitted through the first hidden layer. The recurrent neural network module RNN memorizes the previous information and applies it to the calculation of the current output, that is, the nodes between the first hidden layers are no longer disconnected but connected, and the input of the first hidden layer includes not only the output of the first input layer, but also the output of the first hidden layer at the previous moment; in order to learn to extract accurate edge features of objects in the image, the output layer of the recurrent neural network module is analyzed for edge information, the loss function is calculated based on edge annotation, and the gradient is back-transferred to enable the extracted edge information to supervise the reconstruction process of image details, thereby increasing the credibility of the reconstructed image;

[0015] A gradient feature extraction unit, for extracting gradient features; the gradient feature extraction unit comprises a plurality of gradient feature extraction submodules, each of which comprises a second convolutional layer, a second activation layer, and a second normalization layer, and the gradient information of the image is extracted by effectively connecting the plurality of gradient feature extraction submodules;

[0016] When extracting gradient information, it is necessary to use the real gradient of the low-resolution image for guidance, so that the features extracted by the gradient module are accurate and effective. A gradient label is made for each high-resolution image. The gradient label is calculated using the existing gradient calculation method. Since the gradient information is extracted to produce smooth text edges, the gradient uses the Sobel operator for edge detection. The calculated gradient information and the extracted gradient features are used to calculate the gradient loss, and then the edge reconstruction of the image text is supervised by gradient return.

[0017] The Sobel operator in the edge detection algorithm mainly includes two operators, one operator detects horizontal edges and the other operator detects vertical edges. The Sobel operator weights the influence of the pixel position to reduce the edge blur. The Sobel operator contains two sets of 3x3 matrices, one for horizontal and one for vertical. The convolution operation is performed with the image to obtain the horizontal and vertical brightness difference approximations. If A represents the original image, G represents the vertical brightness difference. x and G y Respectively represent the images after horizontal and vertical edge detection, and the formula is as follows:

[0018]

[0019] The approximate horizontal and vertical gradients of each pixel of the image are combined using the following formula to calculate the magnitude of the gradient;

[0020]

[0021] Then calculate the gradient direction using the following formula;

[0022]

[0023] If the angle θ is equal to zero, it means that the image has a vertical edge at that location and is darker than the right side.

[0024] As an implementation mode of the present invention, the feature fusion module uses a bottleneck layer to reduce the dimension of simply fused features, thereby reducing the redundancy of feature information and reducing the amount of network calculation.

[0025] As an implementation mode of the present invention, the nonlinear mapping module includes a plurality of nonlinear mapping submodules, each of which includes a third convolutional layer, a third activation layer and a third normalization layer;

[0026] The nonlinear mapping module performs a deeper nonlinear transformation on the fused features, so that the features have stronger expressive power; the connection mode between different nonlinear mapping submodules includes at least one of residual connection and dense connection.

[0027] As an embodiment of the present invention, the image super-resolution reconstruction module includes a plurality of convolution submodules and an upsampling module, wherein the convolution submodule is used to extract effective features for the upsampling module; the upsampling module is used to map the extracted features into a high-resolution image of a target size, assign a value to each pixel in the image, and thereby obtain a reconstructed high-resolution target image;

[0028] In image super-resolution reconstruction, according to the upsampling position, SR deep learning methods are classified into four categories: pre-upsampling super-resolution, post-upsampling super-resolution, progressive upsampling super-resolution, and iterative up- and down-sampling super-resolution;

[0029] The image super-resolution reconstruction module adopts a post-upsampling framework. In order to improve computational efficiency and fully utilize deep learning technology to automatically increase resolution, an end-to-end learning layer is integrated at the end of the model to replace the predefined upsampling operation, so that most calculations are performed in a low-dimensional space. This framework is called a post-upsampling framework. The LR image does not need to increase resolution and can be directly input into the CNN network, and an end-to-end learnable upsampling layer is used at the end of the network. In the post-upsampling framework, feature extraction is performed in a low-dimensional space, reducing time and storage space costs.

[0030] According to another aspect of the present invention, the following technical solution is adopted: a method for image edge super-resolution enhancement, the method comprising:

[0031] Feature extraction step: The feature extraction module extracts image features, edge features and gradient features from the low-resolution image;

[0032] Feature fusion step: The feature fusion module fuses the extracted image features, edge features and gradient features, so that the fused features cover more information and have better discrimination and reconstruction capabilities;

[0033] Non-linear mapping step: the non-linear mapping module performs non-linear transformation on the fused features fused in the feature fusion step, so that the features have stronger expression ability;

[0034] Image super-resolution reconstruction step: The image super-resolution reconstruction module reconstructs the image with high resolution using the features that have been nonlinearly transformed in the nonlinear mapping step.

[0035] As an implementation mode of the present invention, the feature extraction step includes:

[0036] Image feature extraction step: an image feature extraction unit extracts image features; the image feature extraction unit includes a plurality of image feature extraction submodules, each of which includes a first convolution layer, a first activation layer, and a first normalization layer; the plurality of image feature extraction submodules included in the image feature extraction unit include a shallow image feature extraction submodule and a deep image feature extraction submodule; the shallow image feature extraction submodule extracts low-level features of the image, including texture and color; the deep image feature extraction submodule extracts high-level features of the image, including image category and target number; the image is continuously feature extracted through a plurality of convolution submodules, so that the extracted convolution features have more discriminative power;

[0037] Edge feature extraction step: the edge feature extraction unit extracts the edge features of the image; the edge feature extraction unit includes a plurality of edge feature extraction submodules; in order to more effectively utilize the information between images and make the edge of the reconstructed image more credible, a recurrent neural network module RNN is added to the edge feature extraction unit; the recurrent neural network module includes a first input layer, a first output layer and a first hidden layer, and the information is transmitted through the first hidden layer. The recurrent neural network module RNN memorizes the previous information and applies it to the calculation of the current output, that is, the nodes between the first hidden layers are no longer disconnected but connected, and the input of the first hidden layer includes not only the output of the first input layer, but also the output of the first hidden layer at the previous moment; in order to learn to extract the accurate edge features of the objects in the picture, the output layer of the recurrent neural network module is analyzed for edge information, the loss function is calculated based on the edge annotation, and the gradient is back-transferred to enable the extracted edge information to supervise the reconstruction process of the image details, thereby increasing the credibility of the reconstructed image;

[0038] Gradient feature extraction step: a gradient feature extraction unit extracts gradient features; the gradient feature extraction unit comprises a plurality of gradient feature extraction submodules, each of which comprises a second convolutional layer, a second activation layer, and a second normalization layer, and the gradient information of the image is extracted by effectively connecting the plurality of gradient feature extraction submodules;

[0039] When extracting gradient information, it is necessary to use the real gradient of the low-resolution image for guidance, so that the features extracted by the gradient module are accurate and effective. A gradient label is made for each high-resolution image. The gradient label is calculated using the existing gradient calculation method. Since the gradient information is extracted to produce smooth text edges, the gradient uses the Sobel operator for edge detection. The calculated gradient information and the extracted gradient features are used to calculate the gradient loss, and then the edge reconstruction of the image text is supervised by gradient return.

[0040] The Sobel operator in the edge detection algorithm mainly includes two operators, one operator detects horizontal edges and the other operator detects vertical edges. The Sobel operator weights the influence of the pixel position to reduce the edge blur. The Sobel operator contains two sets of 3x3 matrices, one for horizontal and one for vertical. The convolution operation is performed with the image to obtain the horizontal and vertical brightness difference approximations. If A represents the original image, G represents the vertical brightness difference. x and G y Respectively represent the images after horizontal and vertical edge detection, and the formula is as follows:

[0041]

[0042] The approximate horizontal and vertical gradients of each pixel of the image are combined using the following formula to calculate the magnitude of the gradient;

[0043]

[0044] Then calculate the gradient direction using the following formula;

[0045]

[0046] If the angle θ is equal to zero, it means that the image has a vertical edge at that location and is darker than the right side.

[0047] As an implementation mode of the present invention, in the feature fusion step, a bottleneck layer is used to reduce the dimension of simply fused features, thereby reducing the redundancy of feature information and reducing the computational complexity of the network.

[0048] As an implementation mode of the present invention, in the nonlinear mapping step, the nonlinear mapping module includes a plurality of nonlinear mapping submodules, each of which includes a third convolutional layer, a third activation layer and a third normalization layer;

[0049] The nonlinear mapping module performs a deeper nonlinear transformation on the fused features, so that the features have stronger expressive power; the connection mode between different nonlinear mapping submodules includes at least one of residual connection and dense connection.

[0050] As an embodiment of the present invention, in the image super-resolution reconstruction step, the image super-resolution reconstruction module includes a plurality of convolution submodules and an up-sampling module, the convolution submodule is used to extract effective features for the up-sampling module; the up-sampling module is used to map the extracted features into a high-resolution image of a target size, assign a value to each pixel in the image, thereby obtaining a reconstructed high-resolution target image;

[0051] In image super-resolution reconstruction, according to the upsampling position, SR deep learning methods are classified into four categories: pre-upsampling super-resolution, post-upsampling super-resolution, progressive upsampling super-resolution, and iterative up- and down-sampling super-resolution;

[0052] The image super-resolution reconstruction module adopts a post-upsampling framework. In order to improve computational efficiency and fully utilize deep learning technology to automatically increase resolution, an end-to-end learning layer is integrated at the end of the model to replace the predefined upsampling operation, so that most calculations are performed in a low-dimensional space. This framework is called a post-upsampling framework. The LR image does not need to increase resolution and can be directly input into the CNN network, and an end-to-end learnable upsampling layer is used at the end of the network. In the post-upsampling framework, feature extraction is performed in a low-dimensional space, reducing time and storage space costs.

[0053] According to another aspect of the present invention, the following technical solution is adopted: an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0054] According to another aspect of the present invention, the following technical solution is adopted: a storage medium stores computer program instructions, and the computer program instructions implement the steps of the above method when executed by a processor.

[0055] The beneficial effect of the present invention is that the image edge super-resolution enhancement system and method proposed by the present invention can improve the clarity and credibility of reconstructed high-resolution text images. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of the composition of an image edge super-resolution enhancement system in one embodiment of the present invention.

[0057] Figure 2 The figure is a flow chart of an image edge super-resolution enhancement method according to an embodiment of the present invention.

[0058] Figure 3 Schematic diagram of the working principle of an image feature extraction unit in one embodiment of the present invention.

[0059] Figure 4 Schematic diagram of the working principle of an edge feature extraction unit in one embodiment of the present invention.

[0060] Figure 5 Schematic diagram of the working principle of the recurrent neural network module in one embodiment of the present invention.

[0061] Figure 6 Schematic diagram of the working principle of a gradient feature extraction unit in one embodiment of the present invention.

[0062] Figure 7 Schematic diagram of the working principle of the feature fusion module in one embodiment of the present invention.

[0063] Figure 8 FIG. 4 is a schematic diagram of the working principle of a nonlinear mapping module in an embodiment of the present invention.

[0064] Fig. 9 Schematic diagram of residual connection of each submodule in a nonlinear mapping process in one embodiment of the present invention.

[0065] Fig.10 It is a schematic diagram of dense connection of submodules in a nonlinear mapping process in one embodiment of the present invention.

[0066] Fig.11 Schematic diagram of the working principle of an image super-resolution reconstruction module in one embodiment of the present invention.

[0067] Fig.12 FIG. 4 is a schematic diagram of the composition of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0069] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0070] The description in this section is only for several typical embodiments, and the present invention is not limited to the scope of the embodiments. The same or similar prior art means and some technical features in the embodiments are mutually replaced within the scope of the present invention.

[0071] The description of the steps in each embodiment in the specification is only for the convenience of explanation, and the implementation method of the present application is not limited by the order of implementation of the steps.

[0072] The term “connection” in the specification includes both direct connection and indirect connection.

[0073] The present invention discloses an image edge super-resolution enhancement system. Figure 1 FIG. 1 is a schematic diagram of the composition of an image edge super-resolution enhancement system according to an embodiment of the present invention; please refer to FIG. Figure 1 The image edge super-resolution enhancement system includes: a feature extraction module 1, a feature fusion module 2, a nonlinear mapping module 3 and an image super-resolution reconstruction module 4.

[0074] The feature extraction module 1 is used to extract image features, edge features and gradient features from low-resolution images; the feature fusion module 2 is used to effectively fuse the extracted image features, edge features and gradient features, so that the fused features cover more information and have better discrimination and reconstruction capabilities. The nonlinear mapping module 3 is used to perform nonlinear transformation on the fused features after fusion by the feature fusion module, so that the features have stronger expression capabilities; the image super-resolution reconstruction module 4 is used to perform high-resolution reconstruction on the image using the features after nonlinear transformation by the nonlinear mapping module.

[0075] In one embodiment of the present invention, the feature extraction module 1 includes: an image feature extraction unit 11 , an edge feature extraction unit 12 and a gradient feature extraction unit 13 .

[0076] Figure 3 is a schematic diagram of the working principle of an image feature extraction unit in an embodiment of the present invention; please refer to Figure 3 In one embodiment of the present invention, the image feature extraction unit 11 is used to extract image features; the image feature extraction unit includes multiple image feature extraction submodules, each of which includes a first convolution layer, a first activation layer and a first normalization layer; the multiple image feature extraction submodules included in the image feature extraction unit include a shallow image feature extraction submodule and a deep image feature extraction submodule; the shallow image feature extraction submodule extracts low-level features of the image, including texture and color; the deep image feature extraction submodule extracts high-level features of the image, including image category and target number; the image is continuously feature extracted through multiple convolution submodules, so that the extracted convolution features are more discriminative.

[0077] Figure 4 FIG. 1 is a schematic diagram showing the working principle of an edge feature extraction unit in an embodiment of the present invention; please refer to FIG. Figure 4 In one embodiment of the present invention, the edge feature extraction unit 12 is used to extract edge features of an image; the edge feature extraction unit includes a plurality of edge feature extraction submodules; in order to more effectively utilize information between images and make the reconstructed image edge more credible, a recurrent neural network module RNN is added to the edge feature extraction unit.

[0078] Figure 5 This is a schematic diagram of the working principle of the recurrent neural network module in one embodiment of the present invention; please refer to Figure 5In one embodiment of the present invention, the recurrent neural network module includes a first input layer, a first output layer and a first hidden layer. Information is transmitted through the first hidden layer, and the recurrent neural network module RNN memorizes the previous information and applies it to the calculation of the current output, that is, the nodes between the first hidden layers are no longer disconnected but connected, and the input of the first hidden layer includes not only the output of the first input layer, but also the output of the first hidden layer at the previous moment; in order to learn to extract accurate edge features of objects in the picture, the output layer of the recurrent neural network module is analyzed for edge information, and the loss function is calculated based on edge annotation. By using gradient backpropagation, the extracted edge information supervises the reconstruction process of image details, thereby increasing the credibility of the reconstructed image.

[0079] Figure 6 FIG. 1 is a schematic diagram showing the working principle of a gradient feature extraction unit in an embodiment of the present invention; please refer to FIG. Figure 6 In one embodiment of the present invention, the gradient feature extraction unit 13 is used to extract gradient features; the gradient feature extraction unit includes a plurality of gradient feature extraction submodules, each of which includes a second convolutional layer, a second activation layer, and a second normalization layer. The gradient information of the image is extracted by effectively connecting the plurality of gradient feature extraction submodules.

[0080] When extracting gradient information, it is necessary to use the real gradient of the low-resolution image for guidance, so that the features extracted by the gradient module are accurate and effective. A gradient label is made for each high-resolution image. The gradient label is calculated using the existing gradient calculation method. Since the gradient information is extracted to produce smooth text edges, the gradient uses the Sobel operator for edge detection. The calculated gradient information and the extracted gradient features are used to calculate the gradient loss, and then the edge reconstruction of the image text is supervised by gradient feedback.

[0081] The Sobel operator in the edge detection algorithm mainly includes two operators, one operator detects horizontal edges and the other operator detects vertical edges. The Sobel operator weights the influence of the pixel position to reduce the edge blur. The Sobel operator contains two sets of 3x3 matrices, one for horizontal and one for vertical. The convolution operation is performed with the image to obtain the horizontal and vertical brightness difference approximations. If A represents the original image, G represents the vertical brightness difference. x and G y Respectively represent the images after horizontal and vertical edge detection, and the formula is as follows:

[0082]

[0083] The approximate horizontal and vertical gradients of each pixel of the image are combined using the following formula to calculate the magnitude of the gradient;

[0084]

[0085] Then calculate the gradient direction using the following formula;

[0086]

[0087] If the angle θ is equal to zero, it means that the image has a vertical edge at that location and is darker than the right side.

[0088] Figure 7 This is a schematic diagram of the working principle of the feature fusion module in one embodiment of the present invention; please refer to Figure 7 In one embodiment of the present invention, the feature fusion module 2 uses a bottleneck layer to reduce the dimension of the simply fused features, thereby reducing the redundancy of feature information and reducing the amount of network calculation.

[0089] Figure 8 This is a schematic diagram of the working principle of the nonlinear mapping module in one embodiment of the present invention; please refer to Figure 8 In one embodiment of the present invention, the nonlinear mapping module 3 includes a plurality of nonlinear mapping submodules, each of which includes a third convolutional layer, a third activation layer, and a third normalization layer; the nonlinear mapping module performs a deeper nonlinear transformation on the fusion features so that the features have stronger expression capabilities; the connection mode between different nonlinear mapping submodules includes residual connection (such as Fig. 9 as shown), densely connected (as shown Fig.10 At least one of the above.

[0090] Fig.11 FIG. 1 is a schematic diagram showing the working principle of an image super-resolution reconstruction module in an embodiment of the present invention; please refer to FIG. Fig.11 In one embodiment of the present invention, the image super-resolution reconstruction module 4 includes a plurality of convolution submodules and an upsampling module. The convolution submodule is used to extract effective features for the upsampling module; the upsampling module is used to map the extracted features into a high-resolution image of the target size, assign a value to each pixel in the image, and thereby obtain a reconstructed high-resolution target image.

[0091] In image super-resolution reconstruction, SR deep learning methods are classified into four categories according to the upsampling position: pre-upsampling super-resolution, post-upsampling super-resolution, progressive upsampling super-resolution, and iterative up- and down-sampling super-resolution.

[0092] The image super-resolution reconstruction module 4 adopts a post-upsampling framework. In order to improve computational efficiency and fully utilize deep learning technology to automatically increase resolution, an end-to-end learning layer is integrated at the end of the model to replace the predefined upsampling operation, so that most calculations are performed in a low-dimensional space. This framework is called a post-upsampling framework; the LR image does not need to increase the resolution and can be directly input into the CNN network, and an end-to-end learnable upsampling layer is used at the end of the network; in the post-upsampling framework, feature extraction is performed in a low-dimensional space, reducing time and storage space costs.

[0093] The present invention further discloses a method for image edge super-resolution enhancement. Figure 2 is a flow chart of an image edge super-resolution enhancement method according to an embodiment of the present invention; Figure 2 , the image edge super-resolution enhancement method comprises:

[0094] [Step S1] Feature extraction step; the feature extraction module extracts image features, edge features and gradient features from the low-resolution image.

[0095] In one embodiment of the present invention, the feature extraction step includes:

[0096] Image feature extraction step: an image feature extraction unit extracts image features; the image feature extraction unit includes a plurality of image feature extraction submodules, each of which includes a first convolution layer, a first activation layer, and a first normalization layer; the plurality of image feature extraction submodules included in the image feature extraction unit include a shallow image feature extraction submodule and a deep image feature extraction submodule; the shallow image feature extraction submodule extracts low-level features of the image, including texture and color; the deep image feature extraction submodule extracts high-level features of the image, including image category and target number; the image is continuously feature extracted through a plurality of convolution submodules, so that the extracted convolution features have more discriminative power;

[0097] Edge feature extraction step: the edge feature extraction unit extracts the edge features of the image; the edge feature extraction unit includes a plurality of edge feature extraction submodules; in order to more effectively utilize the information between images and make the edge of the reconstructed image more credible, a recurrent neural network module RNN is added to the edge feature extraction unit; the recurrent neural network module includes a first input layer, a first output layer and a first hidden layer, and the information is transmitted through the first hidden layer. The recurrent neural network module RNN memorizes the previous information and applies it to the calculation of the current output, that is, the nodes between the first hidden layers are no longer disconnected but connected, and the input of the first hidden layer includes not only the output of the first input layer, but also the output of the first hidden layer at the previous moment; in order to learn to extract the accurate edge features of the objects in the picture, the output layer of the recurrent neural network module is analyzed for edge information, the loss function is calculated based on the edge annotation, and the gradient is back-transferred to enable the extracted edge information to supervise the reconstruction process of the image details, thereby increasing the credibility of the reconstructed image;

[0098] Gradient feature extraction step: a gradient feature extraction unit extracts gradient features; the gradient feature extraction unit comprises a plurality of gradient feature extraction submodules, each of which comprises a second convolutional layer, a second activation layer, and a second normalization layer, and the gradient information of the image is extracted by effectively connecting the plurality of gradient feature extraction submodules;

[0099] When extracting gradient information, it is necessary to use the real gradient of the low-resolution image for guidance, so that the features extracted by the gradient module are accurate and effective. A gradient label is made for each high-resolution image. The gradient label is calculated using the existing gradient calculation method. Since the gradient information is extracted to produce smooth text edges, the gradient uses the Sobel operator for edge detection. The calculated gradient information and the extracted gradient features are used to calculate the gradient loss, and then the edge reconstruction of the image text is supervised by gradient return.

[0100] The Sobel operator in the edge detection algorithm mainly includes two operators, one operator detects horizontal edges and the other operator detects vertical edges. The Sobel operator weights the influence of the pixel position to reduce the edge blur. The Sobel operator contains two sets of 3x3 matrices, one for horizontal and one for vertical. The convolution operation is performed with the image to obtain the horizontal and vertical brightness difference approximations. If A represents the original image, G represents the vertical brightness difference. x and G y Respectively represent the images after horizontal and vertical edge detection, and the formula is as follows:

[0101]

[0102] The approximate horizontal and vertical gradients of each pixel of the image are combined using the following formula to calculate the magnitude of the gradient;

[0103]

[0104] Then calculate the gradient direction using the following formula;

[0105]

[0106] If the angle θ is equal to zero, it means that the image has a vertical edge at that location and is darker than the right side.

[0107] [Step S2] Feature fusion step: The feature fusion module fuses the extracted image features, edge features and gradient features so that the fused features cover more information and have better discrimination and reconstruction capabilities.

[0108] In one embodiment of the present invention, a bottleneck layer is used to reduce the dimension of simply fused features, thereby reducing the redundancy of feature information and reducing the amount of network calculation.

[0109] [Step S3] Non-linear mapping step; the non-linear mapping module performs non-linear transformation on the fused features fused in the feature fusion step, so that the features have stronger expression capabilities.

[0110] In one embodiment of the present invention, in the nonlinear mapping step, the nonlinear mapping module includes a plurality of nonlinear mapping submodules, each of which includes a third convolutional layer, a third activation layer, and a third normalization layer. The nonlinear mapping module performs a deeper nonlinear transformation on the fusion features so that the features have stronger expression capabilities; the connection mode between different nonlinear mapping submodules includes at least one of residual connection and dense connection.

[0111] [Step S4] Image super-resolution reconstruction step; the image super-resolution reconstruction module reconstructs the image with high resolution using the features after nonlinear transformation in the nonlinear mapping step.

[0112] In one embodiment of the present invention, in the image super-resolution reconstruction step, the image super-resolution reconstruction module includes multiple convolution sub-modules and an up-sampling module, the convolution sub-module is used to extract effective features for the up-sampling module; the up-sampling module is used to map the extracted features into a high-resolution image of the target size, assign a value to each pixel in the image, and thus obtain a reconstructed high-resolution target image.

[0113] In image super-resolution reconstruction, SR deep learning methods are classified into four categories according to the upsampling position: pre-upsampling super-resolution, post-upsampling super-resolution, progressive upsampling super-resolution, and iterative up- and down-sampling super-resolution.

[0114] The image super-resolution reconstruction module adopts a post-upsampling framework. In order to improve computational efficiency and fully utilize deep learning technology to automatically increase resolution, an end-to-end learning layer is integrated at the end of the model to replace the predefined upsampling operation, so that most calculations are performed in a low-dimensional space. This framework is called a post-upsampling framework. The LR image does not need to increase resolution and can be directly input into the CNN network, and an end-to-end learnable upsampling layer is used at the end of the network. In the post-upsampling framework, feature extraction is performed in a low-dimensional space, reducing time and storage space costs.

[0115] The present invention also discloses an electronic device, Fig.12 FIG. 1 is a schematic diagram of the composition of an electronic device in an embodiment of the present invention; please refer to FIG. Fig.12 At the hardware level, the electronic device includes a memory, a processor and at least one network interface; the processor may be a microprocessor, and the memory may include a memory, such as a random access memory (RAM), or a non-volatile memory, etc. Of course, the electronic device may also be provided with other hardware as required.

[0116] The processor, network interface and memory may be interconnected via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus may include an address bus, a data bus, a control bus, etc. The memory is used to store programs (which may include operating system programs and application programs); the program may include program code, which may include computer operation instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0117] In one embodiment, the processor can read the corresponding program from the non-volatile memory into the memory and then run it; the processor can execute the program stored in the memory and is specifically used to perform the following operations (such as Figure 2 shown):

[0118] [Step S1] Feature extraction step; the feature extraction module extracts image features, edge features and gradient features from the low-resolution image;

[0119] [Step S2] Feature fusion step: The feature fusion module fuses the extracted image features, edge features and gradient features, so that the fused features contain more information and have better discrimination and reconstruction capabilities;

[0120] [Step S3] Non-linear mapping step: the non-linear mapping module performs non-linear transformation on the fused features fused in the feature fusion step, so that the features have stronger expression ability;

[0121] [Step S4] Image super-resolution reconstruction step; the image super-resolution reconstruction module reconstructs the image with high resolution using the features after nonlinear transformation in the nonlinear mapping step.

[0122] The present invention further discloses a storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the following steps of the image edge super-resolution enhancement method of the present invention (eg Figure 2 shown):

[0123] [Step S1] Feature extraction step; the feature extraction module extracts image features, edge features and gradient features from the low-resolution image;

[0124] [Step S2] Feature fusion step: The feature fusion module fuses the extracted image features, edge features and gradient features, so that the fused features contain more information and have better discrimination and reconstruction capabilities;

[0125] [Step S3] Non-linear mapping step: the non-linear mapping module performs non-linear transformation on the fused features fused in the feature fusion step, so that the features have stronger expression ability;

[0126] [Step S4] Image super-resolution reconstruction step; the image super-resolution reconstruction module reconstructs the image with high resolution using the features after nonlinear transformation in the nonlinear mapping step.

[0127] In summary, the image edge super-resolution enhancement system and method proposed in the present invention can improve the clarity and credibility of reconstructed high-resolution text images.

[0128] It should be noted that the present application can be implemented in software and / or a combination of software and hardware; for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium; for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented in hardware; for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0129] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The description and application of the present invention here are illustrative, and it is not intended to limit the scope of the present invention to the above-mentioned embodiments. The effects or advantages involved in the embodiments may not be embodied in the embodiments due to interference from various factors, and the description of the effects or advantages is not used to limit the embodiments. The deformation and change of the embodiments disclosed here are possible, and the replacement of the embodiments and the various equivalent parts are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential features of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. An image edge super-resolution enhancement system, characterized in that: The image edge super-resolution enhancement system comprises: A feature extraction module is used to extract image features, edge features and gradient features from low-resolution images; Feature fusion module, used to fuse the extracted image features, edge features and gradient features, so that the fused features contain more information and have better discrimination and reconstruction capabilities; A non-linear mapping module, used for performing non-linear transformation on the fused features fused by the feature fusion module, so that the features have stronger expression ability; and An image super-resolution reconstruction module, used to reconstruct the image with high resolution using the features after nonlinear transformation by the nonlinear mapping module; The feature extraction module comprises: An image feature extraction unit is used to extract image features; the image feature extraction unit includes a plurality of image feature extraction submodules, each of which includes a first convolution layer, a first activation layer and a first normalization layer; the plurality of image feature extraction submodules included in the image feature extraction unit include a shallow image feature extraction submodule and a deep image feature extraction submodule; the shallow image feature extraction submodule extracts low-level features of the image, including texture and color; the deep image feature extraction submodule extracts high-level features of the image, including image category and target number; the image is continuously feature extracted through a plurality of convolution submodules, so that the extracted convolution features have more discriminative power; An edge feature extraction unit is used to extract edge features of an image; the edge feature extraction unit includes a plurality of edge feature extraction submodules; in order to more effectively utilize information between images and make the edge of the reconstructed image more credible, a recurrent neural network module RNN is added to the edge feature extraction unit; the recurrent neural network module includes a first input layer, a first output layer and a first hidden layer, and information is transmitted through the first hidden layer. The recurrent neural network module RNN memorizes the previous information and applies it to the calculation of the current output, that is, the nodes between the first hidden layers are no longer disconnected but connected, and the input of the first hidden layer includes not only the output of the first input layer, but also the output of the first hidden layer at the previous moment; in order to learn to extract accurate edge features of objects in the picture, the output layer of the recurrent neural network module is analyzed for edge information, the loss function is calculated based on edge annotation, and the gradient is back-transferred to enable the extracted edge information to supervise the reconstruction process of image details, thereby increasing the credibility of the reconstructed image; A gradient feature extraction unit, for extracting gradient features; the gradient feature extraction unit comprises a plurality of gradient feature extraction submodules, each of which comprises a second convolutional layer, a second activation layer, and a second normalization layer, and the gradient information of the image is extracted by effectively connecting the plurality of gradient feature extraction submodules; When extracting gradient information, the real gradient of the low-resolution image is used as a guide, so that the features extracted by the gradient module are accurate and effective. A gradient label is made for each high-resolution image. The gradient label is calculated using the existing gradient calculation method. Since the gradient information is extracted to produce smooth text edges, the gradient uses the Sobel operator for edge detection. The calculated gradient information and the extracted gradient features are used to calculate the gradient loss, and then the edge reconstruction of the image text is supervised by gradient feedback. The image super-resolution reconstruction module includes a plurality of convolution submodules and an up-sampling module. The convolution submodule is used to extract effective features for the up-sampling module; the up-sampling module is used to map the extracted features into a high-resolution image of a target size, assign a value to each pixel in the image, and thereby obtain a reconstructed high-resolution target image; In image super-resolution reconstruction, according to the upsampling position, SR deep learning methods are classified into four categories: pre-upsampling super-resolution, post-upsampling super-resolution, progressive upsampling super-resolution, and iterative up- and down-sampling super-resolution; The image super-resolution reconstruction module adopts a post-upsampling framework. In order to improve computational efficiency and fully utilize deep learning technology to automatically increase resolution, an end-to-end learning layer is integrated at the end of the model to replace the predefined upsampling operation, so that most calculations are performed in a low-dimensional space. This framework is called a post-upsampling framework. The LR image does not need to increase resolution and can be directly input into the CNN network, and an end-to-end learnable upsampling layer is used at the end of the network. In the post-upsampling framework, feature extraction is performed in a low-dimensional space, reducing time and storage space costs.

2. The image edge super-resolution enhancement system according to claim 1, characterized in that: The Sobel operator in the edge detection algorithm mainly includes two operators, one operator detects horizontal edges and the other operator detects vertical edges; the Sobel operator weights the influence of the pixel position to reduce the edge blur; the Sobel operator contains two sets of 3x3 matrices, one for horizontal and one for vertical, which are convolved with the image to obtain the approximate values ​​of the horizontal and vertical brightness differences respectively; If A represents the original image, G x and G y Respectively represent the images after horizontal and vertical edge detection, and the formula is as follows: The approximate horizontal and vertical gradients of each pixel of the image are combined using the following formula to calculate the magnitude of the gradient; Then calculate the gradient direction using the following formula; If the angle θ is equal to zero, it means that the image has a vertical edge at that location and is darker than the right side.

3. The image edge super-resolution enhancement system according to claim 1, characterized in that: The feature fusion module uses a bottleneck layer to reduce the dimension of simple fused features, reduce the redundancy of feature information, and reduce the amount of network calculation; The nonlinear mapping module includes a plurality of nonlinear mapping submodules, each of which includes a third convolutional layer, a third activation layer and a third normalization layer; The nonlinear mapping module performs a deeper nonlinear transformation on the fused features, so that the features have stronger expressive power; the connection mode between different nonlinear mapping submodules includes at least one of residual connection and dense connection.

4. A method for image edge super-resolution enhancement, characterized in that: The image edge super-resolution enhancement method comprises: Feature extraction step: The feature extraction module extracts image features, edge features and gradient features from the low-resolution image; Feature fusion step: The feature fusion module fuses the extracted image features, edge features and gradient features, so that the fused features cover more information and have better discrimination and reconstruction capabilities; Non-linear mapping step: the non-linear mapping module performs non-linear transformation on the fused features fused in the feature fusion step, so that the features have stronger expression ability; An image super-resolution reconstruction step; an image super-resolution reconstruction module reconstructs the image with high resolution using the features after nonlinear transformation in the nonlinear mapping step; The feature extraction step comprises: Image feature extraction step: an image feature extraction unit extracts image features; the image feature extraction unit includes a plurality of image feature extraction submodules, each of which includes a first convolution layer, a first activation layer, and a first normalization layer; the plurality of image feature extraction submodules included in the image feature extraction unit include a shallow image feature extraction submodule and a deep image feature extraction submodule; the shallow image feature extraction submodule extracts low-level features of the image, including texture and color; the deep image feature extraction submodule extracts high-level features of the image, including image category and target number; the image is continuously feature extracted through a plurality of convolution submodules, so that the extracted convolution features have more discriminative power; Edge feature extraction step: the edge feature extraction unit extracts the edge features of the image; the edge feature extraction unit includes a plurality of edge feature extraction submodules; in order to more effectively utilize the information between images and make the edge of the reconstructed image more credible, a recurrent neural network module RNN is added to the edge feature extraction unit; the recurrent neural network module includes a first input layer, a first output layer and a first hidden layer, and the information is transmitted through the first hidden layer. The recurrent neural network module RNN memorizes the previous information and applies it to the calculation of the current output, that is, the nodes between the first hidden layers are no longer disconnected but connected, and the input of the first hidden layer includes not only the output of the first input layer, but also the output of the first hidden layer at the previous moment; in order to learn to extract the accurate edge features of the objects in the picture, the output layer of the recurrent neural network module is analyzed for edge information, the loss function is calculated based on the edge annotation, and the gradient is back-transferred to enable the extracted edge information to supervise the reconstruction process of the image details, thereby increasing the credibility of the reconstructed image; Gradient feature extraction step: a gradient feature extraction unit extracts gradient features; the gradient feature extraction unit comprises a plurality of gradient feature extraction submodules, each of which comprises a second convolutional layer, a second activation layer, and a second normalization layer, and the gradient information of the image is extracted by effectively connecting the plurality of gradient feature extraction submodules; When extracting gradient information, the real gradient of the low-resolution image is used as a guide, so that the features extracted by the gradient module are accurate and effective. A gradient label is made for each high-resolution image. The gradient label is calculated using the existing gradient calculation method. Since the gradient information is extracted to produce smooth text edges, the gradient uses the Sobel operator for edge detection. The calculated gradient information and the extracted gradient features are used to calculate the gradient loss, and then the edge reconstruction of the image text is supervised by gradient feedback. In the image super-resolution reconstruction step, the image super-resolution reconstruction module includes a plurality of convolution submodules and an up-sampling module, wherein the convolution submodule is used to extract effective features for the up-sampling module; the up-sampling module is used to map the extracted features into a high-resolution image of a target size, and assign a value to each pixel in the image, thereby obtaining a reconstructed high-resolution target image; In image super-resolution reconstruction, according to the upsampling position, SR deep learning methods are classified into four categories: pre-upsampling super-resolution, post-upsampling super-resolution, progressive upsampling super-resolution, and iterative up- and down-sampling super-resolution; The image super-resolution reconstruction module adopts a post-upsampling framework. In order to improve computational efficiency and fully utilize deep learning technology to automatically increase resolution, an end-to-end learning layer is integrated at the end of the model to replace the predefined upsampling operation, so that most calculations are performed in a low-dimensional space. This framework is called a post-upsampling framework. The LR image does not need to increase resolution and can be directly input into the CNN network, and an end-to-end learnable upsampling layer is used at the end of the network. In the post-upsampling framework, feature extraction is performed in a low-dimensional space, reducing time and storage space costs.

5. The image edge super-resolution enhancement method according to claim 4, characterized in that: The Sobel operator in the edge detection algorithm mainly includes two operators, one operator detects horizontal edges and the other operator detects vertical edges; the Sobel operator weights the influence of the pixel position to reduce the edge blur; the Sobel operator contains two sets of 3x3 matrices, one for horizontal and one for vertical, which are convolved with the image to obtain the approximate values ​​of the horizontal and vertical brightness differences respectively; If A represents the original image, G x and G y Respectively represent the images after horizontal and vertical edge detection, and the formula is as follows: The approximate horizontal and vertical gradients of each pixel of the image are combined using the following formula to calculate the magnitude of the gradient; Then calculate the gradient direction using the following formula; If the angle θ is equal to zero, it means that the image has a vertical edge at that location and is darker than the right side.

6. The image edge super-resolution enhancement method according to claim 4, characterized in that: In the feature fusion step, a bottleneck layer is used to reduce the dimension of the simply fused features, thereby reducing the redundancy of feature information and reducing the amount of network calculation; In the nonlinear mapping step, the nonlinear mapping module includes a plurality of nonlinear mapping submodules, each of which includes a third convolutional layer, a third activation layer and a third normalization layer; The nonlinear mapping module performs a deeper nonlinear transformation on the fused features, so that the features have stronger expressive power; the connection mode between different nonlinear mapping submodules includes at least one of residual connection and dense connection.

Images