Terahertz super-resolution reconstruction method and system based on feature convolutional neural network

Through the super-resolution reconstruction method of feature convolution neural network, the problems of low image contrast and blurred defect profile in terahertz imaging are solved, high-resolution defect detection is achieved, and the stability and accuracy of the network are enhanced.

CN114820308BActive Publication Date: 2024-11-08GUILIN UNIV OF ELECTRONIC TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210377676.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-12
Publication Date
2024-11-08
Estimated Expiration
2042-04-12

AI Technical Summary

Technical Problem

In terahertz imaging technology, due to background noise and hardware vibration, the image contrast is low and the defect profile is blurred, making it difficult to find small-size defects. Gradient explosion or disappearance may occur during network training, resulting in weight parameters destruction.

Method used

The super-resolution reconstruction method based on feature convolution neural network is adopted to obtain high-resolution, low-resolution and defect image sets, edge detection, downsampling processing and super-resolution reconstruction are performed to generate feature convolution operators and weight matrix. Reduce noise interference and improve the contour resolution of defective images.

Benefits of technology

The resolution and detection accuracy of the terahertz defect profile are improved, and the super-resolution reconstruction effect is stable, which avoids the fuse of the weight nodes, and enhances the depth and generalization capabilities of the feature convolution neural network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114820308B_ABST
    Figure CN114820308B_ABST
Patent Text Reader

Abstract

The present invention discloses a terahertz super-resolution reconstruction method, system and computer-readable storage medium based on a feature convolutional neural network. The feature convolutional neural network includes an edge detection module, a downsampling module and a super-resolution reconstruction module. The terahertz super-resolution reconstruction method includes: obtaining a training sample set, inputting a defect image set in the training sample set into an edge detection module to generate a binary edge image set; inputting the binary edge image set into a downsampling module for downsampling processing to obtain a feature convolution kernel operator; inputting the training sample set into a super-resolution reconstruction module, so that the super-resolution reconstruction module trains a high-resolution image set and a low-resolution image set according to the feature convolution kernel operator to generate a weight matrix; and generating a target defect image corresponding to the training sample set according to the weight matrix. In an embodiment of the present invention, terahertz super-resolution reconstruction is realized based on a feature convolutional neural network, which can improve the terahertz defect contour resolution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a terahertz super-resolution reconstruction method, system and computer-readable storage medium based on a feature convolutional neural network. Background Art

[0002] Terahertz nondestructive testing mainly uses terahertz time-domain spectroscopy technology to scan the sample to be inspected point by point to obtain the time domain signal of each point of the sample. The terahertz frequency domain spectrum can be obtained through fast Fourier transform, and then the terahertz image is obtained using the relevant imaging algorithm. The difference in terahertz spectral characteristics between the defect and the sample is used as the characteristic parameter to identify the defects on the surface and inside of the sample from the image.

[0003] However, terahertz imaging usually relies on the acquired time domain signals. Background noise and hardware vibration will have a great impact on the terahertz signal, which in turn affects the imaging quality, resulting in low image contrast, blurred defect outlines, and difficulty in detecting small-size defects. In addition, since terahertz images usually rely on spectral characteristics and optical characteristic parameters for imaging, bad point signal data may cause gradient explosion or disappearance during network training, resulting in serious damage to the network weight parameters. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art, and provides a terahertz super-resolution reconstruction method, system and computer-readable storage medium based on a feature convolutional neural network, which can improve the resolution of terahertz defect profiles.

[0005] In a first aspect, the present invention provides a terahertz super-resolution reconstruction method based on a feature convolutional neural network, wherein the feature convolutional neural network includes an edge detection module, a downsampling module and a super-resolution reconstruction module; the terahertz super-resolution reconstruction method includes:

[0006] Acquire a training sample set, wherein the training sample set includes a high-resolution image set, a low-resolution image set, and a defect image set, and the defect image set is obtained by performing terahertz imaging processing on the raw material;

[0007] Inputting the defect image set in the training sample set into the edge detection module, so that the edge detection module generates a binary edge image set based on an edge detection algorithm;

[0008] Inputting the binary edge image set into the downsampling module, so that the downsampling module performs downsampling processing on the binary edge image set to obtain a feature convolution kernel operator;

[0009] Inputting the training sample set into the super-resolution reconstruction module, so that the super-resolution reconstruction module trains the high-resolution image set and the low-resolution image set according to the characteristic convolution kernel operator to generate a weight matrix;

[0010] A target defect image corresponding to the training sample set is generated according to the weight matrix.

[0011] The terahertz super-resolution reconstruction method based on the feature convolutional neural network provided by the embodiment of the present invention has at least the following beneficial effects: first, a training sample set including a high-resolution image set, a low-resolution image set and a defect image set is obtained, and the defect image set is input into the edge detection module to obtain a binary edge image set corresponding to the defect image set, thereby realizing the extraction of the contour edge of the defect image, which is convenient for the subsequent detection of the defect image; thereafter, the binary edge image set is input into the downsampling module for processing to obtain a feature convolution kernel operator, thereby improving the depth of the feature convolutional neural network; the training sample set is input into the super-resolution reconstruction module for training to obtain a weight matrix between the high-resolution image set and the low-resolution image set; finally, a target defect image corresponding to the training sample set is generated according to the weight matrix; the edge detection module, the downsampling module and the super-resolution reconstruction module are trained by the training sample set to obtain the contour resolution of the defect image in the training sample set, thereby achieving a stable super-resolution reconstruction effect.

[0012] According to some embodiments of the present invention, the training sample set is obtained by the following steps, including: performing imaging processing on the raw materials to obtain a high-resolution image set, wherein the high-resolution image set includes multiple original photos; scaling the original photos in the high-resolution image set based on an image interpolation algorithm, and restoring the scaled original photos to obtain a low-resolution image set corresponding to the high-resolution image set; performing terahertz imaging processing on the raw materials to obtain the defect image set; integrating the high-resolution image set, the low-resolution image set and the defect image set to obtain the training sample set, and implementing the process of obtaining the low-resolution image set based on the high-resolution image set through an image difference algorithm, thereby enhancing the contrast effect between the high-resolution image set and the low-resolution image set.

[0013] According to some embodiments of the present invention, the edge detection module includes a horizontal detection matrix and a vertical detection matrix; the step of inputting the defect image set into the edge detection module so that the edge detection module generates a binary edge image set based on an edge detection algorithm comprises: inputting the defect image set into the edge detection module so that the edge detection module performs a planar convolution on the horizontal detection matrix, the vertical detection matrix and the defect image set to obtain a brightness difference approximation; extracting defect contours in the defect image set based on the overall nested edge algorithm and the brightness difference approximation to obtain the binary edge image set, and removing the interference of convolution noise through the edge detection module to eliminate the influence of convolution noise.

[0014] According to some embodiments of the present invention, the overall nested edge algorithm includes a branch function and a weighting function, wherein the branch function is used to characterize the output of each branch of the feature convolutional neural network, and the weighting function is used to characterize the weight of each branch of the feature convolutional neural network; the defect contours in the defect image set are extracted based on the overall nested edge algorithm and the brightness difference approximation to obtain the binary edge image set, including: calculating the defect images in the defect image set according to the branch function and the weighting function so that all the defect images in the defect image set have the same size; performing binary image projection on the calculated defect image set to obtain the binary edge image set, and using the overall nested edge algorithm to make the defect images of all input edge detection modules and the output defect images have the same size, thereby improving the accuracy of extracting the defect image contours.

[0015] According to some embodiments of the present invention, the step of inputting the binary edge image set into the downsampling module so that the downsampling module performs downsampling processing on the binary edge image set to obtain a feature convolution kernel operator includes: inputting the binary edge image set into the downsampling module so that the convolution layer of the downsampling module performs batch normalization processing on the binary edge image set, and calculating the processed binary edge image set based on a linear rectification function to obtain the feature convolution kernel operator. The feature convolution kernel operator is calculated based on the binary edge image set, which improves the depth of the feature convolution neural network and improves the generalization ability of the feature convolution neural network.

[0016] According to some embodiments of the present invention, the convolution layer includes multiple layers of standard convolution layers and target convolution layers; the binary edge image set is input into the downsampling module, the convolution layer of the downsampling module performs batch normalization on the binary edge image set, and the processed binary edge image set is calculated based on a linear rectification function to obtain the feature convolution kernel operator, including: inputting the binary edge image set into the standard convolution layer, so that each of the standard convolution layers performs batch normalization on the binary edge image set; inputting the processed binary edge image set into the target convolution layer, so that the target convolution layer calculates the convolution kernel according to the linear rectification function according to the downsampling operator to obtain the feature convolution kernel operator, thereby reducing parameters while ensuring the accuracy of the feature convolution neural network and improving the effect of the feature convolution neural network.

[0017] According to some embodiments of the present invention, the step of inputting the training sample set into the super-resolution reconstruction module so that the super-resolution reconstruction module trains the high-resolution image set and the low-resolution image set according to the feature convolution kernel operator to generate a weight matrix comprises: inputting the training sample set into the super-resolution reconstruction module so that the super-resolution reconstruction module performs feature mapping processing on the high-resolution image set and the low-resolution image set based on the feature convolution kernel operator to generate the weight matrix, and improving the feature convolution neural network's ability to extract defect image contours through the weight matrix.

[0018] According to some embodiments of the present invention, generating a target defect image corresponding to the training sample set according to the weight matrix includes: inputting the high-resolution image set and the low-resolution image set into the super-resolution reconstruction module, so that the super-resolution reconstruction module generates a loss function based on a fuse function; outputting a pixel threshold between the high-resolution image set and the low-resolution image set according to the loss function; generating a target defect image corresponding to the training sample set according to the pixel threshold and the weight matrix, reducing the influence of bad pixel data through the effects of the fuse function and the loss function, avoiding the fusing of weight nodes, and improving the effect of sharpening the edges of the defect image.

[0019] In a second aspect, the present invention provides a terahertz image super-resolution reconstruction system based on a feature convolutional neural network, comprising:

[0020] A sample acquisition module, used to acquire a training sample set, wherein the training sample set includes a high-resolution image set, a low-resolution image set and a defect image set, and the defect image set is obtained by subjecting the raw material to terahertz imaging processing;

[0021] An edge detection module, used for generating a binary edge image set according to the defect image set;

[0022] A downsampling module, used for downsampling the binary edge image set to obtain a feature convolution kernel operator;

[0023] A super-resolution reconstruction module is used to train the high-resolution image set and the low-resolution image set according to the characteristic convolution kernel operator to generate a weight matrix;

[0024] A target image generation module is used to generate a target defect image corresponding to the training sample set according to the weight matrix.

[0025] In a third aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to enable a computer to execute the terahertz super-resolution reconstruction method based on a feature convolutional neural network as described in the first aspect.

[0026] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0028] Figure 1 It is a schematic diagram of the architecture of a terahertz image super-resolution reconstruction system based on a feature convolutional neural network provided by an embodiment of the present invention;

[0029] Figure 2 is a flow chart of a terahertz super-resolution reconstruction method based on a feature convolutional neural network provided by an embodiment of the present invention;

[0030] Figure 3 is a flow chart of a training sample set method provided by an embodiment of the present invention;

[0031] Figure 4 yes Figure 2 Specific method flow chart of step S200 in FIG.

[0032] Figure 5 yes Figure 4 Specific method flow chart of step S220 in FIG.

[0033] Figure 6 yes Figure 2 Specific method flow chart of step S300 in FIG.

[0034] Figure 7 yes Figure 2 Specific method flow chart of step S500 in . DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0037] The present invention provides a terahertz super-resolution reconstruction method, system and computer-readable storage medium based on a feature convolutional neural network. First, a training sample set including a high-resolution image set, a low-resolution image set and a defect image set is obtained, and the defect image set is input into an edge detection module to obtain a binary edge image set corresponding to the defect image set, thereby realizing the extraction of the contour edge of the defect image, which is convenient for the subsequent detection of the defect image. After that, the binary edge image set is input into a downsampling module for processing to obtain a feature convolution kernel operator, and the depth of the feature convolutional neural network is improved. The training sample set is input into a super-resolution reconstruction module for training, and a weight matrix between the high-resolution image set and the low-resolution image set is obtained through training. Finally, a target defect image corresponding to the training sample set is generated according to the weight matrix. The edge detection module, the downsampling module and the super-resolution reconstruction module are trained by the training sample set to obtain the contour resolution of the defect image in the training sample set, thereby achieving a stable super-resolution reconstruction effect.

[0038] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, Figure 1 It is a schematic diagram of the architecture of a terahertz image super-resolution reconstruction system based on a feature convolutional neural network provided by an embodiment of the present invention.

[0040] exist Figure 1In the example of , the terahertz image super-resolution reconstruction system based on the feature convolutional neural network includes but is not limited to: a sample acquisition module 100, an edge detection module 200, a downsampling module 300, a super-resolution reconstruction module 400 and a target image generation module 500; when performing terahertz image super-resolution reconstruction, first, the edge detection module 200 acquires the training data set in the sample acquisition module 100, and sends the defect image set in the training data set to the edge detection module 200, the edge detection module 200 extracts the binary edge image set of the defect image by binary image projection on the defect image set, and then The rear edge detection module 200 sends the binary edge image set to the downsampling module 300, the downsampling module 300 determines the size of the feature convolution kernel operator according to the binary edge image set, and finally the super-resolution reconstruction module 400 obtains the high-resolution image set and the low-resolution image set in the sample acquisition module 100, trains the high-resolution image set and the low-resolution image set, and sends the training result to the target image generation module 500, thereby completing the training of the edge detection module 200, the downsampling module 300 and the super-resolution reconstruction module 400, and the super-resolution reconstruction of the defect contour in the defect atlas.

[0041] In one embodiment, the sample acquisition module 100 is used to acquire a training sample set, wherein the training sample set includes a high-resolution image set, a low-resolution image set and a defect image set, and the defect image set is obtained by subjecting the raw material to terahertz imaging processing, so as to provide a training sample set for the edge detection module 200 and the super-resolution reconstruction module 400.

[0042] In one embodiment, the edge detection module 200 is used to generate a binary edge image set according to the defect image set, thereby improving the pertinence of defect detection and improving the efficiency of the terahertz image super-resolution reconstruction system.

[0043] In one embodiment, the downsampling module 300 is used to perform downsampling processing on the binary edge image set to obtain a feature convolution kernel operator, so as to extract defect edge features of the defect image.

[0044] In one embodiment, the super-resolution reconstruction module 400 trains the high-resolution image set and the low-resolution image set according to the feature convolution kernel operator to generate a weight matrix to achieve mapping from the high-resolution image set to the low-resolution image set.

[0045] In one embodiment, the target image generation module 500 is used to generate a target defect image corresponding to the training sample set according to the weight matrix, so as to achieve super-resolution reconstruction of the defect image and clearly display the outline of the defect image.

[0046] The terahertz image super-resolution reconstruction system based on the feature convolutional neural network and the application scenario described in the embodiment of the present invention are intended to more clearly illustrate the technical solution of the embodiment of the present invention, and do not constitute a limitation on the technical solution provided by the embodiment of the present invention. Those skilled in the art can know that with the evolution of memory management methods and the emergence of new application scenarios, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.

[0047] Based on the above-mentioned terahertz image super-resolution reconstruction system based on feature convolutional neural network, various embodiments of the terahertz super-resolution reconstruction method based on feature convolutional neural network of the present invention are proposed below.

[0048] like Figure 2 As shown, Figure 2 is a flow chart of a terahertz super-resolution reconstruction method based on a feature convolutional neural network provided by an embodiment of the present invention, wherein the feature convolutional neural network may be but is not limited to the following Figure 1 The terahertz image super-resolution reconstruction system and the terahertz super-resolution reconstruction method shown include but are not limited to steps S100 to S500.

[0049] Step S100: obtaining a training sample set;

[0050] It should be noted that the training sample set includes a high-resolution image set, a low-resolution image set and a defect image set, and the defect image set is obtained by performing terahertz imaging processing on the raw materials.

[0051] Step S200: inputting the defect image set in the training sample set into the edge detection module, so that the edge detection module generates a binary edge image set based on the edge detection algorithm;

[0052] It should be noted that the edge detection algorithm can be an algorithm that uses an edge operator to extract image edges, such as: Prewitt operator algorithm (also known as Prewitt operate), Sobel operator algorithm (also known as Sobel operator), or Roberts cross edge detection algorithm (also known as Roberts Cross operator), etc. This embodiment does not make specific restrictions.

[0053] Step S300: inputting the binary edge image set into a downsampling module, so that the downsampling module performs downsampling processing on the binary edge image set to obtain a feature convolution kernel operator;

[0054] In one embodiment, the binary edge image set is input into a downsampling module, so that the downsampling module performs downsampling processing on the binary edge image set so that the image size in the binary edge image set meets the needs of technical personnel in this field, thereby generating thumbnails of the images in the corresponding binary edge image set, thereby reducing the computational complexity of the feature convolutional neural network.

[0055] Step S400: inputting the training sample set into the super-resolution reconstruction module, so that the super-resolution reconstruction module trains the high-resolution image set and the low-resolution image set according to the characteristic convolution kernel operator to generate a weight matrix;

[0056] In one embodiment, the training sample set is input into the super-resolution reconstruction module for training, so as to obtain the mapping weights from the high-resolution image set to the low-resolution image set, so as to facilitate the extraction of the contour of the defect in the defect image.

[0057] Step S500: generating a target defect image corresponding to the training sample set according to the weight matrix.

[0058] In one embodiment, the mapping weights from the high-resolution image set to the low-resolution image set are obtained through step S400, so that the target defect image corresponding to other defect images in the training sample set can be obtained according to the weight matrix, thereby clearly displaying the outline of the target defect image.

[0059] like Figure 3 As shown, Figure 3 1 is a flowchart of a method for training sample sets provided by an embodiment of the present invention. The method for training sample sets includes but is not limited to steps S110 to S140.

[0060] Step S110: performing imaging processing on the raw material to obtain a high-resolution image set;

[0061] It should be noted that the high-resolution image set includes multiple original photos.

[0062] It is understandable that the raw material may be a steel plate, concrete or wood, etc. In this embodiment, the raw material is a pre-defect embedded quartz fiber composite material.

[0063] In one embodiment, the raw material is imaged by photographing the raw material using an electronic device such as a mobile phone or a camera, thereby obtaining a high-resolution image set.

[0064] Step S120: scaling the original photos in the high-resolution image set based on an image interpolation algorithm, and restoring the scaled original photos to obtain a low-resolution image set corresponding to the high-resolution image set;

[0065] It should be noted that the image interpolation algorithm includes but is not limited to the nearest neighbor interpolation method, the bicubic interpolation method, the edge information based interpolation method or the bilinear interpolation method, and this embodiment does not make any specific limitation.

[0066] In one embodiment, the original photos in the high-resolution image set are reduced by a cubic interpolation downsampling method, and the reduced sizes are respectively 2 times, 3 times and 4 times. Then, the reduced original photos are enlarged by a traditional cubic interpolation method and restored to the original size to obtain a low-resolution image set corresponding to the high-resolution image set, thereby enhancing the contrast effect between the high-resolution image set and the low-resolution image set.

[0067] Step S130: performing terahertz imaging processing on the raw material to obtain a defect image set;

[0068] It should be noted that the defect map obtained through terahertz imaging processing may be a steel plate defect map, a concrete defect map or a wood defect map, etc. In this embodiment, it is a defect map of a quartz fiber composite material with embedded defects.

[0069] In one embodiment, a defect image set is obtained by performing terahertz scanning imaging on a plurality of pre-defect embedded quartz fiber composite materials and performing grayscale imaging by taking time domain amplitudes at different times.

[0070] Step S140: Integrate the high-resolution image set, the low-resolution image set and the defect image set to obtain a training sample set.

[0071] It should be noted that the number of images in the high-resolution image set can be 100, 120 or 150, etc., the number of images in the low-resolution image set can be 110, 150 or 160, etc., and the number of defect images in the defect image set can be 110, 140 or 150, etc., and this embodiment does not make any specific restrictions.

[0072] like Figure 4 As shown, Figure 4 yes Figure 2 Specific method flow chart of step S200 in FIG. 1 , where step S200 includes but is not limited to steps S210 - S220.

[0073] In one embodiment, the edge detection module includes an edge operator, wherein the edge operator is an edge detection operator based on a first-order derivative, and the edge detection operator includes a horizontal detection matrix and a vertical detection matrix.

[0074] Step S210: inputting the defect image set into the edge detection module, so that the edge detection module performs a plane convolution on the horizontal detection matrix, the vertical detection matrix and the defect image set to obtain a brightness difference approximation;

[0075] It should be noted that the brightness difference approximation includes a horizontal brightness difference approximation and a vertical brightness difference approximation.

[0076] In one embodiment, the defect image set is input into the edge detection module, so that the defect image is planarly convolved with the horizontal detection matrix and the vertical detection matrix in the edge detection module, and the horizontal brightness difference approximation and the vertical brightness difference approximation can be obtained, thereby eliminating the influence of convolution noise.

[0077] Step S220: extracting defect contours in the defect image set based on the overall nested edge algorithm and the brightness difference approximation to obtain a binary edge image set.

[0078] In one embodiment, a Holistically-Nested Edge Detection (HED) algorithm is used to extract defect contours in a defect image, thereby obtaining a binary edge image set, thereby improving the accuracy of extracting defect image contours.

[0079] like Figure 5 As shown, Figure 5 yes Figure 4 Specific method flow chart of step S220 in the embodiment, step S220 includes but is not limited to steps S221-S222.

[0080] Step S221: calculating the defect images in the defect image set according to the branch function and the weighting function so that all defect images in the defect image set have the same size;

[0081] It should be noted that the branch function is used to characterize the output of each branch of the feature convolutional neural network, and the weighting function is used to characterize the weight of each branch of the feature convolutional neural network.

[0082] In one embodiment, the defect images in the defect image set are calculated according to the branch function and the weighting function to obtain the output results and weights of each branch, as well as the branch weights of each branch of the convolutional neural network, so that all defect images in the defect image set have the same size.

[0083] Step S222: performing binary image projection on the calculated defect image set to obtain a binary edge image set.

[0084] In one embodiment, the binary edge image set is input into the downsampling module, so that the convolution layer of the downsampling module performs batch normalization processing on the binary edge image set, and calculates the processed binary edge image set based on the linear rectification function to obtain a feature convolution kernel operator.

[0085] It can be understood that the binary edge map is input into the downsampling module for batch normalization processing to improve the generalization ability of the convolutional neural network, thereby optimizing the downsampling module, and the processed binary edge image set is calculated based on the linear rectification function to obtain the feature convolution kernel operator, thereby realizing the training of the feature convolutional neural network and accelerating the convergence of the feature convolutional neural network.

[0086] like Figure 6 As shown, Figure 6 yes Figure 2 Specific method flow chart of step S300 in the embodiment, step S300 includes but is not limited to steps S311-S312.

[0087] It should be noted that the convolution layer includes multiple layers of standard convolution layers and target convolution layers. The standard convolution layers are the first few layers of the downsampling module, which are the same as ordinary convolutional neural networks. The target convolution layer is the last layer of the downsampling module and determines the size of the output feature convolution kernel operator.

[0088] Step S311: inputting the binary edge image set into the standard convolution layer, so that each standard convolution layer performs batch standardization processing on the binary edge image set;

[0089] Step S312: input the processed binary edge image set into the target convolution layer, so that the target convolution layer calculates the convolution kernel according to the linear rectification function according to the downsampling operator to obtain the feature convolution kernel operator.

[0090] In one embodiment, a binary edge image set is input into the convolution layer of a downsampling module. First, the binary edge image set is batch normalized by a standard convolution layer, and then the output is calculated by a target convolution layer to finally obtain a feature convolution kernel operator.

[0091] In one embodiment, the training sample set is input into the super-resolution reconstruction module, so that the super-resolution reconstruction module performs feature mapping processing on the high-resolution image set and the low-resolution image set based on the feature convolution kernel operator to generate a weight matrix.

[0092] It can be understood that the training sample set is input into the super-resolution reconstruction module to train the mapping weights between the high-resolution image set and the low-resolution image set, thereby realizing the mapping from the high-resolution image to the low-resolution image.

[0093] like Figure 7 As shown, Figure 7 yes Figure 2 Specific method flow chart of step S500 in the embodiment, step S500 includes but is not limited to steps S510-S530.

[0094] Step S510: inputting the high-resolution image set and the low-resolution image set into a super-resolution reconstruction module, so that the super-resolution reconstruction module generates a loss function based on the fuse function;

[0095] In one embodiment, a high-resolution image set and a low-resolution image set are input into a super-resolution reconstruction module, so that the super-resolution reconstruction module generates a loss function based on a fuse function, thereby avoiding gradient explosion or gradient disappearance at a certain node during the gradient descent process.

[0096] It should be noted that in this embodiment, a certain threshold value can be set, which is a vector, to determine the modulus and unit direction of the node. If the threshold value is exceeded, the subsequent connection of the node is automatically disconnected to avoid gradient explosion of the node during the gradient descent process.

[0097] Step S520: outputting a pixel threshold between the high-resolution image set and the low-resolution image set according to the loss function;

[0098] Step S530: generating a target defect image corresponding to the training sample set according to the pixel threshold and the weight matrix.

[0099] In one embodiment, the pixel threshold between the high-resolution image set and the low-resolution image set is output according to the loss function to determine whether the mapping from the high-resolution image to the low-resolution image meets the requirements. The mapping result is compared with the target defect image to obtain the pixel threshold. Then, according to the pixel threshold and the weight matrix, the corresponding target defect image is generated to achieve the mapping from the high-resolution image to the low-resolution image. The influence of the bad pixel data is reduced by the action of the fuse function and the loss function to avoid the fuse of the weight node, thereby improving the effect of sharpening the edges of the defect image.

[0100] In order to more clearly illustrate the process of the terahertz super-resolution reconstruction method based on the feature convolutional neural network provided by the embodiment of the present invention, a specific example is given below for illustration.

[0101] Example 1:

[0102] This example is a specific example of an edge detection module generating a binary edge image set based on an edge detection algorithm.

[0103] The edge detection module mainly uses the edge operator, which is an edge detection operator based on the first-order derivative. It introduces an operation similar to the local average, which has a smoothing effect on noise and can effectively eliminate the influence of noise. It contains two groups of matrices, namely the horizontal detection matrix and the vertical detection matrix. By performing a plane convolution with the image, the horizontal and vertical brightness difference approximations can be obtained respectively. Among them, the horizontal detection matrix for detecting horizontal edges and the vertical detection matrix for detecting vertical edges are:

[0104]

[0105] The calculation formulas for the horizontal and vertical gradients in an image are shown in formula (1):

[0106]

[0107] The gradient direction calculation formula is shown in formula (2):

[0108]

[0109] In addition, considering that the data has multi-scale images, this paper adopts the overall nested edge algorithm, with VGG-16 as the backbone network. Before the maximum pooling operation of the four data blocks of VGG-16 is downsampled, four branches are generated through the branch function (also called side branch) and the weighting function (also called fuse branch), and then the activation function is used as the output of the network. The image size of each output is the same as the input image. The mathematical expressions of the side branch and fuse branch are shown in formula (3) and formula (4), respectively:

[0110]

[0111]

[0112] in, The output of each branch, Y is the convolutional neural network output, is the result of the side branch output after passing through the activation function, and σ is the weight of each branch. Suppose all the parameters of the network are W, and the side branch parameter value is w=(w (1) , ..., w (m) ), then the loss function of the side branch is shown in formula (5):

[0113]

[0114] Among them, α m is the weight of the loss function of each side branch, which can be adjusted according to the training log. is the loss function of each side branch. The loss function adopts the category-balanced cross entropy loss function, and its mathematical expression is shown in formula (6) and formula (7):

[0115]

[0116]

[0117] Among them, β, |Y + | represents the number of non-edge pixels, |Y - | represents the number of edge pixels. The objective function of this module training is and As shown in formula (8):

[0118]

[0119] Finally, the binary edge map of the defect image is extracted as the output of the edge detection module.

[0120] Example 2:

[0121] This example is a specific example of the process in which the downsampling module downsamples a binary edge image set to obtain a feature convolution kernel operator.

[0122] The downsampling module includes multiple layers of standard convolutional layers and target convolutional layers, using 32 convolutional kernels of size 5X5, followed by batch normalization and linear rectification function processing. It can be described by formula (9):

[0123]

[0124] Where k = 1, 2, ..., k-1 is the index of the convolutional layer in the downsampling module, and f DS_k represents the output of the first, and Y represents the initial input f of the downsampling module DS_0 =Y,W DS_k represents the weight matrix of the convolutional layer, B DS_k Indicates the offset.

[0125] The first k-1 layers of the standard convolutional layer in the downsampling module are the same as those of ordinary convolutional neural networks, but the last layer (target convolutional layer) is the key layer for performing the downsampling operation, which determines the size of the output feature convolution kernel operator, as shown in formula (10):

[0126]

[0127] in, The final output is It is the downsampling operator, which is also the feature convolution kernel operator in the downsampling module.

[0128] Example 3:

[0129] This example inputs a high-resolution image set and a low-resolution image set into a super-resolution reconstruction module, so that the super-resolution reconstruction module generates a specific example of a loss function based on a fuse function.

[0130] Therefore, the loss function of the super-resolution reconstruction module is shown in formula (11):

[0131]

[0132] Among them, Θ CB represents the trainable parameters in the super-resolution reconstruction module, F CBIt is a fuse function, and its main function is to suppress the problem of gradient explosion or gradient vanishing of a certain node during the gradient descent process. By setting a certain threshold, which is a vector, the modulus and unit direction of the node are judged. If the threshold is exceeded, the subsequent connection of the node is automatically disconnected.

[0133] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or a controller, for example, by a processor in the above system embodiment, so that the above processor can execute the terahertz super-resolution reconstruction method based on the feature convolutional neural network in the above embodiment, for example, execute the above described Figure 2 Steps S100 to S500 of the method, Figure 3 Steps S110 to S140 of the method, Figure 4 Steps S210 to S220 of the method, Figure 5 Method steps S221-S222, Figure 6 Steps S311 to S312 of the method, Figure 7 Method steps S510 to S530 in.

[0134] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0135] It can be understood by those skilled in the art that Figure 2-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0136] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0138] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0139] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0144] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A terahertz super-resolution reconstruction method based on a feature convolutional neural network, characterized in that: The feature convolutional neural network includes an edge detection module, a downsampling module and a super-resolution reconstruction module; the terahertz super-resolution reconstruction method includes: Acquire a training sample set, wherein the training sample set includes a high-resolution image set, a low-resolution image set, and a defect image set, and the defect image set is obtained by performing terahertz imaging processing on the raw material; Inputting the defect image set in the training sample set into the edge detection module, so that the edge detection module generates a binary edge image set based on an edge detection algorithm; Inputting the binary edge image set into the downsampling module, so that the downsampling module performs downsampling processing on the binary edge image set to obtain a feature convolution kernel operator; Inputting the training sample set into the super-resolution reconstruction module, so that the super-resolution reconstruction module trains the high-resolution image set and the low-resolution image set according to the characteristic convolution kernel operator to generate a weight matrix; generating a target defect image corresponding to the training sample set according to the weight matrix; The step of inputting the training sample set into the super-resolution reconstruction module so that the super-resolution reconstruction module trains the high-resolution image set and the low-resolution image set according to the feature convolution kernel operator to generate a weight matrix includes: Inputting the training sample set into the super-resolution reconstruction module, so that the super-resolution reconstruction module performs feature mapping processing on the high-resolution image set and the low-resolution image set based on the feature convolution kernel operator to generate the weight matrix; Wherein, generating a target defect image corresponding to the training sample set according to the weight matrix includes: Inputting the high-resolution image set and the low-resolution image set into the super-resolution reconstruction module, so that the super-resolution reconstruction module generates a loss function based on the fuse function; Outputting a pixel threshold between the high-resolution image set and the low-resolution image set according to the loss function; A target defect image corresponding to the training sample set is generated according to the pixel threshold and the weight matrix.

2. The terahertz super-resolution reconstruction method based on feature convolutional neural network according to claim 1, characterized in that: The training sample set is obtained by the following steps, including: Performing imaging processing on the raw material to obtain a high-resolution image set, wherein the high-resolution image set includes a plurality of original photos; Scaling the original photos in the high-resolution image set based on an image interpolation algorithm, and restoring the scaled original photos to obtain a low-resolution image set corresponding to the high-resolution image set; Performing terahertz imaging processing on the raw material to obtain the defect image set; The high-resolution image set, the low-resolution image set and the defect image set are integrated to obtain the training sample set.

3. The terahertz super-resolution reconstruction method based on feature convolutional neural network according to claim 1, characterized in that: The edge detection module includes a horizontal detection matrix and a vertical detection matrix; the step of inputting the defect image set into the edge detection module so that the edge detection module generates a binary edge image set based on an edge detection algorithm includes: Inputting the defect image set into the edge detection module, so that the edge detection module performs a plane convolution on the horizontal detection matrix, the vertical detection matrix and the defect image set to obtain a brightness difference approximation; The defect contours in the defect image set are extracted based on the overall nested edge algorithm and the brightness difference approximation to obtain the binary edge image set.

4. The terahertz super-resolution reconstruction method based on feature convolutional neural network according to claim 3 is characterized in that: The overall nested edge algorithm includes a branch function and a weighting function, wherein the branch function is used to characterize the output of each branch of the feature convolutional neural network, and the weighting function is used to characterize the weight of each branch of the feature convolutional neural network; the defect contours in the defect image set are extracted based on the overall nested edge algorithm and the brightness difference approximation to obtain the binary edge image set, including: Calculating the defect images in the defect image set according to the branch function and the weighting function so that all the defect images in the defect image set have the same size; The calculated defect image set is subjected to binary image projection to obtain the binary edge image set.

5. The terahertz super-resolution reconstruction method based on feature convolutional neural network according to claim 1, characterized in that: The step of inputting the binary edge image set into the downsampling module so that the downsampling module performs downsampling processing on the binary edge image set to obtain a feature convolution kernel operator comprises: The binary edge image set is input into the downsampling module, so that the convolution layer of the downsampling module performs batch normalization processing on the binary edge image set, and calculates the processed binary edge image set based on a linear rectification function to obtain the feature convolution kernel operator.

6. The terahertz super-resolution reconstruction method based on feature convolutional neural network according to claim 5, characterized in that: The convolution layer includes a plurality of standard convolution layers and a target convolution layer; the binary edge image set is input into the downsampling module, the convolution layer of the downsampling module performs batch normalization processing on the binary edge image set, and the processed binary edge image set is calculated based on a linear rectification function to obtain the feature convolution kernel operator, including: Inputting the binary edge image set into the standard convolution layer, so that each of the standard convolution layers performs batch standardization processing on the binary edge image set; The processed binary edge image set is input into the target convolution layer, so that the target convolution layer calculates the convolution kernel according to the linear rectification function and the downsampling operator to obtain the feature convolution kernel operator.

7. A terahertz image super-resolution reconstruction system based on feature convolutional neural network, characterized in that: include: A sample acquisition module, used to acquire a training sample set, wherein the training sample set includes a high-resolution image set, a low-resolution image set and a defect image set, and the defect image set is obtained by subjecting the raw material to terahertz imaging processing; An edge detection module, used for generating a binary edge image set according to the defect image set; A downsampling module, used for downsampling the binary edge image set to obtain a feature convolution kernel operator; A super-resolution reconstruction module is used to receive the training sample set, and train the high-resolution image set and the low-resolution image set according to the feature convolution kernel operator to generate a weight matrix, wherein the training of the high-resolution image set and the low-resolution image set according to the feature convolution kernel operator to generate a weight matrix includes: receiving the training sample set, performing feature mapping processing on the high-resolution image set and the low-resolution image set based on the feature convolution kernel operator, and generating the weight matrix; A target image generation module is used to generate a target defect image corresponding to the training sample set according to the weight matrix, wherein the generating the target defect image corresponding to the training sample set according to the weight matrix includes: inputting the high-resolution image set and the low-resolution image set into the super-resolution reconstruction module, so that the super-resolution reconstruction module generates a loss function based on a fuse function; outputting a pixel threshold between the high-resolution image set and the low-resolution image set according to the loss function; and generating the target defect image corresponding to the training sample set according to the pixel threshold and the weight matrix.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the terahertz super-resolution reconstruction method based on a feature convolutional neural network as described in any one of claims 1 to 6.