A terahertz image data expansion method based on a space-frequency domain degradation model

Abstract

The present application belongs to the field of terahertz imaging and image processing, and particularly relates to a terahertz image data expansion method based on a space-frequency degradation model. The method mainly comprises the following steps: pre-screening and pre-processing a high-resolution image database to establish a high-resolution image library; simulating a terahertz Gaussian beam to perform fuzzy degradation processing on the high-resolution image using a Gaussian fuzzy kernel; simulating terahertz wave source fluctuation and sensor noise to add Gaussian noise; simulating low resolution of terahertz imaging, and reducing the image resolution using a bicubic downsampling method; simulating mutual interference of multiple terahertz waves during imaging, and using a scheme of adding a frequency domain mask to change the frequency domain features; simulating the case that low image contrast is caused by low terahertz wave source power, and using a gray scale compression method to compress the gray scale histogram of the image to obtain a final low-resolution image data set. The present application can effectively expand the super-resolution training set of terahertz images, thereby improving the training effect of the terahertz image super-resolution reconstruction task based on the deep learning method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of terahertz imaging and image processing, in particular to a terahertz image data expansion method based on a space-frequency domain degradation model. BACKGROUND

[0002] Terahertz (THz) refers to electromagnetic waves in the frequency range from 100 GHz to 10,000 GHz (10 THz), corresponding to a wavelength range from 3 mm to 3 μm, which is between millimeter waves and infrared light. The photon energy is relatively low, but it has high penetration power. Because the electromagnetic waves in this frequency band are in the transition region from macroscopic electronics to microscopic photonics, they have some unique characteristics. Therefore, terahertz imaging technology has great prospects and is a research hotspot. In non-destructive testing[1], biomedical[2], security inspection[3], etc., terahertz imaging has irreplaceable advantages.

[0003] However, there are many problems with terahertz wave images, resulting in unsatisfactory imaging results. In practical applications, there are many reasons why terahertz image results are not ideal: first, due to wave source fluctuations and sensor noise, the image noise is very complex; second, multiple electromagnetic waves interfere with each other in the detector, causing image strip interference; and third, the low power of the wave source results in low image contrast, and the image color change is not obvious, and the terahertz camera resolution is low, resulting in low terahertz image resolution.

[0004] Single Image Super Resolution (SISR) algorithm refers to the process of recovering a high-resolution (High Resolution, HR) image from a low-resolution (Low Resolution, LR) image, while refining image details and improving image quality. Specific methods include srcnn, edsr, srgan, etc.

[0005] In recent years, this method has also been gradually applied to the field of terahertz image super-resolution. These methods have the same problem of insufficient terahertz image data, and usually use degradation methods to expand the data. However, when the pre-assumed degradation model of image super-resolution (Single Image Super Resolution, SISR) algorithm does not match the degradation mode of the real image, the performance of the model will decrease, and even negative effects may occur. Therefore, the single image super-resolution (Single Image Super Resolution, SISR) algorithm trained on the simulated data set often does not work well in practical application scenarios. SUMMARY

[0006] The present application is directed to the above-mentioned current terahertz imaging deep learning super-resolution field, the insufficient data of the image and the incomplete image degradation problem.The present application provides a high-efficiency, fitting all terahertz image degradation condition based on a spatial-frequency domain degradation model terahertz image data expansion method.The core is that sufficient high-resolution image data set related to the image super-resolution field is used as input, the image is blurred through the blur kernel, the terahertz frequency domain features are added, the image is down-sampled, and the noise is added, and finally the required simulated terahertz image, the terahertz image database is obtained.The method can be used for deep learning terahertz super-resolution problems and other electromagnetic wave interference, noise serious imaging problems.

[0007] The object of the present application is realized by the following technical solutions:

[0008] S1. Pre-screening and preprocessing of high-resolution image database.

[0009] S2. The method of using the time domain degradation model is used to blur the high-resolution image and simulate the low-resolution imaging caused by wave source fluctuation interference and sensor measurement noise in the noise simulation terahertz imaging process.

[0010] S3. The image obtained by the above process is used to simulate the mutual interference of multiple electromagnetic waves in the detector by using the frequency domain method.

[0011] S4. The obtained image is subjected to contrast degradation processing, i.e., gray scale fitting, by using the image gray scale processing method, to simulate the problem of low terahertz wave source power.

[0012] S5. The low-resolution image obtained by the above steps is one-to-one corresponding to the high-resolution image after pre-screening and preprocessing, and a text file recording the positions of all corresponding image files is obtained.

[0013] S6. Image data expansion script is used to expand the database image.

[0014] The preprocessing in step S1 screens out the images with clear edges as original images, divides the original images into 400x400 small images, redefines the file name and performs numbering sorting, and establishes a high-resolution image library.

[0015] The time domain degradation model in step S2 selects different size Gaussian blur kernels according to the frequency of electromagnetic waves, the condition of wave source equipment, and the condition of imaging equipment.The expression of the time domain degradation model is:

[0016] o1(x,y)=[i(x,y)*PSF(x,y)]↓s+n

[0017] Wherein o1(x, y) is the output image with blur and noise degradation; i(x, y) is the input high-resolution image; PSF(x, y) is a Gaussian blur kernel, simulating the Gaussian beam of the terahertz source to the object imaging; ↓s is a bicubic down-sampling, simulating the low resolution problem in the terahertz imaging process; n is an additive Gaussian noise, simulating the complex noise of the terahertz device in the imaging process.

[0018] The frequency domain method in the step S3 adopts two-dimensional Fourier transform, adds a frequency domain mask equivalent image and a time domain stripe artifact. First, the image is subjected to FFT algorithm, two-dimensional discrete Fourier frequency domain decomposition is carried out, the space domain image is converted into a two-dimensional frequency domain image. And in the high frequency part of the two-dimensional frequency domain image, an additive mask or a multiplicative mask is added. The additive mask means that a certain amplitude is added at the same time in the image feature frequency domain, and the multiplicative mask means that a certain amplitude is multiplied at the same time in the image feature frequency domain. The above method changes the image feature frequency coefficient to make it larger, and then the simulated terahertz imaging image containing the stripe artifact is obtained, so that the image degradation in the frequency domain is ensured. Different frequency domain masks are selected for processing according to different sizes of the stripe.

[0019] The image gray processing method in the step S4 is to compress and degrade the gray histogram of the image. The contrast degradation processing method usually has a logarithmic method, an exponential or power function compression method, which compresses the gray change range and then reduces the gray contrast.

[0020] The image data expansion script in the step S6 uses the methods of random cropping, random horizontal flipping and random rotation angle. The script acts on the images in the low-resolution image lr folder and the images in the high-resolution image hr folder at the same time, and then the database after the expansion of the images is obtained. The expansion expression is:

[0021]

[0022] Y is the image after data expansion, v is the data expansion transformation, and X is the image before data expansion. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 It is a terahertz image data expansion method flow chart based on a space domain-frequency domain degradation model.

[0024] Figure 2 It is a performance diagram of the multiplicative mask and the additive frequency domain mask.

[0025] Figure 3 It is a picture before applying the degradation model to the image and a picture after applying the degradation model to the image. DETAILED DESCRIPTION

[0026] The embodiment is a possible implementation based on MATLAB and Python. The technical solutions in the embodiments of the application will be clearly and completely described in combination with the drawings in the embodiments of the application.

[0027] Figure 1 The flowchart of the terahertz image data expansion method based on the spatial-frequency domain degradation method is shown in the figure. The steps include:

[0028] S1. Pre-screening and preprocessing of the high-resolution image database. The preprocessing is to divide the original image into 400x400 small images, redefine the file name and number, establish the high-resolution image library, and label the categories. Here, we perform the above steps on the open-source high-resolution image training database, BSD200, DIV2K, and the test database Set5, Set14.

[0029] S2-S6 is the main implementation process of the spatial-frequency domain degradation model. The expression is as follows, where i(x, y) is the original image.

[0030] o1(x,y)=deg{IFFT{FFT{[i(x,y)*PSF(x,y)]↓s+n}*Y}}

[0031] S2. Using the spatial domain Gaussian blur method to blur the high-resolution image, which is represented as PSF(x, y) in the above formula. x and y are the center of the Gaussian blur kernel, i.e. the blur center. The symbol σ represents the Gaussian radius and the blur radius, which is determined by the working frequency of the wave source, the structure of the antenna and the lens / reflector. The calculation is relatively complex, and here we only use the blur result of σ=5.0. At the same time, a two-dimensional numpy matrix with the same size as the original image is established according to the image size, and the values are all set to 0 for later frequency domain mask processing.

[0032] S3. Down-sampling processing of the image, which is represented as ↓s in the above formula. Bicubic down-sampling is used to simulate the low resolution problem in the terahertz imaging process. Bicubic down-sampling is a commonly used down-sampling method. Here we use the resize function of the opencv open source library, and set the parameter to INTER_CUBIC. To simulate different degrees of image blur, the parameters are usually set to 1 / 2, 1 / 4, 1 / 8, and the libraries are established respectively. Here we only use 1 / 2 down-sampling.

[0033] S4. Adding additive noise to the image, which is represented as n in the above formula. Here we set it as Gaussian white noise, where the standard deviation is σ=1.0, simulating the complex noise of the terahertz device in the imaging process.

[0034] S5. The image is processed in the frequency domain, and an additive mask or a multiplicative mask is constructed and applied to the original image to simulate the mutual interference of multiple electromagnetic waves in the detector. First, the image is two-dimensionally discrete Fourier frequency domain decomposition to obtain the two-dimensional frequency spectrum image of the original image, which is FFT. Then, according to the mask template obtained in S2, the size of which is rows matrix row number and cols matrix column number. Combined with the frequency spectrum image of the real terahertz fringe image, two kinds of mask methods are adopted, one is additive mask, and the other is multiplicative mask, the structure of which is shown in Figure 2 The left graph is a multiplicative mask, and the right graph is an additive mask. The multiplicative mask method is adopted in this degradation algorithm, all values of the mask template are assigned to 1, and the values of the surrounding 20*20 size at the top (rows / 2, cols / 4) and bottom (rows / 2, 3*cols / 4) of the center of the template are modified to 5. The obtained frequency domain template is multiplied by the frequency spectrum image of the image after space-time domain degradation to obtain the degradation image with added fringe artifacts. The frequency domain degradation image is obtained by two-dimensional inverse Fourier transform, which is IFFT.

[0035] S6. The obtained image is subjected to resolution degradation processing and gray scale fitting using image gray scale processing method to simulate the low power problem of terahertz wave source, which is deg. The gray scale image processing method includes power function, gamma function, and logarithmic function method for gray scale compression. Here, the gamma method is adopted for gray scale image compression, and the formula is:

[0036] s=c*k r

[0037] Where c and r are constants, c is the gamma transformation hyperparameter, and r is the gamma value. Both of them determine the gray scale degradation result, here only c=1 and r=0.95 are taken.

[0038] The degradation image obtained by the above degradation algorithm is shown in Figure 3 .

[0039] Finally, the image data meeting the requirements is selected and the data set is expanded by image expansion method, and the image data is finally saved. For the supervised problem of image super-resolution, create a lr folder and put the degraded images into it; create an hr folder and put the pre-degradation files into it. The pictures under the lr and hr folders must correspond one by one, and the file names must be consistent. Write a python script to expand the database image, and the methods used include random cropping, random horizontal flipping and random rotation angle. This script acts on the images under the lr folder and the images under the hr folder at the same time to obtain the expanded database after the expansion images. The expansion expression is:

[0040]

[0041] Y is the image after data augmentation, v is the data augmentation transformation, and X is the image before data augmentation.

[0042] It should be noted that although the present application is only directed to the degradation of terahertz images, the present application is not limited to terahertz images, but can also include the degradation of microwave and millimeter wave images. At the same time, the application range is not limited, and can be used in the fields of nondestructive testing and security inspection.

Claims

1. A terahertz image data augmentation method based on a spatial-frequency degradation model, characterized in that... By combining a Gaussian downsampling model and the frequency domain characteristics of terahertz images, a simulated terahertz image database is obtained on existing open-source image databases. This database is suitable for terahertz image super-resolution reconstruction tasks using deep learning methods and includes the following steps: S1. Pre-screen and preprocess images from open-source image databases, label their categories, and establish a high-resolution image library; S2. A method based on a spatial degradation model is used to perform blur and noise degradation on high-resolution images to simulate low-resolution imaging caused by wave source fluctuation interference and sensor measurement noise during terahertz imaging. S3. Using a frequency domain mask model-based method, change the high-frequency characteristics of the image obtained in step S2, add stripe artifacts, and then convert to the time domain to simulate the mutual interference of multiple terahertz electromagnetic waves in the detector during terahertz imaging. S4. Using an image grayscale processing-based method, the image obtained in step S3 is subjected to resolution degradation processing and grayscale fitting to simulate the effect of low power of terahertz electromagnetic wave source during terahertz imaging and to establish a low-resolution image library. S5. Match the low-resolution images obtained from the degradation in step S4 with the high-resolution images obtained from the pre-screening and preprocessing in step S1, and create a tag file containing all the high-resolution and low-resolution images. S6. Image augmentation is performed on the high-resolution image database and the low-resolution image database using data augmentation methods.

2. For example Claim 1 The terahertz image data augmentation method based on the spatial-frequency degradation model is characterized in that... Step S1 includes: S11. Select images with high contrast and prominent foreground and background from open-source image databases; S12. Using image preprocessing methods, the color format of the high-resolution image is changed and the RGB image is transformed into a grayscale image, thereby obtaining a high-resolution grayscale image with more distinct foreground and background, which matches the degraded low-resolution image. S13. Divide the high-resolution image database into a training set and a test set according to a certain ratio.

3. For example Claim 1 The terahertz image data augmentation method based on the spatial-frequency degradation model is characterized in that... The joint expression for the spatial degradation model in step S2 and the frequency mask model in step S3 is: o1(x,y)=deg{IFFT{FFT{[i(x,y)*PSF(x,y)]↓s+n}*Y}} where o1(x,y) is the output image with added blur and noise degradation, i(x,y) is the input high-resolution image, deg is the image grayscale processing method; PSF(x,y) is the Gaussian blur kernel, used to simulate the Gaussian beam imaging of objects from a terahertz source; ↓s is bicubic downsampling, used to simulate the low resolution problem in the terahertz imaging process; n is additive Gaussian noise, used... The simulation addresses the complex noise of terahertz equipment during imaging. FFT is a frequency domain transformation method used to perform a two-dimensional fast Fourier transform on the image, generating a two-dimensional frequency domain image. Y is a frequency domain mask used to add additive or multiplicative masks to the characteristic frequency part of the two-dimensional frequency domain image, changing the high-frequency coefficients of the frequency domain image to increase them. IFFT refers to performing an inverse Fourier transform on the image to obtain a simulated terahertz imaging image containing stripe artifacts. The above frequency domain transformations mainly simulate the complex changes in the image frequency domain during terahertz imaging.

4. For example Claim 1 The terahertz image data augmentation method based on the spatial-frequency degradation model is characterized in that... The image grayscale processing method in step S4 compresses and degrades the grayscale histogram of the image by using logarithmic, exponential, or power functions to compress the grayscale variation range, thereby reducing the grayscale contrast. This step simulates the problem of low power of terahertz wave sources.

5. For example Claim 1 The terahertz image data augmentation method based on the spatial-frequency degradation model is characterized in that... The data augmentation methods in step S6 include random cropping, random horizontal flipping, and random rotation angle. The data augmentation expression is as follows: In the formula, Y is the image after data augmentation, v is the data augmentation transformation, X is the image before data augmentation, lr is the image after degradation, and hr is the image before degradation.

6. For example Claim 1 The terahertz image data augmentation method based on the spatial-frequency degradation model is characterized in that... The obtained high-resolution and low-resolution image libraries are used for training and testing of supervised algorithm models in the process of deep learning terahertz image super-resolution reconstruction.

Images