Training method and device of image denoising model, image denoising method and device

Abstract

Provided are a training method and device of an image denoising model, and an image denoising method and device. The training method comprises: obtaining a training sample set, comprising a first training sample and a second training sample, the first training sample comprising an original image data sample and a noise data sample, and the second training sample comprising a clean image data sample; inputting the original image data sample into a first convolutional neural network; the first convolutional neural network processes the original image data sample to obtain image feature data; inputting the noise data sample into a second convolutional neural network; the second convolutional neural network processes the noise data sample to obtain noise feature data; fusing the image feature data and the noise feature data to obtain fused feature data; performing a convolution operation on the fused feature data to obtain enhanced feature data for representing image features of an object to be identified; and adjusting parameters of the image denoising model according to a difference between the clean image data sample and the enhanced feature data.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to a training method and apparatus for an image denoising model, an image denoising method and apparatus, an electronic device, a computer-readable storage medium, and a program product. Background Technology

[0002] Terahertz passive imaging has been widely used in security inspections due to its advantages such as real-time imaging, no radiation, no need for human intervention, and strong penetration capabilities. The main principle of terahertz passive imaging is to use a terahertz sensor to receive radiation from the human body and the surrounding environment and output corresponding voltage values. Generally, the human body temperature is higher than the ambient temperature, and the side of a suspect carried on the body facing outwards is also close to the ambient temperature. The terahertz sensor senses the radiation brightness and temperature of surrounding objects, and after basic processing, highlights the temperature difference between the human body and the surrounding environment to form the final image. The main factor affecting the quality of a terahertz image is the signal-to-noise ratio (SNR), which is the ratio of the energy of the effective signal in the image to the power of the noise energy. Based on the above principle of terahertz passive imaging, the amount of effective information in a terahertz image depends on the temperature difference between the human body and the surrounding environment. Since the human body temperature can be considered constant, when the ambient temperature is low, the temperature difference between the human body and the environment is large, resulting in a large effective input signal for the terahertz sensor. The noise in terahertz passive imaging images mainly comes from the inherent background noise of the terahertz sensor itself. Therefore, in order to obtain terahertz images with high signal-to-noise ratio, the first approach is to increase the input of the effective signal. A common method is to reduce the radiation temperature of the surrounding environment, such as building a special building or passage to fix the security check area and using air conditioning equipment or low emissivity materials to reduce the radiation temperature of this area. The second approach is to directly reduce the noise in the image.

[0003] The first approach requires a dedicated area and maintaining a low temperature in the imaging region, which is detrimental to the deployment of terahertz passive imaging devices and increases the overall cost of the imaging system. The second approach suffers from limitations in existing image denoising methods for terahertz images.

[0004] The information disclosed in this section is only for understanding the background of the inventive concept of this disclosure, and therefore may include information that does not constitute prior art. Summary of the Invention

[0005] In view of at least one of the above-mentioned technical problems, a training method and apparatus for an image denoising model, an image denoising method and apparatus, an electronic device, a computer-readable storage medium, and a program product are proposed.

[0006] In one aspect, a method for training an image denoising model is provided, the image denoising model comprising a first convolutional neural network and a second convolutional neural network, the method comprising:

[0007] A training sample set is obtained, which includes a first training sample and a second training sample. The first training sample and the second training sample are used to represent the input and output of the image denoising model, respectively. The first training sample includes original image data samples and noisy data samples, and the second training sample includes clean image data samples. Both the original image data samples and the clean image data samples include image data of the object to be identified.

[0008] The original image data sample is input into the first convolutional neural network;

[0009] The first convolutional neural network processes the original image data samples to obtain image feature data;

[0010] The noise data sample is input into the second convolutional neural network;

[0011] The second convolutional neural network processes the noise data samples to obtain noise feature data;

[0012] The image feature data and the noise feature data are fused to obtain fused feature data;

[0013] The fused feature data is convolved to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified; and

[0014] Based on the difference between the clean image data sample and the enhanced feature data, the parameters of the image denoising model are adjusted.

[0015] According to some exemplary embodiments, fusing the image feature data and the noise feature data includes: removing the noise feature data from the image feature data.

[0016] According to some exemplary embodiments, the first convolutional neural network includes a first convolutional layer and a second convolutional layer connected in sequence, wherein the first convolutional layer includes a one-dimensional convolutional layer and the second convolutional layer includes a two-dimensional convolutional layer.

[0017] According to some exemplary embodiments, the second convolutional neural network includes a fifth convolutional layer and a sixth convolutional layer connected in sequence, wherein the fifth convolutional layer includes a one-dimensional convolutional layer and the sixth convolutional layer includes a two-dimensional convolutional layer.

[0018] According to some exemplary embodiments, the first convolutional neural network processes the original image data samples as follows:

[0019] The original image data sample is input into the first convolutional layer, and after the first convolution operation, the first image features are obtained; and

[0020] The first image features are input into the second convolutional layer, and after the second convolution operation, the second image features are obtained.

[0021] According to some exemplary embodiments, the second convolutional neural network processes the noisy data samples as follows:

[0022] The noise data sample is input into the fifth convolutional layer, and after the fifth convolution operation, the first noise feature is obtained; and

[0023] The first noise feature is input into the sixth convolutional layer, and after the sixth convolution operation, the second noise feature is obtained.

[0024] According to some exemplary embodiments, fusing the image feature data and the noise feature data includes:

[0025] The second image feature and the second noise feature are fused in a first manner to obtain the third image feature.

[0026] According to some exemplary embodiments, the first convolutional neural network further includes a third convolutional layer, which includes a two-dimensional convolutional layer.

[0027] The method further includes: inputting the third image feature into the third convolutional layer, and obtaining the fourth image feature through the third convolution operation.

[0028] According to some exemplary embodiments, the second convolutional neural network further includes a seventh convolutional layer, which comprises a two-dimensional convolutional layer.

[0029] The method further includes: inputting the second noise feature into the seventh convolutional layer, and obtaining the third noise feature after the seventh convolution operation.

[0030] According to some exemplary embodiments, fusing the image feature data and the noise feature data further includes:

[0031] The fourth image feature and the third noise feature are fused together to obtain the fifth image feature.

[0032] According to some exemplary embodiments, the first convolutional neural network further includes a fourth convolutional layer, which includes a two-dimensional convolutional layer.

[0033] The step of performing a convolution operation on the fused feature data to obtain enhanced feature data specifically includes: inputting the fifth image features into the fourth convolutional layer, and obtaining enhanced feature data through the fourth convolution operation.

[0034] According to some exemplary embodiments, obtaining the training sample set includes:

[0035] Acquire noisy data samples within a specified time period;

[0036] The noisy data samples within the specified time period are averaged to obtain the desired data.

[0037] Based on the noisy data sampling and the expected data, noise data within a specified time period is obtained;

[0038] A noise signal model is established based on the noise data within the specified time period;

[0039] Based on the noise signal model, noise data samples are obtained.

[0040] According to some exemplary embodiments, obtaining the training sample set further includes:

[0041] Under the first condition, acquire the first sample of raw image data;

[0042] A second image data sample is obtained based on the first original image data sample and the noise data sample, wherein the second image data sample includes clean image data after removing noise;

[0043] Based on the second image data sample, at least one third image data sample under the second condition is generated, wherein the third image data sample includes clean image data with noise removed;

[0044] The clean image data sample is obtained based on the second image data sampling and the third image data sampling.

[0045] Wherein, the signal-to-noise ratio of the original image data sampled under the first condition is higher than that of the original image data sampled under the second condition.

[0046] According to some exemplary embodiments, obtaining the training sample set further includes:

[0047] The original image data sample is obtained based on the obtained noise data sample and the clean image data sample.

[0048] According to some exemplary embodiments, the step of acquiring the first raw image data sample under the first condition specifically includes:

[0049] Under conditions where the ambient temperature is below a threshold temperature, the imaging area is scanned using a terahertz imaging device to obtain first raw image data samples.

[0050] The object to be identified is located in the imaging area.

[0051] According to some exemplary embodiments, acquiring noisy data samples within a specified time period includes:

[0052] Within a specified time period, the background area in the imaging region is scanned by a terahertz imaging device to obtain noise data samples.

[0053] According to some exemplary embodiments, generating at least one third image data sample under a second condition based on the second image data sample specifically includes:

[0054] Based on the temperature values ​​in the first condition and the second condition, determine the scaling factor; and

[0055] The second image data sample is scaled according to the scaling factor to generate at least one third image data sample under the second condition.

[0056] According to some exemplary embodiments, the first convolutional layer includes a first sub-convolutional layer and a second sub-convolutional layer connected in sequence, both of which are one-dimensional convolutional layers; and / or, the second convolutional layer includes a third sub-convolutional layer and a fourth sub-convolutional layer connected in sequence, both of which are two-dimensional convolutional layers.

[0057] According to some exemplary embodiments, the fifth convolutional layer includes a fifth sub-convolutional layer and a sixth sub-convolutional layer connected in sequence, both of which are one-dimensional convolutional layers; and / or, the sixth convolutional layer includes a seventh sub-convolutional layer and an eighth sub-convolutional layer connected in sequence, both of which are two-dimensional convolutional layers.

[0058] On the other hand, an image denoising method is provided, comprising:

[0059] Acquire raw image data, which includes image data of the object to be identified;

[0060] Acquire noise data, which is associated with the original image data;

[0061] The original image data is input into the first convolutional neural network;

[0062] The first convolutional neural network processes the original image data to obtain image feature data;

[0063] The noise data is input into the second convolutional neural network;

[0064] The second convolutional neural network processes the noise data to obtain noise feature data;

[0065] The image feature data and the noise feature data are fused to obtain fused feature data;

[0066] The fused feature data is convolved to obtain enhanced feature data, which is used to characterize the image features of the object to be identified.

[0067] According to some exemplary embodiments, fusing the image feature data and the noise feature data includes: removing the noise feature data from the image feature data.

[0068] According to some exemplary embodiments, the first convolutional neural network includes a first convolutional layer and a second convolutional layer connected in sequence, wherein the first convolutional layer includes a one-dimensional convolutional layer and the second convolutional layer includes a two-dimensional convolutional layer.

[0069] According to some exemplary embodiments, the second convolutional neural network includes a fifth convolutional layer and a sixth convolutional layer connected in sequence, wherein the fifth convolutional layer includes a one-dimensional convolutional layer and the sixth convolutional layer includes a two-dimensional convolutional layer.

[0070] According to some exemplary embodiments, the first convolutional neural network processes the raw image data as follows:

[0071] The original image data is input into the first convolutional layer, and after the first convolution operation, the first image features are obtained; and

[0072] The first image features are input into the second convolutional layer, and after the second convolution operation, the second image features are obtained.

[0073] According to some exemplary embodiments, the second convolutional neural network processes the noisy data by:

[0074] The noise data is input into the fifth convolutional layer, and after the fifth convolution operation, the first noise feature is obtained; and

[0075] The first noise feature is input into the sixth convolutional layer, and after the sixth convolution operation, the second noise feature is obtained.

[0076] According to some exemplary embodiments, fusing the image feature data and the noise feature data includes:

[0077] The second image feature and the second noise feature are fused in a first manner to obtain the third image feature.

[0078] According to some exemplary embodiments, the first convolutional neural network further includes a third convolutional layer, which includes a two-dimensional convolutional layer.

[0079] The method further includes: inputting the third image feature into the third convolutional layer, and obtaining the fourth image feature through the third convolution operation.

[0080] According to some exemplary embodiments, the second convolutional neural network further includes a seventh convolutional layer, which comprises a two-dimensional convolutional layer.

[0081] The method further includes: inputting the second noise feature into the seventh convolutional layer, and obtaining the third noise feature after the seventh convolution operation.

[0082] According to some exemplary embodiments, fusing the image feature data and the noise feature data further includes:

[0083] The fourth image feature and the third noise feature are fused together to obtain the fifth image feature.

[0084] According to some exemplary embodiments, the first convolutional neural network further includes a fourth convolutional layer, which includes a two-dimensional convolutional layer.

[0085] The step of performing a convolution operation on the fused feature data to obtain enhanced feature data specifically includes: inputting the fifth image features into the fourth convolutional layer, and obtaining enhanced feature data through the fourth convolution operation.

[0086] According to some exemplary embodiments, the method further includes: interpolating the enhanced feature data to obtain an interpolated image.

[0087] According to some exemplary embodiments, acquiring raw image data includes: scanning an imaging area using a terahertz imaging device to acquire raw image data, wherein the object to be identified is located in the imaging area.

[0088] According to some exemplary embodiments, acquiring noise data includes: scanning the background area in the imaging region using a terahertz imaging device to acquire noise data.

[0089] In another aspect, a training apparatus for an image denoising model is provided, comprising:

[0090] The training sample acquisition module is used to acquire a training sample set, which includes a first training sample and a second training sample. The first training sample and the second training sample are used to represent the input and output of the image denoising model, respectively. The first training sample includes original image data samples and noisy data samples, and the second training sample includes clean image data samples. Both the original image data samples and the clean image data samples include image data of the object to be identified.

[0091] The first convolutional neural network module is used to: receive the input of the original image data sample and process the original image data sample to obtain image feature data;

[0092] The second convolutional neural network module is used to: receive the input of the noise data sample and process the noise data sample to obtain noise feature data;

[0093] A fusion module is used to fuse the image feature data and the noise feature data to obtain fused feature data;

[0094] A convolution operation module is used to perform convolution operations on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified; and

[0095] The parameter adjustment module is used to adjust various parameters of the image denoising model based on the difference between the clean image data sample and the enhanced feature data.

[0096] In another aspect, an image denoising apparatus is provided, comprising:

[0097] The raw image data acquisition module is used to acquire raw image data, which includes image data of the object to be identified.

[0098] A noise data acquisition module is used to acquire noise data, which is associated with the original image data;

[0099] The first convolutional neural network module is used to: receive the input of the original image data and process the original image data to obtain image feature data;

[0100] The second convolutional neural network module is used to: receive the input of the noise data and process the noise data to obtain noise feature data;

[0101] A fusion module is used to fuse the image feature data and the noise feature data to obtain fused feature data;

[0102] The convolution operation module is used to perform convolution operations on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified.

[0103] In another aspect, an electronic device is provided, comprising:

[0104] One or more processors;

[0105] Storage device for storing one or more programs.

[0106] When the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0107] According to some exemplary embodiments, the electronic device is a passive terahertz imaging device.

[0108] In another aspect, a computer-readable storage medium is provided that stores executable instructions thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0109] In another aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0110] In the embodiments according to this disclosure, the method of using a dual-branch input neural network for denoising can recover the real signal under different signal-to-noise ratios and has a good and stable denoising effect. Attached Figure Description

[0111] To better understand the present invention, it will be described in detail with reference to the following drawings:

[0112] Figure 1 This is a schematic diagram of the structure of a passive terahertz imaging device according to an exemplary embodiment of the present disclosure.

[0113] Figure 2A and Figure 2B These are schematic diagrams of images with different signal-to-noise ratios acquired by a passive terahertz imaging apparatus according to exemplary embodiments of the present disclosure.

[0114] Figure 3 This is a schematic flowchart of a training method for an image denoising model according to an exemplary embodiment of the present disclosure.

[0115] Figure 4 This is a schematic diagram of a dual-branch neural network included in an image denoising model according to an exemplary embodiment of the present disclosure.

[0116] Figure 5 Schematic illustration based on Figure 4 The diagram shows a schematic flowchart of image denoising using a dual-branch neural network.

[0117] Figures 6 to 8 These are schematic flowcharts illustrating the process of obtaining a training sample set in a training method for an image denoising model according to exemplary embodiments of the present disclosure.

[0118] Figure 9 This is a schematic diagram of noise data according to an exemplary embodiment of the present disclosure.

[0119] Figure 10 yes Figure 9 The diagram shows a power spectrum of the noise data.

[0120] Figure 11 This is a schematic diagram of raw image data acquired by a passive terahertz imaging apparatus according to an exemplary embodiment of the present disclosure.

[0121] Figure 12 This is a schematic flowchart of an image denoising method according to an exemplary embodiment of the present disclosure.

[0122] Figures 13A to 13C These are schematic diagrams of the original images with different signal-to-noise ratios.

[0123] Figures 14A to 14C These are respectively using the traditional BM3D denoising algorithm to... Figures 13A to 13C A schematic diagram of the image obtained after denoising the image.

[0124] Figures 15A to 15C These are the image denoising methods according to embodiments of the present disclosure. Figures 13A to 13C A schematic diagram of the image obtained after denoising the image.

[0125] Figure 16 This is a structural block diagram of a training apparatus for an image denoising model according to an exemplary embodiment of the present disclosure.

[0126] Figure 17 This is a structural block diagram of an image denoising apparatus according to an exemplary embodiment of the present disclosure.

[0127] Figure 18 The diagram schematically illustrates a structural block diagram of an electronic device suitable for implementing a training method or image denoising method for an image denoising model, according to exemplary embodiments of the present disclosure. Detailed Implementation

[0128] Specific embodiments of the present invention will now be described in detail. It should be noted that the embodiments described herein are for illustrative purposes only and are not intended to limit the invention. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to practice the invention. In other instances, well-known structures, materials, or methods have not been specifically described to avoid obscuring the invention.

[0129] Throughout this specification, references to "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, the phrases "in an embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. Moreover, those skilled in the art will understand that the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0130] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0131] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0132] In this document, terms such as "raw image," "raw image data," and "raw data" refer to noisy images, image data, or data; that is, they are the original images, image data, or data acquired by a sensor or imaging device without noise removal. Terms such as "clean image," "clean image data," and "clean data" refer to denoised images, image data, or data; that is, they are images, image data, or data acquired by a sensor or imaging device and then denoised.

[0133] The inventors discovered that terahertz image denoising techniques can be divided into two main categories. In the first category, denoising is applied to the display image of the terahertz imaging device. At this stage, the image is already a standard image, with each pixel ranging from 0 to 255. For such images, denoising algorithms such as traditional bilateral filtering and BM3D can be used to improve image quality, or an end-to-end image denoising neural network can be trained to achieve the desired effect. Training an end-to-end denoising neural network requires a large number of noisy and noise-free image pairs. Obtaining noise-free images typically involves multiple exposures of the same scene, followed by superposition of these exposures. A challenge of standard image denoising neural networks is their high training cost, primarily due to the high cost of acquiring training samples. Furthermore, the diversity of real-world data leads to weak model stability. The traditional BM3D method performs poorly on images with low signal-to-noise ratios, and it cannot recover the image when the contrast between the target and background in the original image is low. In the second major category of denoising techniques, the raw data generated by the terahertz imaging device is directly denoised, and then the denoised data is used for image reconstruction. The advantage of this method is that it directly processes the raw data, avoiding the data precision loss that occurs when quantizing to the 0-255 range. Denoising the raw data and then reconstructing the image maximizes the contrast between the target and the background. The challenge in processing the raw data lies in the fact that the data range is not fixed, requiring a certain understanding of the characteristics of the raw data.

[0134] To this end, embodiments of this disclosure provide a training method for an image denoising model, the image denoising model including a first convolutional neural network and a second convolutional neural network, the method comprising: acquiring a training sample set, the training sample set including a first training sample and a second training sample, the first training sample and the second training sample being used to represent the input and output of the image denoising model respectively, the first training sample including original image data samples and noisy data samples, the second training sample including clean image data samples, both the original image data samples and the clean image data samples including image data of an object to be identified; inputting the original image data samples into the first convolutional neural network; the first convolutional neural network processing the original image data samples to obtain image feature data; inputting the noisy data samples into the second convolutional neural network; the second convolutional neural network processing the noisy data samples to obtain noise feature data; fusing the image feature data and the noise feature data to obtain fused feature data; performing a convolution operation on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to represent the image features of the object to be identified; and adjusting various parameters of the image denoising model according to the difference between the clean image data samples and the enhanced feature data.

[0135] Embodiments of this disclosure also provide an image denoising method, comprising: acquiring original image data, the original image data including image data of an object to be identified; acquiring noise data, the noise data being associated with the original image data; inputting the original image data into a first convolutional neural network; the first convolutional neural network processing the original image data to obtain image feature data; inputting the noise data into a second convolutional neural network; the second convolutional neural network processing the noise data to obtain noise feature data; fusing the image feature data and the noise feature data to obtain fused feature data; and performing a convolution operation on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified.

[0136] In the method according to embodiments of this disclosure, a dual-branch input neural network is used for denoising, where one branch receives the original terahertz image data to be denoised, and the other branch receives noise data. This method can recover the true signal under different signal-to-noise ratios and has a good and stable denoising effect.

[0137] Figure 1 This is a schematic diagram of the structure of a passive terahertz imaging device according to an exemplary embodiment of the present disclosure.

[0138] like Figure 1As shown, a passive terahertz imaging device according to an exemplary embodiment of this disclosure may include a reflector 2 and its servo system, a lens 3, a detector array 4, a data acquisition and processing device 6, a display device 7, and a power distribution box 5. Terahertz waves (or millimeter waves) spontaneously emitted by the object under inspection, as well as terahertz waves (or millimeter waves) reflecting the background environment, are incident on the reflector 2 through a window 1 on the housing and reflected by the reflector 2 onto the lens 3. After being converged by the lens 3, they are received by the detector array 4, which converts the received terahertz waves (or millimeter waves) into electrical signals. The data acquisition and processing device 6 is connected to the detector array 4 to receive the electrical signals from the detector array 4 and generate millimeter-wave / terahertz wave images. The display device 7 is connected to the data acquisition and processing device 6 for receiving and displaying the terahertz wave images (or millimeter-wave images) generated by the data acquisition and processing device 6. The power distribution box 5 is configured to provide power to the entire passive terahertz imaging device.

[0139] In actual operation, the servo system of reflector 2 controls reflector 2 to reciprocate, and the reciprocal of the period T of the motion is the imaging frame rate s. When reflector 2 swings from the maximum elevation angle to the minimum elevation angle, the swing angle is θ, which completes the scanning of the field of view angle with a height direction of 2θ within the depth of field. Reflector 2 forms an image from the maximum elevation angle to the minimum elevation angle. Data acquisition and processing device 6 continuously acquires data during this process. The control system of reflector 2 can be equipped with a position encoder to provide high-precision feedback on the scanning position of the reflector. When acquiring data, data acquisition and processing device 6 first marks the acquired data according to the information of the position encoder to distinguish the data for the next image. Then, data acquisition and processing device 6 processes the acquired data and performs image reconstruction to generate a terahertz / millimeter-wave image. Then, data acquisition and processing device 6 can transmit the image data to display device 7 for display, suspicious object marking, and automatic alarm.

[0140] Because passive terahertz sensors are large and expensive, passive terahertz imaging devices use a scanning mechanism to time-division multiplex the optical paths of linearly arranged sensors into a single area array, thereby creating a two-dimensional image of a specific region. Since the number of terahertz sensors is relatively small, related technologies typically obtain a usable terahertz image by performing two-dimensional interpolation on the data from these sensors. When the signal-to-noise ratio of the raw sensor data is low, the image obtained through two-dimensional interpolation will introduce many new interference features, and the detailed features of some real targets will be destroyed due to the significant noise during interpolation.

[0141] Furthermore, the inventors discovered that the data after two-dimensional interpolation needs to be normalized to 0-1 or 0-255 to become a truly displayable image. A major factor affecting the visual effect of terahertz images is the contrast between the human body parts and the suspected objects carried on the body, as well as the background. When the signal-to-noise ratio of the original data is low, the normalization process will result in a low contrast between the human body and the target object or background, making it difficult to achieve higher image contrast when denoising such an image.

[0142] Figure 2A and Figure 2B These are schematic diagrams of images with different signal-to-noise ratios acquired by a passive terahertz imaging apparatus according to exemplary embodiments of the present disclosure. When the signal-to-noise ratio of the original signal is different, the normalized data will have different contrasts. (Refer to...) Figure 2A When the signal-to-noise ratio of the original signal is low, the obtained region of interest (e.g.) Figure 2A The contrast between the area marked as ROI (shown by the dashed box) and the human body is low. (Refer to...) Figure 2B When the signal-to-noise ratio of the original signal is high, the obtained region of interest (e.g.) Figure 2B The contrast between the area marked as ROI (shown in the dashed box) and the human body is relatively high.

[0143] In the embodiments of this disclosure, an image denoising model is proposed, which includes a dual-branch input neural network that can achieve good denoising effect on image data with different signal-to-noise ratios.

[0144] Figure 3 This is a schematic flowchart of a training method for an image denoising model according to an exemplary embodiment of the present disclosure.

[0145] like Figure 3 As shown, the training method for an image denoising model according to an exemplary embodiment of this disclosure may include operations S310 to S380. This training method may be executed by a processor or by any electronic device including a processor. The image denoising model may include a first convolutional neural network and a second convolutional neural network.

[0146] In operation S310, a training sample set is obtained, which includes a first training sample and a second training sample. The first training sample and the second training sample are used to represent the input and output of the image denoising model, respectively. The first training sample includes original image data samples and noisy data samples, and the second training sample includes clean image data samples. Both the original image data samples and the clean image data samples include image data of the object to be identified.

[0147] For example, the actual sampling data of the terahertz imaging device can be directly used as the input signal of the input branch of the data to be denoised (i.e., as the original image data sample), and the input of the noise sampling branch is an approximately pure noise sequence.

[0148] In operation S320, the original image data sample is input into the first convolutional neural network.

[0149] In operation S330, the first convolutional neural network processes the original image data sample to obtain image feature data.

[0150] In operation S340, the noise data sample is input into the second convolutional neural network.

[0151] In operation S350, the second convolutional neural network processes the noise data sample to obtain noise feature data.

[0152] In operation S360, the image feature data and the noise feature data are fused to obtain fused feature data.

[0153] In embodiments of this disclosure, fusing the image feature data and the noise feature data includes: removing the noise feature data from the image feature data.

[0154] In operation S370, a convolution operation is performed on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified.

[0155] In operation S380, the parameters of the image denoising model are adjusted based on the difference between the clean image data sample and the enhanced feature data.

[0156] It should be understood that, in the embodiments of this disclosure, the neural network model may include pooling layers, loss functions, etc., and these pooling layers and loss functions can utilize various pooling layers and loss functions in known convolutional neural networks, which will not be elaborated here.

[0157] Figure 4 This is a schematic diagram of a dual-branch neural network included in an image denoising model according to an exemplary embodiment of the present disclosure. Figure 5 Schematic illustration based on Figure 4 The diagram shows a schematic flowchart of image denoising using a dual-branch neural network.

[0158] Combined with reference Figure 3 , Figure 4 and Figure 5The first convolutional neural network 10 may include a first convolutional layer 11 and a second convolutional layer 12 connected in sequence. In embodiments of this disclosure, the first convolutional layer 11 includes a one-dimensional convolutional layer, and the second convolutional layer 12 includes a two-dimensional convolutional layer. For example, the one-dimensional convolutional layer may be at least one of a 7*1 convolutional layer and a 5*1 convolutional layer, and the two-dimensional convolutional layer may be a 3*3 convolutional layer.

[0159] For example, the first convolutional layer 11 includes a first sub-convolutional layer 111 and a second sub-convolutional layer 112 connected in sequence, where both the first sub-convolutional layer 111 and the second sub-convolutional layer 112 are one-dimensional convolutional layers. For example, the first sub-convolutional layer 111 is a 7*1 convolutional layer, and the second sub-convolutional layer 112 is a 5*1 convolutional layer.

[0160] For example, the second convolutional layer 12 includes a third sub-convolutional layer 121 and a fourth sub-convolutional layer 122 connected in sequence, both of which are two-dimensional convolutional layers. For example, both the third sub-convolutional layer 121 and the fourth sub-convolutional layer 122 are 3*3 convolutional layers.

[0161] The second convolutional neural network 20 may include a fifth convolutional layer 21 and a sixth convolutional layer 22 connected in sequence. In embodiments of this disclosure, the fifth convolutional layer 21 includes a one-dimensional convolutional layer, and the sixth convolutional layer 22 includes a two-dimensional convolutional layer. For example, the one-dimensional convolutional layer may be at least one of a 7*1 convolutional layer and a 5*1 convolutional layer, and the two-dimensional convolutional layer may be a 3*3 convolutional layer.

[0162] For example, the fifth convolutional layer 21 includes a fifth sub-convolutional layer 211 and a sixth sub-convolutional layer 212 connected in sequence, both of which are one-dimensional convolutional layers. For example, the fifth sub-convolutional layer 211 is a 7*1 convolutional layer, and the sixth sub-convolutional layer 212 is a 5*1 convolutional layer.

[0163] For example, the sixth convolutional layer 22 includes a seventh sub-convolutional layer 221 and an eighth sub-convolutional layer 222 connected in sequence, both of which are two-dimensional convolutional layers. For example, both the seventh sub-convolutional layer 221 and the eighth sub-convolutional layer 222 are 3*3 convolutional layers.

[0164] In the embodiments of this disclosure, for the input branch of the signal to be denoised (i.e., the original image data branch) and the noise sampling branch, since each sensor is independent and its noise can be considered uncorrelated, one-dimensional convolutional kernels are used for feature extraction in the early stages of the network. For example, in the first convolutional neural network described above, the first layer uses a one-dimensional convolutional kernel of length 7, and the second layer uses a one-dimensional convolutional kernel of length 5. One-dimensional convolution operations extract signal features from a single sensor. Through this operation, the dimensionality of the data can be reduced quickly while preserving the characteristics of the signal, which is beneficial for reducing the computational load.

[0165] Furthermore, terahertz imaging devices employ multiple identical sensors, and the input signals between adjacent sensors exhibit a certain correlation during imaging. Therefore, in the aforementioned convolutional neural network, a two-dimensional convolutional kernel is used for feature extraction starting from the third layer. This approach leverages the correlation between channels to extract richer signal features.

[0166] Continue to refer to Figure 4 The first convolutional neural network 10 may further include a third convolutional layer 13 and a fourth convolutional layer 14. The third convolutional layer 13 includes a two-dimensional convolutional layer. For example, the third convolutional layer 13 includes a 3x3 convolutional layer. The fourth convolutional layer 14 includes a two-dimensional convolutional layer. For example, the fourth convolutional layer 14 includes a 3x3 convolutional layer.

[0167] The second convolutional neural network 20 may further include a seventh convolutional layer 23. The seventh convolutional layer 23 includes a two-dimensional convolutional layer. For example, the seventh convolutional layer 23 includes a 3*3 convolutional layer.

[0168] Below, we will refer to Figures 3 to 5 The training method of the image denoising model according to embodiments of the present disclosure will be further described.

[0169] In operation S330, the first convolutional neural network 10 processes the original image data sample, which may specifically include the following sub-operations.

[0170] In sub-operation S3301, the original image data sample is input into the first convolutional layer, and after the first convolution operation, the first image feature is obtained.

[0171] For example, the original image data sample can be input into the first sub-convolutional layer 111 for a 7*1 convolution operation; then, the output of the first sub-convolutional layer 111 can be input into the second sub-convolutional layer 112 for a 5*1 convolution operation, and the output of the second sub-convolutional layer 112 is the first image feature. That is, the first convolution operation includes a 7*1 convolution operation and a 5*1 convolution operation.

[0172] In sub-operation S3302, the first image features are input into the second convolutional layer, and after the second convolution operation, the second image features are obtained.

[0173] For example, the first image features can be input into the third sub-convolutional layer 121 and a 3*3 convolution operation can be performed; then, the output of the third sub-convolutional layer 121 can be input into the fourth sub-convolutional layer 122 and a 3*3 convolution operation can be performed, and the output of the fourth sub-convolutional layer 122 is the second image features.

[0174] In operation S350, the second convolutional neural network processes the noisy data samples, which may specifically include the following sub-operations.

[0175] In sub-operation S3501, the noise data sample is input into the fifth convolutional layer, and after the fifth convolution operation, the first noise feature is obtained.

[0176] For example, noisy data samples can be input into the fifth sub-convolutional layer 211 for a 7*1 convolution operation; then, the output of the fifth sub-convolutional layer 211 can be input into the sixth sub-convolutional layer 212 for a 5*1 convolution operation, and the output of the sixth sub-convolutional layer 212 is the first noise feature. That is, the fifth convolution operation includes both a 7*1 convolution operation and a 5*1 convolution operation.

[0177] In sub-operation S3502, the first noise feature is input into the sixth convolutional layer, and after the sixth convolution operation, the second noise feature is obtained.

[0178] For example, the first noise feature can be input into the seventh sub-convolutional layer 221 and a 3*3 convolution operation can be performed; then, the output of the seventh sub-convolutional layer 221 can be input into the eighth sub-convolutional layer 222 and a 3*3 convolution operation can be performed, and the output of the eighth sub-convolutional layer 222 is the second noise feature.

[0179] In operation S360, fusing the image feature data and the noise feature data may specifically include sub-operation S3601. In sub-operation S3601, a first fusion is performed on the second image feature and the second noise feature to obtain a third image feature. The second image feature is a noisy image feature, and the second noise feature is a feature sampled by noise. For example, the first fusion may include removing the noise sampled feature from the noisy image feature; that is, in the first fusion, a first denoising is performed.

[0180] In operation S330, the first convolutional neural network 10 processing the original image data sample may further include sub-operation S3303. In sub-operation S3303, the third image feature is input into the third convolutional layer 13, and after the third convolution operation, the fourth image feature is obtained. For example, the third convolution operation can be a 3*3 convolution operation.

[0181] In operation S350, the second convolutional neural network's processing of the noisy data samples may further include sub-operation S3503. In sub-operation S3503, the second noise feature is input into the seventh convolutional layer 23, and after the seventh convolution operation, the third noise feature is obtained. For example, the seventh convolution operation can be a 3*3 convolution operation.

[0182] In operation S360, fusing the image feature data and the noise feature data may specifically include sub-operation S3602. In sub-operation S3602, a second fusion is performed on the fourth image feature and the third noise feature to obtain a fifth image feature. The fourth image feature is a noisy image feature, and the fifth noise feature is a noise-sampled feature. For example, the second fusion may include further removing the noise-sampled feature from the noisy image feature; that is, a second denoising is performed in the second fusion.

[0183] Specifically, in operation S370, the fifth image features can be input into the fourth convolutional layer 14, and after the fourth convolution operation, enhanced feature data is obtained. For example, the fourth convolution operation can be a 3*3 convolution operation.

[0184] Optionally, in embodiments of this disclosure, multiple fourth convolutional layers 14 can be set. For example, two fourth convolutional layers 14 can be set, each of which is a 3*3 convolutional layer. That is, in operation S370, the fifth image features can be subjected to two 3*3 convolution operations to obtain the enhanced feature data.

[0185] In the embodiments of this disclosure, at a deeper level of the neural network model, features at various levels of abstraction of the two input branches are fused by stacking feature layers, which is beneficial for better noise removal.

[0186] Furthermore, in the embodiments of this disclosure, a method for denoising using a dual-branch input neural network is proposed. One branch takes the original terahertz image data to be denoised as input, and the other branch takes noise sampling data as input. Since each frame of data lasts for a short period of time, and the noise in the terahertz data mainly originates from the thermal noise of the sensor itself, the noise characteristics can be considered relatively stable within such a short time. Therefore, if the noise branch can acquire a portion of the noise sampling from the same frame of data, then this portion can be used to extract noise features, and the extracted noise features can then be integrated into the denoising network. Such a neural network model will have strong robustness in practical applications.

[0187] It should be understood that training a denoising neural network requires a large number of noisy and noise-free sample pairs. Obtaining noise-free data (with an extremely high signal-to-noise ratio) by superimposing data from multiple exposures of the same scene is costly. Therefore, embodiments of this disclosure propose a method for obtaining a training sample set.

[0188] Figures 6 to 8 These are schematic flowcharts illustrating the process of obtaining a training sample set in a training method for an image denoising model according to exemplary embodiments of the present disclosure.

[0189] Reference Figure 6 In the training method of the image denoising model according to an exemplary embodiment of the present disclosure, the following operations may be performed to obtain a training sample set, for example, obtaining noise data samples in the training sample set.

[0190] In operation S610, noisy data samples are acquired within a specified time period.

[0191] For example, the terahertz imaging device can be placed in a stable background environment, i.e., the observation scene is static and there are no changing targets. An absorbing material can be placed in the observation scene to purify the background. In this case, the terahertz imaging device can acquire raw image data from the sensor within a specified time period, i.e., obtain noisy data samples within the specified time period. For example, it can acquire data for 5 minutes every hour.

[0192] In operation S620, the noisy data samples within the specified time period are averaged to obtain the desired data.

[0193] Since a passive terahertz imaging device is similar to a camera, a clean background region exists within its imaging range, such as the area above the imaging region. This area contains only the background and no other constantly changing targets. Because terahertz imaging measures radiation brightness temperature, and since the emissivity of an object remains constant, its radiation brightness temperature also remains constant when the object's temperature is constant, tracking this simple background data can yield its expected value. That is, after the imaging device has been running for a certain period, for example, collecting data for 5 minutes every hour, the average of the noisy data collected during this period is calculated. Due to the random nature of noise, the averaged data sample should be the noise-free desired signal, i.e., the aforementioned expected value.

[0194] In operation S630, noise data within a specified time period is obtained based on the noise-bearing data sampling and the desired data.

[0195] For example, by subtracting the expected value from each frame of noisy data samples, the resulting data is the noisy data. Figure 9As shown, noise data is schematically illustrated according to some exemplary embodiments of the present disclosure. For example, in Figure 9 In the diagram, the horizontal axis can represent the count of sampling points, and the vertical axis can represent the signal strength, such as the voltage value.

[0196] Since the noise of terahertz passive imaging equipment mainly comes from the thermal noise of the sensor, and the thermal noise of the sensor is relatively stable at room temperature, a second-order stationary signal can be used for modeling.

[0197] In operation S640, a noise signal model is established based on the noise data within the specified time period.

[0198] Because a small portion of the imaging region of a terahertz imaging device is pure background, each frame can capture a small sample of the noise signal. Moreover, the noise can be considered stationary over a short period of time, so statistically speaking, the characteristics of this small portion of the noise signal are the same as those of the remaining noise.

[0199] For example, for noise data within a specified time period, a noise signal model M can be obtained using the power spectrum estimation method and the autoregressive modeling method. i，k (n), where the subscript i represents the noise signal model obtained using the sampling data of the i-th round, k represents the k-th sensor, and n is the index value of the discrete sequence. Experimental observations show that the noise signal models acquired by the sensors after running for a period of time are basically stable, with only minor differences. Therefore, selecting at least one model from the generated noise signal models can represent most of the power characteristics of the noise. Figure 10 The illustration schematically shows a noise power model estimated using collected noise data. For example, in Figure 10 In the diagram, the horizontal axis represents the normalized angular frequency, and the vertical axis represents the power spectral density of the signal.

[0200] In operation S650, noise data samples are obtained according to the noise signal model.

[0201] For example, according to the noise signal model M i，k (n) can generate noisy data samples.

[0202] Reference Figure 7 In the training method of the image denoising model according to an exemplary embodiment of the present disclosure, the following operations may be performed to obtain a training sample set, for example, obtaining clean image data samples in the training sample set.

[0203] In operation S710, under the first condition, the first raw image data sample is acquired.

[0204] Specifically, under conditions where the ambient temperature is below a threshold temperature, the imaging area can be scanned using a terahertz imaging device to obtain first raw image data samples. Figure 11 The illustration schematically shows a diagram of raw image data acquired under a first condition by a passive terahertz imaging apparatus according to an exemplary embodiment of the present disclosure. For example, in Figure 11 In the diagram, the horizontal axis can represent the count of sampling points, and the vertical axis can represent the signal strength, such as the voltage value.

[0205] For example, the object to be identified is located in the imaging area. For example, the threshold temperature can be 20°C.

[0206] For example, to obtain a clean signal, the imaging device can be placed in a scene with a low ambient temperature, such as 15°C. Different people carrying different targets can then freely move within the imaging range of the device, and raw image data can be collected. In this embodiment, the raw image can be 220 pixels wide and 440 pixels high. The imaging device can include 40 sensors, each collecting 512 rows of data, meaning the raw image data comprises a 40*512 matrix.

[0207] In embodiments of this disclosure, the first condition is that the sensor or imaging device can acquire image data with a high signal-to-noise ratio.

[0208] In operation S720, a second image data sample is obtained based on the first original image data sample and the noise data sample, wherein the second image data sample includes clean image data with noise removed.

[0209] For example, the noise data sample can be subtracted from the first original image data sample to obtain the second image data sample. Since the first original image data sample obtained under the first condition has a high signal-to-noise ratio, subtracting the noise data from this high signal-to-noise ratio first original image data sample can achieve a better denoising effect, resulting in a cleaner image data. That is, the second image data sample can better represent the clean image that can be obtained under the first condition.

[0210] In operation S730, at least one third image data sample under a second condition is generated based on the second image data sample, wherein the third image data sample includes clean image data with noise removed.

[0211] Specifically, in operation S730, a scaling factor can be determined based on the temperature value in the first condition and the temperature value in the second condition; then, the second image data sample is scaled according to the scaling factor to generate at least one third image data sample under the second condition.

[0212] For example, refer to Figure 11 At 15°C, the temperature difference between the human body and the environment is significant, resulting in a voltage value of approximately 6000. To obtain clean image data at other temperatures, it is not necessary to perform actual sampling again. Instead, the second image data sample needs to be scaled by a certain factor based on the ratio between other temperatures and the temperature under the first condition (e.g., 15°C) to simulate at least one third image data sample under the second condition.

[0213] In the embodiments of this disclosure, when the original image has a high signal-to-noise ratio, denoising algorithms such as BM3D can be used to improve the signal-to-noise ratio of the image. The denoised image is sampled and scaled, and clean image data under different ambient temperatures can be simulated by changing the scaling factor.

[0214] In operation S740, the clean image data sample is obtained based on the second image data sampling and the third image data sampling.

[0215] For example, the clean image data sample can be obtained by combining the second image data sample and the third image data sample.

[0216] In embodiments of this disclosure, the signal-to-noise ratio of the original image data sampled under the first condition is higher than that of the original image data sampled under the second condition.

[0217] Since the signal of a terahertz imaging device is mainly formed by the temperature difference between the human body and the environment, in the embodiments of this disclosure, the imaging device is placed in a scene with a low ambient temperature, so that the acquired image itself has a high signal-to-noise ratio.

[0218] Reference Figure 8 In the training method of the image denoising model according to an exemplary embodiment of the present disclosure, the following operations may be performed to obtain a training sample set, for example, obtaining original image data samples in the training sample set.

[0219] In operation S810, the original image data sample is obtained based on the obtained noise data sample and the clean image data sample.

[0220] Based on the above-mentioned acquisition of noise data samples and clean image data samples, for example, the above-mentioned acquisition of noise data samples and clean image data samples can be added together to synthesize a noisy original image data sample.

[0221] As described above, in the embodiments of this disclosure, clean image data samples under different conditions can be obtained. Furthermore, based on the clean image data samples under different conditions, original image data under different conditions can be synthesized. In this way, noisy original image data samples can be obtained efficiently and at low cost.

[0222] In embodiments of this disclosure, different people carrying different targets can be arranged to move freely within the imaging scene to enhance the diversity of the training signals. This diversity can be reflected in the diversity of human postures and the diversity of the targets carried. In this way, each acquired terahertz image can be used to generate raw image data (i.e., noisy image data) and clean image data (i.e., noise-removed image data) at different ambient temperatures. In this manner, a large number of training sample pairs can be generated with low cost and high efficiency.

[0223] In comparison, related technologies require generating training data pairs by superimposing multiple exposures to synthesize clean images. For example, if each scene is superimposed 50 times, the time required to collect this data would be 50 times longer than that of the embodiments of this disclosure. Moreover, traditional acquisition methods cannot easily simulate different ambient temperatures. Therefore, the method for obtaining the training sample set proposed in the embodiments of this disclosure has significant advantages.

[0224] The embodiments of this disclosure also propose an image denoising method. This image denoising method can use the image denoising model obtained by the training method described above to denoise the image. It should be noted that the image denoising method described below corresponds to the training method of the image denoising model described above. To make this document more concise, some exemplary descriptions will be omitted in the following description of the image denoising method. For these omitted parts, unless there is a conflict, the corresponding parts of the training method of the image denoising model described above can be referred to.

[0225] Figure 12 This is a schematic flowchart of an image denoising method according to an exemplary embodiment of the present disclosure.

[0226] like Figure 12 As shown, the image denoising method according to an exemplary embodiment of the present disclosure may include operations S1210 to S1280. This image denoising method may be executed by a processor or by any electronic device including a processor. For example, it may be executed by the imaging apparatus described above.

[0227] In operation S1210, raw image data is acquired, which includes image data of the object to be identified.

[0228] In operation S1220, noise data is acquired, which is associated with the original image data.

[0229] In operation S1230, the original image data is input into the first convolutional neural network.

[0230] In operation S1240, the first convolutional neural network processes the original image data to obtain image feature data.

[0231] In operation S1250, the noise data is input into the second convolutional neural network.

[0232] In operation S1260, the second convolutional neural network processes the noise data to obtain noise feature data.

[0233] In operation S1270, the image feature data and the noise feature data are fused to obtain fused feature data.

[0234] In operation S1280, a convolution operation is performed on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified.

[0235] In embodiments of this disclosure, fusing the image feature data and the noise feature data includes: removing the noise feature data from the image feature data.

[0236] In embodiments of this disclosure, the method may further include: interpolating the enhanced feature data to obtain an interpolated image. For example, the interpolation may be two-dimensional interpolation.

[0237] In the embodiments of this disclosure, noise is denoised before interpolating the original image data. This effectively avoids the destruction of real features or the introduction of new interfering features due to large noise. When normalizing the interpolated data, a higher signal-to-noise ratio can be obtained.

[0238] For example, acquiring raw image data includes: scanning the imaging area using a terahertz imaging device to acquire raw image data, wherein the object to be identified is located in the imaging area.

[0239] For example, acquiring noise data includes scanning the background area in the imaging region using a terahertz imaging device to acquire noise data.

[0240] Figures 13A to 13C These are schematic diagrams of the original images with different signal-to-noise ratios. Figures 14A to 14C These are respectively using the traditional BM3D denoising algorithm to... Figures 13A to 13C A schematic diagram of the image obtained after denoising the image. Figures 15A to 15C These are the image denoising methods according to embodiments of the present disclosure. Figures 13A to 13CA schematic diagram of the image obtained after denoising the image. Wherein, for Figures 13A to 13C The signal-to-noise ratio of the original image gradually increases.

[0241] By comparison Figure 13A , Figure 14A and Figure 15A It can be seen that for original images with low signal-to-noise ratio, the denoised images obtained by the traditional BM3D denoising algorithm have a low signal-to-noise ratio and cannot distinguish between suspects and human bodies well; the denoised images obtained by the image denoising method according to the embodiments of this disclosure have a high signal-to-noise ratio and can better recover the real signal.

[0242] By comparison Figures 13A to 15C It can be seen that for original images with low, medium, and high signal-to-noise ratios, the denoised images obtained by the image denoising method according to the embodiments of this disclosure all have high signal-to-noise ratios. That is, the image denoising method can recover the real signal to the greatest extent under different signal-to-noise ratios and has high stability.

[0243] Based on the above-described image denoising model training method, this disclosure also provides an image denoising model training device. The following will combine... Figure 16 The device is described in detail.

[0244] Figure 16 This is a structural block diagram of a training apparatus for an image denoising model according to an exemplary embodiment of the present disclosure.

[0245] like Figure 16 As shown, the training device 800 for the image denoising model includes a training sample acquisition module 810, a first convolutional neural network module 820, a second convolutional neural network module 830, a fusion module 840, a convolution operation module 850, and a parameter adjustment module 860.

[0246] The training sample acquisition module 810 is used to acquire a training sample set, which includes a first training sample and a second training sample. The first training sample and the second training sample are used to represent the input and output of the image denoising model, respectively. The first training sample includes original image data samples and noisy data samples, and the second training sample includes clean image data samples. Both the original image data samples and the clean image data samples include image data of the object to be identified. In some exemplary embodiments, the training sample acquisition module 810 can be used to perform the operation S310 and its sub-operations described above, which will not be repeated here.

[0247] The first convolutional neural network module 820 is configured to: receive the input of the original image data sample and process the original image data sample to obtain image feature data. In some exemplary embodiments, the first convolutional neural network module 820 may be used to perform the operations S320, S330 and their sub-operations described above, which will not be repeated here.

[0248] The second convolutional neural network module 830 is configured to: receive the input of the noise data sample and process the noise data sample to obtain noise feature data. In some exemplary embodiments, the second convolutional neural network module 830 may be used to perform the operations S340, S350 and their sub-operations described above, which will not be repeated here.

[0249] The fusion module 840 is used to fuse the image feature data and the noise feature data to obtain fused feature data. In some exemplary embodiments, the fusion module 840 can be used to perform the operation S360 and its sub-operations described above, which will not be repeated here.

[0250] The convolution operation module 850 is used to perform a convolution operation on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified. In some exemplary embodiments, the convolution operation module 850 can be used to perform the operation S370 and its sub-operations described above, which will not be repeated here.

[0251] The parameter adjustment module 860 is used to adjust various parameters of the image denoising model based on the difference between the clean image data sample and the enhanced feature data. In some exemplary embodiments, the parameter adjustment module 860 can be used to perform the operation S380 and its sub-operations described above, which will not be repeated here.

[0252] Figure 17 This is a structural block diagram of an image denoising apparatus according to an exemplary embodiment of the present disclosure.

[0253] like Figure 17 As shown, the image denoising device 900 includes an original image data acquisition module 910, a noise data acquisition module 920, a first convolutional neural network module 930, a second convolutional neural network module 940, a fusion module 950, and a convolution operation module 960.

[0254] The raw image data acquisition module 910 is used to acquire raw image data, which includes image data of the object to be identified. In some exemplary embodiments, the raw image data acquisition module 910 can be used to perform the operation S1210 and its sub-operations described above, which will not be repeated here.

[0255] The noise data acquisition module 920 is used to acquire noise data, which is associated with the original image data. In some exemplary embodiments, the noise data acquisition module 920 can be used to perform the operation S1220 and its sub-operations described above, which will not be repeated here.

[0256] The first convolutional neural network module 930 is configured to: receive the input of the original image data and process the original image data to obtain image feature data. In some exemplary embodiments, the first convolutional neural network module 930 may be used to perform the operations S1230, S1240 and their sub-operations described above, which will not be repeated here.

[0257] The second convolutional neural network module 940 is configured to: receive the noise data input and process the noise data to obtain noise feature data. In some exemplary embodiments, the second convolutional neural network module 940 may be used to perform the operations S1250, S1260 and their sub-operations described above, which will not be repeated here.

[0258] The fusion module 950 is used to fuse the image feature data and the noise feature data to obtain fused feature data. In some exemplary embodiments, the fusion module 950 can be used to perform the operation S1270 and its sub-operations described above, which will not be repeated here.

[0259] The convolution operation module 960 is used to perform a convolution operation on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified. In some exemplary embodiments, the convolution operation module 960 can be used to execute the operation S1280 and its sub-operations described above, which will not be repeated here.

[0260] According to embodiments of this disclosure, the image denoising model training device 800 includes a training sample acquisition module 810, a first convolutional neural network module 820, a second convolutional neural network module 830, a fusion module 840, a convolution operation module 850, and a parameter adjustment module 860. The image denoising device 900 includes an original image data acquisition module 910, a noise data acquisition module 920, a first convolutional neural network module 930, a second convolutional neural network module 940, a fusion module 950, and a convolution operation module 960. Any multiple modules can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some functions of one or more of these modules can be combined with at least some functions of other modules and implemented in one module. According to embodiments of this disclosure, the image denoising model training device 800 includes a training sample acquisition module 810, a first convolutional neural network module 820, a second convolutional neural network module 830, a fusion module 840, a convolution operation module 850, and a parameter adjustment module 860. The image denoising device 900 includes an original image data acquisition module 910, a noise data acquisition module 920, a first convolutional neural network module 930, a second convolutional neural network module 940, a fusion module 950, and a convolution operation module 960. At least one of these components can be implemented, at least partially, as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or as hardware or firmware. Alternatively, it can be implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the following components—training sample acquisition module 810, first convolutional neural network module 820, second convolutional neural network module 830, fusion module 840, convolution operation module 850, and parameter adjustment module 860—in the image denoising model training device 800, and the image denoising device 900—original image data acquisition module 910, noise data acquisition module 920, first convolutional neural network module 930, second convolutional neural network module 940, fusion module 950, and convolution operation module 960—can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0261] Figure 18 The diagram schematically illustrates a structural block diagram of an electronic device suitable for implementing a training method or image denoising method for an image denoising model, according to exemplary embodiments of the present disclosure.

[0262] like Figure 18As shown, an electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0263] For example, the electronic device may be a passive terahertz imaging device.

[0264] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0265] According to embodiments of this disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0266] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0267] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.

[0268] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the item recommendation method provided in the embodiments of this disclosure.

[0269] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0270] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0271] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0272] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0273] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0274] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0275] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A training method for an image denoising model, wherein the image denoising model comprises a first convolutional neural network and a second convolutional neural network, characterized in that, The method includes: A training sample set is obtained, which includes a first training sample and a second training sample. The first training sample and the second training sample are used to represent the input and output of the image denoising model, respectively. The first training sample includes original image data samples and noisy data samples, and the second training sample includes clean image data samples. Both the original image data samples and the clean image data samples include image data of the object to be identified. The original image data sample is input into the first convolutional neural network; The first convolutional neural network processes the original image data samples to obtain image feature data; The noise data sample is input into the second convolutional neural network; The second convolutional neural network processes the noise data samples to obtain noise feature data; The image feature data and the noise feature data are fused to obtain fused feature data; The fused feature data is convolved to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified; and Based on the difference between the clean image data sample and the enhanced feature data, adjust the parameters of the image denoising model; The acquisition of the training sample set includes: Under the first condition, first raw image data samples are acquired; A second image data sample is obtained based on the first original image data sample and the noise data sample, wherein the second image data sample includes clean image data after removing noise; Determine the scaling factor based on the temperature value in the first condition and the temperature value in the second condition; The second image data sample is scaled according to the scaling factor to generate at least one third image data sample under a second condition, wherein the third image data sample includes clean image data with noise removed; The clean image data sample is obtained based on the second image data sampling and the third image data sampling. Wherein, the signal-to-noise ratio of the original image data sampled under the first condition is higher than that of the original image data sampled under the second condition.

2. The method according to claim 1, wherein, The fusion of the image feature data and the noise feature data includes: removing the noise feature data from the image feature data.

3. The method according to claim 1 or 2, wherein, The first convolutional neural network includes a first convolutional layer and a second convolutional layer connected in sequence. The first convolutional layer includes a one-dimensional convolutional layer, and the second convolutional layer includes a two-dimensional convolutional layer.

4. The method according to claim 3, wherein, The second convolutional neural network includes a fifth convolutional layer and a sixth convolutional layer connected in sequence. The fifth convolutional layer includes a one-dimensional convolutional layer, and the sixth convolutional layer includes a two-dimensional convolutional layer.

5. The method according to claim 4, wherein, The first convolutional neural network processes the original image data samples including: The original image data sample is input into the first convolutional layer, and after the first convolution operation, the first image features are obtained; and The first image features are input into the second convolutional layer, and after the second convolution operation, the second image features are obtained.

6. The method according to claim 5, wherein, The second convolutional neural network processes the noisy data samples by: The noise data sample is input into the fifth convolutional layer, and after the fifth convolution operation, the first noise feature is obtained; and The first noise feature is input into the sixth convolutional layer, and after the sixth convolution operation, the second noise feature is obtained.

7. The method according to claim 6, wherein, The fusion of the image feature data and the noise feature data includes: The second image feature and the second noise feature are fused in a first manner to obtain the third image feature.

8. The method according to claim 7, wherein, The first convolutional neural network further includes a third convolutional layer, which is a two-dimensional convolutional layer. The method further includes: inputting the third image feature into the third convolutional layer, and obtaining the fourth image feature through the third convolution operation.

9. The method according to claim 8, wherein, The second convolutional neural network further includes a seventh convolutional layer, which is a two-dimensional convolutional layer. The method further includes: inputting the second noise feature into the seventh convolutional layer, and obtaining the third noise feature after the seventh convolution operation.

10. The method according to claim 9, wherein, The fusion of the image feature data and the noise feature data further includes: The fourth image feature and the third noise feature are fused together to obtain the fifth image feature.

11. The method according to claim 10, wherein, The first convolutional neural network further includes a fourth convolutional layer, which is a two-dimensional convolutional layer. The step of performing a convolution operation on the fused feature data to obtain enhanced feature data specifically includes: inputting the fifth image features into the fourth convolutional layer, and obtaining enhanced feature data through the fourth convolution operation.

12. The method according to claim 1 or 2, wherein, The acquisition of the training sample set includes: Acquire noisy data samples within a specified time period; The noisy data samples within the specified time period are averaged to obtain the desired data. Based on the noisy data sampling and the expected data, noise data within a specified time period is obtained; A noise signal model is established based on the noise data within the specified time period; Based on the noise signal model, noise data samples are obtained.

13. The method according to claim 1, wherein, The acquisition of the training sample set also includes: The original image data sample is obtained based on the obtained noise data sample and the clean image data sample.

14. The method according to claim 1, wherein, Under the first condition, acquiring the first original image data sample specifically includes: Under conditions where the ambient temperature is below a threshold temperature, the imaging area is scanned using a terahertz imaging device to obtain first raw image data samples. The object to be identified is located in the imaging area.

15. The method according to claim 12, wherein, The acquisition of noisy data sampling within a specified time period includes: Within a specified time period, the background area in the imaging region is scanned by a terahertz imaging device to obtain noise data samples.

16. The method according to claim 3, wherein, The first convolutional layer includes a first sub-convolutional layer and a second sub-convolutional layer connected in sequence, both of which are one-dimensional convolutional layers; and / or, the second convolutional layer includes a third sub-convolutional layer and a fourth sub-convolutional layer connected in sequence, both of which are two-dimensional convolutional layers.

17. The method according to claim 4, wherein, The fifth convolutional layer includes a fifth sub-convolutional layer and a sixth sub-convolutional layer connected in sequence, both of which are one-dimensional convolutional layers; and / or, the sixth convolutional layer includes a seventh sub-convolutional layer and an eighth sub-convolutional layer connected in sequence, both of which are two-dimensional convolutional layers.

18. An image denoising method, characterized in that, include: Acquire raw image data, which includes image data of the object to be identified; Acquire noise data, which is associated with the original image data; The original image data is input into the first convolutional neural network; The first convolutional neural network processes the original image data to obtain image feature data; The noise data is input into the second convolutional neural network; The second convolutional neural network processes the noise data to obtain noise feature data; The image feature data and the noise feature data are fused to obtain fused feature data; The fused feature data is convolved to obtain enhanced feature data, which is used to characterize the image features of the object to be identified. The first convolutional neural network and the second convolutional neural network constitute an image denoising model, which is trained using the training method described in claim 1.

19. The method according to claim 18, wherein, The fusion of the image feature data and the noise feature data includes: removing the noise feature data from the image feature data.

20. The method according to claim 18 or 19, wherein, The first convolutional neural network includes a first convolutional layer and a second convolutional layer connected in sequence. The first convolutional layer includes a one-dimensional convolutional layer, and the second convolutional layer includes a two-dimensional convolutional layer.

21. The method according to claim 20, wherein, The second convolutional neural network includes a fifth convolutional layer and a sixth convolutional layer connected in sequence. The fifth convolutional layer includes a one-dimensional convolutional layer, and the sixth convolutional layer includes a two-dimensional convolutional layer.

22. The method according to claim 21, wherein, The first convolutional neural network processes the original image data including: The original image data is input into the first convolutional layer, and after the first convolution operation, the first image features are obtained; and The first image features are input into the second convolutional layer, and after the second convolution operation, the second image features are obtained.

23. The method according to claim 22, wherein, The second convolutional neural network processes the noisy data by: The noise data is input into the fifth convolutional layer, and after the fifth convolution operation, the first noise feature is obtained; and The first noise feature is input into the sixth convolutional layer, and after the sixth convolution operation, the second noise feature is obtained.

24. The method according to claim 23, wherein, The fusion of the image feature data and the noise feature data includes: The second image feature and the second noise feature are fused in a first manner to obtain the third image feature.

25. The method according to claim 24, wherein, The first convolutional neural network further includes a third convolutional layer, which is a two-dimensional convolutional layer. The method further includes: inputting the third image feature into the third convolutional layer, and obtaining the fourth image feature through the third convolution operation.

26. The method of claim 25, wherein, The second convolutional neural network further includes a seventh convolutional layer, which is a two-dimensional convolutional layer. The method further includes: inputting the second noise feature into the seventh convolutional layer, and obtaining the third noise feature after the seventh convolution operation.

27. The method according to claim 26, wherein, The fusion of the image feature data and the noise feature data further includes: The fourth image feature and the third noise feature are fused together to obtain the fifth image feature.

28. The method according to claim 27, wherein, The first convolutional neural network further includes a fourth convolutional layer, which is a two-dimensional convolutional layer. The step of performing a convolution operation on the fused feature data to obtain enhanced feature data specifically includes: inputting the fifth image features into the fourth convolutional layer, and obtaining enhanced feature data through the fourth convolution operation.

29. The method according to claim 28, wherein, The method further includes: interpolating the enhanced feature data to obtain an interpolated image.

30. The method according to claim 18 or 19, wherein, The acquisition of raw image data includes: scanning the imaging area using a terahertz imaging device to acquire raw image data, wherein the object to be identified is located in the imaging area.

31. The method according to claim 30, wherein, The acquisition of noise data includes: scanning the background area in the imaging region using a terahertz imaging device to acquire noise data.

32. A training device for an image denoising model, characterized in that, include: The training sample acquisition module is used to acquire a training sample set, which includes a first training sample and a second training sample. The first training sample and the second training sample are used to represent the input and output of the image denoising model, respectively. The first training sample includes original image data samples and noisy data samples, and the second training sample includes clean image data samples. Both the original image data samples and the clean image data samples include image data of the object to be identified. The first convolutional neural network module is used to: receive the input of the original image data sample and process the original image data sample to obtain image feature data; The second convolutional neural network module is used to: receive the input of the noise data sample and process the noise data sample to obtain noise feature data; A fusion module is used to fuse the image feature data and the noise feature data to obtain fused feature data; A convolution operation module is used to perform convolution operations on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified; and The parameter adjustment module is used to adjust various parameters of the image denoising model based on the difference between the clean image data sample and the enhanced feature data. The training sample acquisition module is further used for: Under the first condition, first raw image data samples are acquired; A second image data sample is obtained based on the first original image data sample and the noise data sample, wherein the second image data sample includes clean image data after removing noise; Determine the scaling factor based on the temperature value in the first condition and the temperature value in the second condition; The second image data sample is scaled according to the scaling factor to generate at least one third image data sample under a second condition, wherein the third image data sample includes clean image data with noise removed; The clean image data sample is obtained based on the second image data sampling and the third image data sampling. Wherein, the signal-to-noise ratio of the original image data sampled under the first condition is higher than that of the original image data sampled under the second condition.

33. An image denoising device, characterized in that, include: The raw image data acquisition module is used to acquire raw image data, which includes image data of the object to be identified. A noise data acquisition module is used to acquire noise data, which is associated with the original image data; The first convolutional neural network module is used to: receive the input of the original image data and process the original image data to obtain image feature data; The second convolutional neural network module is used to: receive the input of the noise data and process the noise data to obtain noise feature data; A fusion module is used to fuse the image feature data and the noise feature data to obtain fused feature data; The convolution operation module is used to perform convolution operations on the fused feature data to obtain enhanced feature data, wherein the enhanced feature data is used to characterize the image features of the object to be identified. The first convolutional neural network module and the second convolutional neural network module constitute an image denoising model, which is trained using the training method described in claim 1.

34. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 31.

35. The electronic device according to claim 34, wherein, The electronic device is a passive terahertz imaging device.

36. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 31.

37. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 31.

Images