Photo-thermal microscopic imaging system and method based on cyclic generative adversarial neural network

Abstract

The invention relates to a photo-thermal microscopic imaging system and method based on a cyclic generative adversarial neural network, and belongs to the technical field of photo-thermal science and detection and signal processing. Comprising a reflector, a sleeve lens, a first slide, a scanning lens, a platform, a galvanometer controller, a dichroscope, a polarization cube, a reflector group, a second slide, a third slide, a spatial light filter, an acousto-optic modulator, a first optical filter, a polarization cube, a fourth slide, an exciting light source, a polarization cube, a spatial light filter and a second optical filter. A probe light source; and a photo-thermal microscopic imaging system. According to the method, the three-dimensional diffusion of the heat flow is fully utilized, and the defect is accurately reconstructed from a two-dimensional angle by changing different spatial fluctuation states of the heat flow, so that the detection depth and the detection efficiency are improved, and meanwhile, the quantitative characterization of the material and the three-dimensional chromatography reconstruction of the geometric structure can be realized; according to the method, efficient and accurate detection and three-dimensional feature reconstruction of superficial layer defects of composite materials, metal materials and high-molecular polymers are achieved.

Description

Technical Field

[0001] The invention relates to a photothermal microscopic imaging system and method, and belongs to the technical field of photothermal science, detection and signal processing. Background Art

[0002] Optical materials and optical components have a huge demand for high spatial resolution test and characterization methods, especially the detection and analysis of surface contamination residues after optical material processing, the characterization of optical film preparation quality, and the measurement of micro-nano structures such as gratings. At present, high spatial resolution measurement technologies mainly include high-power optical microscopes, confocal microscopes, near-field optical microscopes, and fluorescence microscopes. The first three technologies can only observe the surface contours, geometric morphology, and visually visible defects of components, and cannot reflect the absorption distribution differences and invisible defects of processed materials, which limits the comprehensive judgment of the preparation quality. Fluorescence microscopy requires that the material has a strong fluorescence emission effect, while the fluorescence effect of most optical materials is very weak or even non-existent. Therefore, the application of fluorescence microscopy in this field is limited. Photothermal absorption measurement technology can obtain the absorption characteristics of materials and components through the physical effect of photothermal conversion, and can measure the absorption distribution and invisible defects of materials. Based on photothermal absorption measurement, researchers use high-power microscope objective lenses to focus the pump light source to reduce the size of the measurement spot, thereby providing measurement spatial resolution. However, due to the diffraction limit, the measurement resolution can only reach the submicron level.

[0003] In existing research, for example: Publication No. CN107084917B, the invention is named a photothermal differential microscopy imaging device and a single particle imaging method, the device described in the technical scheme uses an optical imaging illumination system to find the location of the sample marker, uses a detection beam to extract the weak thermal radiation signal around the sample, realizes a two-dimensional scanning of the sample through a plane scanning system, and processes the extracted weak thermal signal into a two-dimensional intensity distribution map by computer to realize photothermal differential microscopy imaging of a single particle; Publication No. CN109238968A, the invention is named a A photothermal heterodyne microscopy detection system and method, the advantage of which is that the signal extraction and resolution capabilities are greatly improved by using heterodyne detection, thereby improving the signal-to-noise ratio of photothermal microscopy; the publication number is CN110470639B, and the invention is named a multi-mode scanning microscope imaging system based on laser-induced photothermal effect, and the technical solution of the system includes two observation modes: laser-induced reflectivity microscopy and laser-induced surface thermal lens microscopy, and detects and analyzes multiple parameters such as absorption distribution, thermal conductivity, doping concentration, and doping depth of different samples , which greatly broadens the application scope of the photothermal microscopy system; the patent number is CN111912813B, and the invention is named as a nano-thermal infrared microscopy device, which is used to realize the high-resolution microscopy imaging of the infrared absorbance of the sample material at the nanoscale in the technical scheme. The device includes a nano-infrared photothermal excitation module, a photothermal signal frequency domain conversion module, and an infrared absorbance microscopy module, which realizes the in-situ detection of the nanoscale photothermal fundamental frequency signal by the atomic force microscope thermal probe, and displays the high-resolution microscopy imaging results of the infrared absorbance micro-area distribution of the sample material, which is related to the nanoscale physical composition and structure of the material. The invention provides an ultra-high resolution in-situ characterization device for infrared absorbance microscopic imaging closely related to structural analysis; the publication number is CN112097951B, and the invention is applicable to the fields of image processing and microscopic imaging. The invention is named a photothermal reflection microscopic thermal imaging device and a drift correction method. In the technical solution, a modulator and a dimming device are added to the photothermal reflection microscopic thermal imaging device used, so that the influence of light source intensity drift and camera response drift on the error of the collected image can be effectively suppressed, and the temperature measurement of the static target can be achieved without modulating the temperature of the object to be measured. However, the current photothermal microscopic imaging technology is limited by the diffraction limit and cannot break through the disadvantage of the minimum resolution limit.

[0004] Therefore, it is urgent to propose a photothermal microscopy system and method based on recurrent generative adversarial neural networks to solve the above technical problems. Summary of the invention

[0005] To solve the above problems, a photothermal microscopy system and method based on a recurrent generative adversarial neural network is provided. A brief overview of the present invention is given below to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to determine the key or important parts of the present invention, nor is it intended to limit the scope of the present invention.

[0006] The technical solution of the present invention:

[0007] The photothermal microscopy imaging system based on a recurrent generative adversarial neural network comprises a reflector, a sleeve lens, a first glass slide, a scanning lens, a platform, a galvanometer controller, a dichroic mirror, a polarization cube, a reflector group, a second glass slide, a third glass slide, a spatial light filter, an acousto-optic modulator, a first filter, a polarization cube, a fourth glass slide, an excitation light source, a polarization cube, a spatial light filter, a second filter, a detection light source and a photothermal microscopy imaging system. The excitation light source, the fourth glass slide, the polarization cube, the first filter, the acousto-optic modulator, the spatial light filter, the reflector group, the polarization cube and the dichroic mirror are arranged in sequence to form a light beam path. The detection light source, the second filter, the spatial light filter, the polarization cube, the third glass slide, the second glass slide and the dichroic mirror are arranged in sequence from right to left to form an excitation laser path. The dichroic mirror, the galvanometer controller, the scanning lens, the reflector, the first glass slide and the platform are arranged in sequence. The acousto-optic modulator, the galvanometer controller and the polarization cube are all coordinated with the photothermal microscopy imaging system.

[0008] Preferably, the platform is a piezoelectric moving platform, the loading surface of the platform is used for placing the sample to be tested, and the loading surface of the platform is arranged horizontally.

[0009] Preferably: the first glass slide is a 1 / 4 glass slide (532nm), the second glass slide is a 1 / 4 glass slide (532nm), the third glass slide is a 1 / 4 glass slide (633nm), the fourth glass slide is a 1 / 2 glass slide, the first glass slide is parallel to the platform, and the second, third and fourth glass slides are perpendicular to the platform.

[0010] Preferably: it also includes a spiral phase plate, the reflector group includes a first reflector, a second reflector, a third reflector, a fourth reflector and a fifth reflector, the reflective surface of the first reflector is arranged opposite to the reflective surface of the second reflector, the first reflector is arranged parallel to the second reflector, the reflective surface of the second reflector is arranged opposite to the reflective surface of the third reflector, the reflective surface of the third reflector, the spiral phase plate, the polarization cube, and the dichroic mirror are arranged vertically in sequence from top to bottom, the angle between the dichroic mirror and the platform loading surface is 45°, the second reflector and the third reflector are arranged vertically, the second reflector located in the same horizontal plane is on the left side of the third reflector, the angle between the second reflector and the platform loading surface is 45°, the reflective surface of the fourth reflector is arranged opposite to the reflective surface of the fifth reflector, the fourth reflector is arranged parallel to the fifth reflector, the reflective surface of the fifth reflector is arranged back to back with the reflective surface of the first reflector, the fifth reflector is arranged parallel to the first reflector, and the first reflector, the second reflector, the fourth reflector and the fifth reflector are located in the same vertical plane.

[0011] Preferably: there are two spatial light filters, two acousto-optic modulators, and two first filters, the two first filters arranged up and down are both arranged correspondingly on the left side of the polarization cube, one first filter, one acousto-optic modulator, one spatial light filter, the reflection surface of the first reflector, the reflection surface of the second reflector, the reflection surface of the third reflector, the horizontal spiral phase plate, and the polarization cube are arranged in sequence to form a first light beam path, another first filter, another acousto-optic modulator, another spatial light filter, the reflection surface of the fourth reflector, the reflection surface of the fifth reflector, and the polarization cube are arranged in sequence to form a second light beam path, the polarization cube is arranged correspondingly to the upper surface of the dichroic mirror, the second glass slide is arranged correspondingly to the lower surface of the dichroic mirror, the galvanometer controller, the scanning lens, and the reflection surface of the reflector are arranged in sequence from right to left, the reflection surface of the reflector, the first glass slide, the platform, and the sample are arranged in sequence from bottom to top, and the reflector is arranged at an angle to the object plane (the horizontal plane is 45°).

[0012] Preferably: the photothermal microscopy system is a super-resolution photothermal microscopy system, comprising a computer, a data acquisition card, a detector, a phase-locked amplifier and a function generator. The computer, the data acquisition card and the phase-locked amplifier are electrically connected in sequence, the output end of the computer is electrically connected to the platform, the output end of the data acquisition card is electrically connected to the galvanometer controller, the input end of the phase-locked amplifier is electrically connected to the acousto-optic modulator through the function generator, the input end of the phase-locked amplifier is electrically connected to the detector, and the detector is connected to the polarization cube.

[0013] Preferably, it also includes a reflector and a convex lens, and the polarization cube, the long-pass filter, the convex lens, and the detector are arranged in sequence from top to bottom to form a detection optical laser path, and the convex surface of the convex lens faces upward.

[0014] The photothermal microscopy imaging method based on a recurrent generative adversarial neural network adopts the photothermal microscopy imaging system based on a recurrent generative adversarial neural network, and comprises the following steps:

[0015] Step 1: Identify the sample to be tested and place it on the stage of the piezoelectric moving stage;

[0016] Step 2: Turn on the super-resolution photothermal microscopy imaging system based on the recurrent generative adversarial neural network. This step includes turning on the computer, data acquisition card, phase-locked amplifier, galvanometer controller, function generator and other equipment;

[0017] Step 3: Turn on the power supply of the excitation light and detection light lasers and the refrigerator, so that the working temperature of the detection light laser is maintained at 20°C to ensure the normal operation of the laser equipment;

[0018] Step 4: setting detection parameters, laser power / current parameters, modulation parameters in the super-resolution photothermal microscopy imaging system software based on recurrent generative adversarial neural network in the computer, and performing scanning detection on the detection sample;

[0019] Step 5: Perform multiple tests, and the computer reads the signals at each point from the data acquisition card and processes them to obtain the test result image, summarize the test results, and generate a training data set;

[0020] Step 6: Train the cyclic generative adversarial neural network based on the training data set;

[0021] Step 7: Perform super-resolution processing on the photothermal microscopy image based on the trained P-CG network to obtain a super-resolution image;

[0022] Step 8: Change different laser parameters, detect samples, repeat steps 4 to 7, and obtain a super-resolution image set;

[0023] Step 9: After the test is completed, after a few minutes, turn off the detection light laser power supply, function generator, phase-locked amplifier, data acquisition card, galvanometer controller and computer.

[0024] Preferably, step five comprises the following steps:

[0025] Step 5.1: Input 256x256 pixel image img, point spread function PSF PT ;

[0026] Step 5.2: Randomly generate the number of particles M, M∈[20,30], i=0;

[0027] Step 5.3: Randomly generate particle diameter D, particle position X0, Y0, D∈[4.8, 7.2], X0, Y0∈[0, 255];

[0028]

[0029] x i ,y j ∈[-R:1:R];

[0030]

[0031] S=V / V0×255

[0032] img=img+S(X0, Y0), i=i+1;

[0033] Step 5.4: i ≥ M, "No" returns to step 5.3, "Yes" continues;

[0034] Step 5.5: img GT =img;

[0035] Step 5.6: Output img GT ;

[0036] or

[0037] n~N(0,σ n 2 )

[0038] img PT =PSF PT *img GT +n

[0039] img PT =↓ 4x img PT

[0040] Output img GR .

[0041] Preferably: in step 6, the data set construction process includes a network training process and a network testing process;

[0042] The network training process includes:

[0043] Step 61.1: Import img GT ,img PT ;

[0044] Step 61.2: Initialize network weights Θ, Φ~N(0, σ ini 2 ), img restored =0, i=0, epochs=100;

[0045] Step 61.3: Calculate l D (img GT ,imgrestored );

[0046] Update Φ, Φ = min Φ (l D );

[0047] img restored =G Θ (img PT )

[0048] img recoverde =↓ 4x (PSF PT *im grestored );

[0049] Step 61.4: Calculate l GD (img GT ,img restored );

[0050] Update Θ, Θ = min Θ (l G )

[0051] i=i+1;

[0052] Step 61.5: if i≥epochs, if yes, output Θ; if no, return to step 51.3.

[0053] The network testing process includes:

[0054] Step 62.1: Import img PT , Θ;

[0055] Step 62.2: img restored =G Θ (img PT );

[0056] Step 62.3: Output img restored .

[0057] The present invention has the following beneficial effects:

[0058] 1. The present invention discloses a method for photothermal three-dimensional feature reconstruction based on phase-controlled array laser beam excitation. Compared with traditional detection methods such as pulse thermal wave imaging, phase-locked thermal wave imaging and thermal wave radar imaging, this method can make full use of the three-dimensional diffusion of heat flow, and accurately reconstruct defects from a two-dimensional perspective by changing the different spatial fluctuation states of heat flow, thereby improving the detection depth and detection efficiency. At the same time, it can realize the quantitative characterization of materials and the three-dimensional tomographic reconstruction of geometric structures, and is aimed at realizing efficient and accurate detection and three-dimensional feature reconstruction of shallow surface defects (<4mm, diameter / depth ratio>1.5) of composite materials, metal materials and high molecular polymers;

[0059] 2. The present invention discloses a super-resolution photothermal microscopy system based on a recurrent generative adversarial neural network. The system uses a Gaussian light-hollow light combined beam to thermally load the sample, and can use a piezoelectric moving stage and a galvanometer to perform scanning and detection of different area sizes, thereby achieving high-precision detection of materials or particles. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of the structure of the photothermal microscopy imaging system based on the recurrent generative adversarial neural network;

[0061] Figure 2 This is a schematic diagram of the super-resolution photothermal microscopy method based on a recurrent generative adversarial neural network;

[0062] Figure 3 It is a flowchart of dataset construction;

[0063] Figure 4 It is the flow chart of P-CG network training and testing;

[0064] Figure 5 It is the processing result of P-CG network experiment.

[0065] In the figure: 1-computer, 2-data acquisition card, 3-reflector, 4-tube lens, 5-first glass slide, 6-scanning lens, 7-platform, 8-sample, 9-galvanometer controller, 10-dichroic mirror, 11-polarization cube, 12-spiral phase plate, 13-reflector group, 14-second glass slide, 15-third glass slide, 16-spatial light filter, 17-acoustic-optic modulator, 18-first filter, 19-polarization cube, 20-fourth glass slide, 21-excitation light source, 22-polarization cube, 23-reflector, 24-spatial light filter, 25-second filter, 26-detection light source, 27-convex lens, 28-detector, 29-lock-in amplifier, 30-function generator. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is described below by the specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0067] Specific implementation method 1: Combination Figure 1The present embodiment is described. The photothermal microscopy imaging system based on the recurrent generative adversarial neural network of the present embodiment includes a reflector 3, a sleeve lens 4, a first glass slide 5, a scanning lens 6, a platform 7, a galvanometer controller 9, a dichroic mirror 10, a polarization cube 11, a reflector group 13, a second glass slide 14, a third glass slide 15, a spatial light filter 16, an acousto-optic modulator 17, a first filter 18, a polarization cube 19, a fourth glass slide 20, an excitation light source 21, a polarization cube 22, a spatial light filter 24, a second filter 25, a detection light source 26 and a photothermal microscopy imaging system, wherein the excitation light source 21, the fourth glass slide The glass slide 20, polarization cube 19, first filter 18, acousto-optic modulator 17, spatial light filter 16, reflector group 13, polarization cube 11, and dichroic mirror 10 are arranged in sequence to form a light beam path. The detection light source 26, second filter 25, spatial light filter 24, polarization cube 22, third glass slide 15, second glass slide 14, and dichroic mirror 10 are arranged in sequence from right to left to form an excitation laser path. The dichroic mirror 10, galvanometer controller 9, scanning lens 6, reflector 3, first glass slide 5, and platform 7 are arranged in sequence. The acousto-optic modulator 17, galvanometer controller 9, and polarization cube 22 are all coordinated with the photothermal microscopy imaging system.

[0068] The platform 7 is a piezoelectric moving platform, and the loading surface of the platform 7 is used to place the sample 8 to be tested, and the loading surface of the platform 7 is set horizontally;

[0069] The first glass slide 5 is a 1 / 4 glass slide (532nm), the second glass slide 14 is a 1 / 4 glass slide (532nm), the third glass slide 15 is a 1 / 4 glass slide (633nm), and the fourth glass slide 20 is a 1 / 2 glass slide. The first glass slide 5 is arranged parallel to the platform 7, and the second glass slide 14, the third glass slide 15, and the fourth glass slide 20 are arranged perpendicular to the platform 7;

[0070] It also includes a spiral phase plate 12, and a reflector group 13 includes a first reflector, a second reflector, a third reflector, a fourth reflector and a fifth reflector, wherein a reflective surface of the first reflector is arranged opposite to a reflective surface of the second reflector, the first reflector is arranged parallel to the second reflector, the reflective surface of the second reflector is arranged opposite to a reflective surface of the third reflector, the reflective surface of the third reflector, the spiral phase plate 12, the polarization cube 11, and the dichroic mirror 10 are arranged vertically in sequence from top to bottom, the angle between the dichroic mirror 10 and the object-carrying surface of the platform 7 is 45°, the second reflector and the third reflector are arranged vertically, the second reflector located in the same horizontal plane is on the left side of the third reflector, the angle between the second reflector and the object-carrying surface of the platform 7 is 45°, the reflective surface of the fourth reflector is arranged opposite to a reflective surface of the fifth reflector, the fourth reflector is arranged parallel to the fifth reflector, the reflective surface of the fifth reflector is arranged back to back with the reflective surface of the first reflector, the fifth reflector is arranged parallel to the first reflector, and the first reflector, the second reflector, the fourth reflector and the fifth reflector are located in the same vertical plane;

[0071] There are two spatial light filters 16, two acousto-optic modulators 17, and two first optical filters 18. The two first optical filters 18 arranged up and down are correspondingly arranged on the left side of the polarization cube 19. One first optical filter 18, one acousto-optic modulator 17, one spatial light filter 16, the reflection surface of the first reflector, the reflection surface of the second reflector, the reflection surface of the third reflector, the horizontal spiral phase plate 12, and the polarization cube 11 are arranged in sequence to form a first light beam path. Another first optical filter 18, another acousto-optic modulator 17, and another A spatial light filter 16, a reflecting surface of a fourth reflecting mirror, a reflecting surface of a fifth reflecting mirror, and a polarization cube 11 are sequentially arranged to form a second light beam path, the polarization cube 11 is arranged corresponding to the upper surface of the dichroic mirror 10, the second glass slide 14 is arranged corresponding to the lower surface of the dichroic mirror 10, the galvanometer controller 9, the scanning lens 6, and the reflecting surface of the reflecting mirror 3 are sequentially arranged from right to left, the reflecting surface of the reflecting mirror 3, the first glass slide 5, the platform 7, and the sample 8 are sequentially arranged from bottom to top, and the reflecting mirror 3 is arranged at an angle of 45° to the object plane (the horizontal plane);

[0072] The photothermal microscopy system is a super-resolution photothermal microscopy system, including a computer 1, a data acquisition card 2, a detector 28, a phase-locked amplifier 29 and a function generator 30. The computer 1, the data acquisition card 2 and the phase-locked amplifier 29 are electrically connected in sequence. The output end of the computer 1 is electrically connected to the platform 7, the output end of the data acquisition card 2 is electrically connected to the galvanometer controller 9, the input end of the phase-locked amplifier 29 is electrically connected to the acousto-optic modulator 17 through the function generator 30, the input end of the phase-locked amplifier 29 is electrically connected to the detector 28, and the detector 28 is connected to the polarization cube 22.

[0073] It also includes a reflector 23 and a convex lens 27. The polarization cube 22, the long-pass filter 23, the convex lens 27, and the detector 28 are arranged in sequence from top to bottom to form a detection optical laser circuit, and the convex surface of the convex lens 27 faces upward.

[0074] The system uses a Gaussian light-hollow light combined beam to thermally load the sample, and can use a piezoelectric moving stage and a galvanometer to perform scanning inspections of different area sizes, thereby achieving high-precision inspections of materials or particles.

[0075] Specific implementation method 2: Combination Figure 2-5 The present embodiment is described, and the photothermal microscopy imaging method based on the cyclic generative adversarial neural network of the present embodiment is as follows: Figure 2 As shown in the figure, when the point spread function is known, the structure of the recurrent generative adversarial neural network can be simplified to two generators and one discriminator, where the first generator is responsible for the input photothermal microscopy image img PT Process and generate the restored image img restored, and upsample the input image by 4 times, for example, img PT The number of pixels of img is 64×64. After the first generator restored The number of pixels is changed to 256×256; this upsampling process helps to better restore the details of the original image and can effectively shorten the image scanning time; the second generator is responsible for converting the generated img restored Then converted into a photothermal microscopy image, recorded as img recovered ;

[0076] The steps include:

[0077] Step 1: Identify the sample to be tested and place it on the piezoelectric moving table;

[0078] Step 2: Turn on the super-resolution photothermal microscopy imaging system based on the recurrent generative adversarial neural network. This step includes turning on the computer, data acquisition card, phase-locked amplifier, galvanometer controller, function generator and other equipment;

[0079] Step 3: Turn on the excitation light, detection light laser power supply and refrigerator, and then maintain the operating temperature of the laser power supply at 20°C to ensure the normal operation of the laser equipment;

[0080] Step 4: setting detection parameters, laser power / current parameters, modulation parameters in the super-resolution photothermal microscopy imaging system software based on recurrent generative adversarial neural network in the computer, and performing scanning detection on the detection sample;

[0081] Step 5: Perform multiple tests, and the computer reads the signals at each point from the data acquisition card and processes them to obtain the test result image. The test results are summarized to generate a training data set, such as Figure 3 As shown;

[0082] Step 6: Train the cyclic generative adversarial neural network based on the training data set obtained in step 5. The training and testing process is as follows: Figure 4 As shown;

[0083] Step 7: Perform super-resolution processing on the photothermal microscopy image based on the trained P-CG network to obtain a super-resolution image;

[0084] Step 8: Change different laser parameters, detect samples, repeat steps 4 to 7, and obtain a super-resolution image set;

[0085] Step 9: After the test, turn off the laser power supply, function generator, phase-locked amplifier, data acquisition card, galvanometer controller, computer and other equipment after 5 minutes;

[0086] Make full use of the thermal lens effect of materials to achieve three-dimensional accurate detection of defect size / morphology. Based on vector diffraction theory, heat transfer and electromagnetic field theory, it can realize super-resolution photothermal microscopic imaging detection (with a minimum image resolution of 60nm) and three-dimensional feature reconstruction for shallow surface defects of composite materials, metal materials and polymers. This detection method and system are suitable for accurate non-destructive detection and evaluation of material defects / damage in aerospace, microelectronics and micro-nano structures.

[0087] The present invention discloses a photothermal three-dimensional feature reconstruction method based on phase-controlled array laser beam excitation. Compared with traditional detection methods such as pulse thermal wave imaging, phase-locked thermal wave imaging and thermal wave radar imaging, this method can make full use of the three-dimensional diffusion of heat flow, and accurately reconstruct defects from a two-dimensional perspective by changing different spatial fluctuation states of heat flow, thereby improving the detection depth and detection efficiency. At the same time, it can realize quantitative characterization of materials and three-dimensional tomographic reconstruction of geometric structures, especially realizing efficient and accurate detection and three-dimensional feature reconstruction of shallow surface defects (<4mm, diameter / depth ratio>1.5) of composite materials, metal materials and high-molecular polymers.

[0088] Specific implementation method three: Combination Figure 1-5 The present embodiment is described. The photothermal microscopy imaging method based on a cyclic generative adversarial neural network of the present embodiment adopts a photothermal microscopy imaging system based on a cyclic generative adversarial neural network. The present invention is applicable to the field of nondestructive testing of conductive materials such as aerospace, microelectronics and micro-nano structures.

[0089] The computer 1 of the system is provided with two signal terminals, one of the signal output terminals of the computer 1 is connected to the signal input terminal of the piezoelectric moving stage 7, and the second signal input terminal of the computer 1 is connected to the signal output terminal of the data acquisition card 2; the data acquisition card is provided with two groups of signal terminals, one of the signal output terminals of the data acquisition card 2 is connected to the signal input terminal of the galvanometer controller 9, and the signal input terminal of the data acquisition card 2 is connected to the signal output terminal of the phase-locked amplifier 29; the first signal input terminal of the phase-locked amplifier 29 is connected to the signal output terminal of the detector 28, and the second signal input terminal of the phase-locked amplifier 29 is connected to the signal output terminal of the function generator 30; the signal output terminal of the function generator 30 is connected to the signal input terminal of the acousto-optic modulator 17; the excitation light source 21 emits an excitation laser, which is divided into two paths at the polarization cube 19 after passing through the 1 / 2 glass slide 20, the first path is polarized by the spiral phase plate 12 after passing through the filter 18, the acousto-optic modulator 17, the spatial light filter 16, and the reflector group 13, and the second path is polarized by the spiral phase plate 12 after passing through the filter 18, the acousto-optic modulator 17, the spatial light filter After the detector 16 and the reflector group 13, the first beam is combined with the first beam at the polarization cube 11 and emitted to the dichroic mirror 10; the detection light source 26 emits a detection laser, which passes through the filter 25, the spatial light filter 24, the polarization cube 22, the 1 / 4 glass (633nm) 15, and the 1 / 4 glass (532nm) 14, and then is combined with the excitation laser at the dichroic mirror 10; the mixed laser starts from the dichroic mirror 10, passes through the galvanometer controller 9, the scanning lens 6, the sleeve lens 4, the reflector 3, the 1 / 4 glass (532nm) After m5, the sample 8 to be tested placed on the piezoelectric moving stage 7 is irradiated, and scanning detection is realized by the movement of the piezoelectric moving stage 7 and the setting of the galvanometer controller 9; the reflected light irradiated on the sample 8 to be tested passes through the 1 / 4 glass slide (532nm) 5, the reflector 3, the sleeve lens 4, the scanning lens 6, the galvanometer controller 9, the dichroic mirror 10, the 1 / 4 glass slide (532nm) 14, the 1 / 4 glass slide (633nm) 15, and the polarization cube 22, and then is directed to the long-pass filter 23, and is focused by the convex lens 27 and then directed to the detector 28;

[0090] The method comprises the following steps:

[0091] Step 1: Identify the sample 8 to be tested and place the sample 8 on the stage of the piezoelectric moving stage;

[0092] Step 2: Turn on the super-resolution photothermal microscopy imaging system based on the recurrent generative adversarial neural network. This step includes turning on the computer 1, the data acquisition card 2, the phase-locked amplifier 29, the galvanometer controller 9, the function generator 30 and other devices;

[0093] Step 3: Turn on the power supply and refrigerator of the excitation light (excitation light source 21), the detection light laser (detection light source 26), and the refrigerator, so as to maintain the working temperature of the detection light laser at 20° C. to ensure the normal operation of the laser equipment;

[0094] Step 4: setting detection parameters, laser power / current parameters, and modulation parameters in the super-resolution photothermal microscopy imaging system software based on recurrent generative adversarial neural network in the computer 1, and performing scanning detection on the detection sample 8;

[0095] Step 5: Perform multiple tests, and the computer 1 reads the signals at each point from the data acquisition card 2 and obtains the test result image after processing. The test results are summarized to generate a training data set, such as Figure 3 As shown;

[0096] Step 6: Train the cyclic generative adversarial neural network based on the training data set obtained in step 5. The training and testing process is as follows: Figure 4 As shown;

[0097] Step 7: Perform super-resolution processing on the photothermal microscopy image based on the trained P-CG network to obtain a super-resolution image;

[0098] Step 8: Change different laser parameters, detect samples, repeat steps 4 to 7, and obtain a super-resolution image set;

[0099] Step 9: After the test is finished, after an interval of 5 minutes, turn off the detection light laser power supply, function generator 30, phase-locked amplifier 29, data acquisition card 2, galvanometer controller 9 and computer 1 and other equipment;

[0100] Step five includes the following steps:

[0101] Step 5.1: Input 256x256 pixel image img, point spread function PSF PT ;

[0102] Step 5.2: Randomly generate the number of particles M, M∈[20,30], i=0;

[0103] Step 5.3: Randomly generate particle diameter D, particle position X0, Y0, D∈[4.8, 7.2], X0, Y0∈[0, 255];

[0104]

[0105] x i ,y j ∈[-R:1:R];

[0106]

[0107] S=V / V0×255

[0108] img=img+S(X0, Y0), i=i+1;

[0109] Where: R – radius of generated particles;

[0110] x i ,y i ——The distance from the center of the ith nanoparticle;

[0111] V——the volume of the ith simulated nanoparticle;

[0112] V0——the volume of the first simulated nanoparticle;

[0113] S——relative volume information of nanoparticles in the image;

[0114] Step 5.4: i ≥ M, "No" returns to step 5.3, "Yes" continues;

[0115] Step 5.5: img GT =img;

[0116] Where: img GT ——Real image;

[0117] Step 5.6: Output img GT ;

[0118] and / or

[0119] n~N(0,σ n 2 )

[0120] img PT =PSF PT *img GT +n

[0121] img PT =↓ 4x img PT

[0122] Output img GT ;

[0123] Where: n——measurement noise signal;

[0124] n~N(0,σ n 2 )——The measurement noise signal n conforms to the Gaussian normal distribution, where the expected value is 0 and the standard deviation is σ n ;img PT - Photothermal microscopy images fed into the neural network;

[0125] PSFPT ——Photothermal microscopy point spread function;

[0126] ↓ 4x ——Downsampling 4 times;

[0127] In step six, the data set construction process includes the network training process and the network testing process;

[0128] The network training process includes:

[0129] Step 61.1: Import img GT ,img PT ;

[0130] Step 61.2: Initialize network weights Θ, Φ~N(0, σ ini 2 ), img restored =0, i=0, epochs=100;

[0131] Where: Φ——network parameters of the discriminator;

[0132] Φ~N(0,σ ini 2 )——The network parameter Φ of the discriminator is distributed in accordance with the Gaussian normal distribution, and its standard deviation is σ ini ;

[0133] img restored ——Photothermal microscopy images generated by neural networks;

[0134] epochs — the number of iterations of the neural network algorithm;

[0135] Step 61.3: Calculate lD(img GT ,img restored );

[0136] Update Φ, Φ = min Φ (l D );

[0137] img restored =G Θ (im gP T)

[0138] img recovered =↓ 4x (PSF PT *img restored );

[0139] Step 61.4: Calculate l GD (img GT ,img restored );

[0140] Update Θ, Θ = min Θ (l G )

[0141] i=i+1;

[0142] Where: l GD ——The loss function of the discriminator;

[0143] Θ — network parameters of the generator;

[0144] l G ——The loss function when training the generator;

[0145] min Θ (l G )——The loss function l when training the generator G Achieving the minimum generator network parameter value;

[0146] Step 61.5: i ≥ epochs, if yes, output Θ, if no, return to step 51.3;

[0147] The network testing process includes:

[0148] Step 62.1: Import img PT , Θ;

[0149] Step 62.2: img restored =GΘ(img PT );

[0150] Step 62.3: Output img restored ;

[0151] Figure 5 The results given are super-resolution photothermal microscopy imaging results obtained after P-CG network processing;

[0152] The present invention solves the problem that the current photothermal microscopy technology is limited by the diffraction limit and cannot break through the minimum resolution limit. The super-resolution photothermal microscopy method and system based on the recurrent generative adversarial neural network integrates multiple technical fields such as thermal lens characteristic mechanism, photothermal signal modulation, far-field microscopy, multi-source signal unmixing and feature extraction, image deconvolution, and deep learning. It can realize super-resolution photothermal microscopy detection (with a minimum image resolution of 60nm) and three-dimensional feature reconstruction for shallow surface defects of composite materials, metal materials and polymers.

[0153] It should be noted that in the above embodiments, as long as the technical solutions are not contradictory, they can be arranged and combined, and those skilled in the art can exhaust all possibilities based on the mathematical knowledge of arrangement and combination. Therefore, the present invention will no longer describe the technical solutions after arrangement and combination one by one, but it should be understood that the technical solutions after arrangement and combination have been disclosed by the present invention.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A photothermal microscopy system based on a recurrent generative adversarial neural network, characterized by: The invention comprises a reflector (3), a sleeve lens (4), a first glass slide (5), a scanning lens (6), a platform (7), a galvanometer controller (9), a dichroic mirror (10), a polarization cube (11), a reflector group (13), a second glass slide (14), a third glass slide (15), a spatial light filter (16), an acousto-optic modulator (17), a first filter (18), a polarization cube (19), a fourth glass slide (20), an excitation light source (21), a polarization cube (22), a spatial light filter (24), a second filter (25), a detection light source (26) and a photothermal microscopy imaging system, wherein the excitation light source (21), the fourth glass slide (20), the polarization cube (1 9), a first filter (18), an acousto-optic modulator (17), a spatial light filter (16), a reflector group (13), a polarization cube (11), and a dichroic mirror (10) are arranged in sequence; a detection light source (26), a second filter (25), a spatial light filter (24), a polarization cube (22), a third glass slide (15), a second glass slide (14), and a dichroic mirror (10) are arranged in sequence; a dichroic mirror (10), a galvanometer controller (9), a scanning lens (6), a reflector (3), a first glass slide (5), and a platform (7) are arranged in sequence; and an acousto-optic modulator (17), a galvanometer controller (9), and a polarization cube (22) are all coordinated with a photothermal microscopy imaging system.

2. The photothermal microscopy imaging system based on a recurrent generative adversarial neural network according to claim 1, characterized in that: The platform (7) is a piezoelectric moving platform.

3. The photothermal microscopy imaging system based on recurrent generative adversarial neural network according to claim 1, characterized in that: The first glass slide (5) is a 1 / 4 glass slide, the second glass slide (14) is a 1 / 4 glass slide, the third glass slide (15) is a 1 / 4 glass slide, and the fourth glass slide (20) is a 1 / 2 glass slide. The first glass slide (5) is arranged parallel to the platform (7), and the second glass slide (14), the third glass slide (15), and the fourth glass slide (20) are arranged perpendicular to the platform (7).

4. The photothermal microscopy imaging system based on a recurrent generative adversarial neural network according to claim 1 or 2, characterized in that: It also comprises a spiral phase plate (12); a reflector group (13) comprises a first reflector, a second reflector, a third reflector, a fourth reflector and a fifth reflector; a reflective surface of the first reflector is arranged opposite to a reflective surface of the second reflector; the first reflector is arranged parallel to the second reflector; a reflective surface of the second reflector is arranged opposite to a reflective surface of the third reflector; a reflective surface of the third reflector, a spiral phase plate (12), a polarization cube (11) and a dichroic mirror (10) are arranged in sequence; an angle between the dichroic mirror (10) and the platform (7) is 45°; the second reflector and the third reflector are arranged vertically; an angle between the second reflector and the platform (7) is 45°; a reflective surface of the fourth reflector is arranged opposite to a reflective surface of the fifth reflector; the fourth reflector is arranged parallel to the fifth reflector; a reflective surface of the fifth reflector is arranged opposite to a reflective surface of the first reflector; and the fifth reflector is arranged parallel to the first reflector.

5. The photothermal microscopy imaging system based on recurrent generative adversarial neural network according to claim 4, characterized in that: There are two spatial optical filters (16), two acousto-optic modulators (17), and two first optical filters (18), and the two first optical filters (18) are arranged on the left side of the polarization cube (19) respectively. One first optical filter (18), one acousto-optic modulator (17), one spatial optical filter (16), and a first reflector are arranged in sequence from right to left, and another first optical filter (18), another acousto-optic modulator (17), another spatial optical filter (16), and a fourth reflector are arranged in sequence from right to left.

6. The photothermal microscopy imaging system based on recurrent generative adversarial neural network according to claim 5, characterized in that: The photothermal microscopy imaging system comprises a computer (1), a data acquisition card (2), a detector (28), a phase-locked amplifier (29) and a function generator (30). The computer (1), the data acquisition card (2) and the phase-locked amplifier (29) are connected in sequence. The output end of the computer (1) is connected to a platform (7), the output end of the data acquisition card (2) is connected to a galvanometer controller (9), the input end of the phase-locked amplifier (29) is connected to an acousto-optic modulator (17) through the function generator (30), the input end of the phase-locked amplifier (29) is connected to the detector (28), and the detector (28) is connected to a polarization cube (22).

7. The photothermal microscopy imaging system based on recurrent generative adversarial neural network according to claim 4, characterized in that: It also includes a reflector (23) and a convex lens (27), and a polarization cube (22), a long-pass filter (23), a convex lens (27), and a detector (28) are arranged in sequence.

8. A photothermal microscopy imaging method based on a recurrent generative adversarial neural network, characterized in that: The photothermal microscopy imaging system based on a cyclic generative adversarial neural network according to any one of claims 1 to 7 comprises the following steps: Step 1: placing the sample (8) on the piezoelectric moving stage; Step 2: Turn on the computer (1), the data acquisition card (2), the lock-in amplifier (29), the galvanometer controller (9), and the function generator (30); Step 3: Turn on the excitation light source 21 and the detection light source 26; Step 4: setting detection parameters, laser power / current parameters, and modulation parameters in the computer (1), and performing scanning detection on the detection sample (8); Step 5: Perform multiple tests, the computer (1) reads the signals of each point from the data acquisition card (2) and obtains the test result image after processing, summarizes the test results, and generates a training data set; Step 6: Train the cyclic generative adversarial neural network based on the training data set; Step 7: Perform super-resolution processing on the photothermal microscopy image based on the trained P-CG network to obtain a super-resolution image; Step 8: Change different laser parameters, detect samples, repeat steps 4 to 7, and obtain a super-resolution image set; Step 9: After the test is finished, turn off the power supply of the detection light laser, the function generator (30), the phase-locked amplifier (29), the data acquisition card (2), the galvanometer controller (9) and the computer (1).

9. The photothermal microscopy imaging method based on a recurrent generative adversarial neural network according to claim 8, characterized in that: Step five includes the following steps: Step 5.1: Input 256x256 pixel image img, point spread function PSF PT ; Step 5.2: Randomly generate the number of particles M, M∈[20,30], i=0; Step 5.3: Randomly generate particle diameter D, particle position X0, Y0, D∈[4.8, 7.2], X0, Y0∈[0, 255]; x i ,y j ∈[-R:1:R]; S=V / V0×255 img=img+S(X0, Y0), i=i+1; Step 5.4: if i≥M, "No" returns to step 5.3, "Yes" continues; Step 5.5: img GT =img; Step 5.6: Output img GT ; or n~N(0,σ n 2 ) img PT =PSF PT *img GT +n img PT =↓ 4x img PT Output img GT .

10. The photothermal microscopy imaging method based on recurrent generative adversarial neural network according to claim 9, characterized in that: Step six includes the network training process and the network testing process; The network training process includes: Step 61.1: Import img GT ,img PT ; Step 61.2: Initialize network weights Θ, Φ~N(0, σ ini 2 ), img restored =0, i=0, epochs=100; Step 61.3: Calculate l D (img GT ,img restored ); Update Φ, Φ = min Φ (l D ); img restored =G Θ (img PT ) img recovered =↓ 4x (PSF PT *im grestored ); Step 61.4: Calculate l GD (img GT ,img restored ); Update Θ, Θ=min Θ (l G ) i=i+1; Step 61.5: if i≥epochs, if “yes”, output Θ; if “no”, return to step 51.

3. The network testing process includes: Step 62.1: Import img PT ,Θ; Step 62.2: img restored =G Θ (img PT ); Step 62.3: Output img restored .

Images