A multi-band fusion-based target positioning photoelectric detection system and method

Abstract

The application provides a target positioning photoelectric detection system and method based on multi-band fusion, relates to the technical field of optical detection, and obtains a first input data set of a photoelectric detection target based on a terahertz quantum pulse source; a dynamic bias current is applied to the photoelectric detection target to obtain a second input data set; then the first input data set and the second input data set are fused in features to obtain a multi-modal feature matrix; the multi-modal feature matrix is fused in features for monitoring to obtain a detection discrimination matrix and a fusion feature vector, the micro-evolution trend of defects of the photoelectric detection target is determined based on the fusion feature vector and the detection discrimination matrix; the weight parameters of a multi-band deep fusion network are updated in negative feedback according to the micro-evolution trend, and then the defect result of the photoelectric detection target is output based on the updated multi-band deep fusion network, so that the application can realize micro-positioning and evolution prediction of defects based on multi-modal fusion detection, and the precision of photoelectric detection is improved.

Description

Technical Field

[0001] This application relates to the field of optical detection technology, and more specifically, to a target positioning photoelectric detection system and method based on multi-band fusion. Background Technology

[0002] Optical inspection is commonly used in materials analysis, device defect detection, and performance evaluation. Common optical inspection methods include infrared spectroscopy, electroluminescence imaging, Raman spectroscopy, and terahertz time-domain spectroscopy. Among them, terahertz detection can penetrate non-conductive materials to detect internal structures, while electroluminescence imaging can directly present defect distribution information in the device's operating state. Optical inspection has the characteristics of being non-contact, highly sensitive, and high-resolution, and has become an important inspection method in the fields of semiconductors, optoelectronic devices, and precision manufacturing.

[0003] Multi-band fusion-based target localization photoelectric detection fuses time-domain reflection signals, spectral fingerprints, and phase delay features from different terahertz frequency bands. This allows for a more comprehensive characterization of the physical and structural features of the target, thereby improving defect localization accuracy. However, existing multi-band fusion-based target localization photoelectric detection methods only fuse data from different terahertz bands, lacking joint analysis with other detection modalities such as electroluminescence images. This results in incomplete characterization information, and the use of fixed feature weights prevents the integration of time-series information to model and predict the microscopic evolution of defects. This limits their application value in device lifetime prediction and failure early warning, and fails to fully utilize the feature extraction and adaptive optimization capabilities of deep learning networks. Therefore, how to achieve microscopic defect localization and evolution prediction based on multi-modal fusion detection to improve the accuracy of photoelectric detection remains a challenge for the industry. Summary of the Invention

[0004] This application provides a target positioning photoelectric detection system and method based on multi-band fusion, which can realize the microscopic positioning and evolution prediction of defects based on multi-modal fusion detection, so as to improve the accuracy of photoelectric detection.

[0005] In a first aspect, this application provides a target localization photoelectric detection method based on multi-band fusion, the photoelectric detection method comprising the following steps: The time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies are obtained based on the terahertz quantum pulse source, and then the first input dataset is obtained. A dynamic bias current is applied to the photoelectric detection target, and then the electroluminescence image of the photoelectric detection target is acquired. The second input dataset is extracted from the electroluminescence image. After joint preprocessing of the first input dataset and the second input dataset, a feature fusion operation is performed to obtain a multimodal feature matrix; The multi-band deep fusion network based on cross-domain feature interaction mechanism is used to perform feature monitoring and fusion on the multi-modal feature matrix to obtain a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, the defects of the photoelectric detection target are microscopically located and monitored to obtain the microscopic evolution trend of the defects of the photoelectric detection target. The weight parameters of the multi-band deep fusion network are updated with negative feedback based on the micro-evolution trend, and then the defect results of the photoelectric detection target are output based on the updated multi-band deep fusion network.

[0006] In this embodiment, the spectral fingerprint is a set of characteristic spectral lines based on terahertz band signals, and the set of characteristic spectral lines includes: absorption peaks, phase delays, and time-domain attenuation features.

[0007] In this embodiment, a dynamic bias current is applied to the photoelectric detection target by an adjustable current source, which is adjusted based on the intensity of the detected electroluminescence signal.

[0008] In this embodiment, an infrared imaging device is used to acquire the electroluminescence pattern of the photoelectric detection target.

[0009] In this embodiment, extracting the second input dataset from the electroluminescence image specifically includes: The electroluminescence image is subjected to noise reduction, brightness normalization, and background light interference suppression to obtain a defect mask; The second input dataset is formed by extracting brightness distribution features and geometric contour parameters from the defect mask using a deep learning segmentation network.

[0010] In this embodiment, performing joint preprocessing on the first input dataset and the second input dataset followed by feature fusion to obtain a multimodal feature matrix specifically includes: Based on the artifact removal algorithm, synchronous noise suppression is performed on all data in the first input dataset and all data in the second input dataset to obtain a noise-suppressed multimodal dataset; Spatiotemporal feature alignment and fusion are performed on the noise-suppressed multimodal dataset to obtain the multimodal feature matrix.

[0011] In this embodiment, a multi-band deep fusion network based on a cross-domain feature interaction mechanism is used to perform feature monitoring and fusion on the multimodal feature matrix to obtain a detection and discrimination matrix and a fused feature vector, specifically including: The terahertz band signal in the multimodal feature matrix is ​​extracted using a time-frequency convolutional sub-network in a multi-band deep fusion network to obtain a terahertz feature map. The texture pattern and brightness distribution of the electroluminescence map in the multimodal feature matrix are encoded using the image convolutional coding subnetwork in the multi-band deep fusion network to obtain the defect space feature map; The terahertz feature map and the defect space feature map are subjected to cross-domain interaction to obtain a fused feature vector; The multimodal feature matrix is ​​reconstructed by weighting based on the fused feature vector to obtain the detection and discrimination matrix.

[0012] In this embodiment, the microscopic location monitoring of defects in the photoelectric detection target based on the fused feature vector and the detection discrimination matrix, and the resulting microscopic evolution trend of the defects in the photoelectric detection target, specifically include: Identify candidate defect regions for photoelectric detection targets; The defect candidate region is located at the sub-pixel level to obtain the microscopic spatial coordinates of the defect in the photoelectric detection target; The defect candidate region is used to determine the micro-defect type by the fused feature vector, and the defect type result is obtained. Based on the detection discrimination matrix and the defect type results, the evolution prediction of the defect type in the microscopic spatial coordinates is performed to obtain the microscopic evolution trend of the photoelectric detection target defect.

[0013] In this embodiment, the cross-domain feature interaction mechanism refers to the interactive fusion of multimodal features in the time-frequency domain and the spatial domain.

[0014] Secondly, this application provides a target positioning photoelectric detection system based on multi-band fusion, used to execute a target positioning photoelectric detection method based on multi-band fusion, the photoelectric detection system comprising: The terahertz feature acquisition module is used to acquire the time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies based on the terahertz quantum pulse source, thereby obtaining the first input dataset. An electroluminescence imaging module is used to apply a dynamic bias current to a photoelectric detection target, thereby acquiring an electroluminescence image of the photoelectric detection target and extracting a second input dataset from the electroluminescence image; The multimodal feature preprocessing module is used to perform feature fusion operation after jointly preprocessing the first input dataset and the second input dataset to obtain a multimodal feature matrix; The cross-domain deep fusion analysis module is used to perform feature monitoring and fusion on the multimodal feature matrix using a multi-band deep fusion network based on a cross-domain feature interaction mechanism, to obtain a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, the module performs microscopic positioning and monitoring of defects in the photoelectric detection target, and obtains the microscopic evolution trend of the defects in the photoelectric detection target. An adaptive feedback optimization module is used to perform negative feedback updates on the weight parameters of the multi-band deep fusion network based on the micro-evolution trend, and then output the defect results of the photoelectric detection target based on the updated multi-band deep fusion network.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies are obtained based on a terahertz quantum pulse source to obtain a first input dataset. A dynamic bias current is applied to the photoelectric detection target to collect its electroluminescence image, and a second input dataset is extracted from the electroluminescence image. The first and second input datasets are jointly preprocessed and then a feature fusion operation is performed to obtain a multimodal feature matrix. The multimodal feature matrix is ​​fused using a multi-band deep fusion network based on a cross-domain feature interaction mechanism to obtain a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, the defects of the photoelectric detection target are microscopically located and monitored to obtain the microscopic evolution trend of the defects. The weight parameters of the multi-band deep fusion network are updated with negative feedback according to the microscopic evolution trend, and the defect results of the photoelectric detection target are output based on the updated multi-band deep fusion network.

[0016] Therefore, this application demonstrates the ability to achieve microscopic defect localization and evolution prediction based on multimodal fusion detection. First, by acquiring the time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies using a terahertz quantum pulse source, it provides highly sensitive internal structure characterization data for subsequent defect detection. Then, by applying a dynamic bias current to the photoelectric detection target, its electroluminescence map is acquired, directly reflecting the spatial brightness distribution and electrical mismatch region of the defect under device operating conditions, achieving accurate acquisition of surface and near-surface microscopic defects. Second, by performing feature fusion after joint preprocessing of the terahertz quantum pulse signal and the electroluminescence map, a multimodal feature matrix is ​​obtained. This eliminates noise differences and feature redundancy between multi-source data, unifying time-frequency information and spatial image information into a fused feature space, providing a basis for cross-modal detection. Modal information interaction and multi-dimensional defect analysis provide high-quality input data. Then, a multi-band deep fusion network based on a cross-domain feature interaction mechanism is used to perform feature monitoring and fusion on the multi-modal feature matrix. This enables deep interaction between terahertz features and electroluminescence features, forming a high-precision model of the spatial location, type, and micro-evolution of defects, significantly improving the accuracy and predictability of defect detection. Finally, the weight parameters of the multi-band deep fusion network are updated with negative feedback based on the micro-evolution trend. Based on the updated multi-band deep fusion network, the defect results of the photoelectric detection target are output, which can construct an adaptive detection closed loop. This allows the detection model to continuously optimize the feature extraction weights, improve the robustness of defect identification, and thus realize the micro-localization and evolution prediction of defects based on multi-modal fusion detection, which is beneficial to improving the overall accuracy of photoelectric detection.

[0017] In summary, the technical solution adopted in this application can realize microscopic defect localization and evolution prediction based on multimodal fusion detection, thereby improving the accuracy of photoelectric detection. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a target positioning photoelectric detection method based on multi-band fusion provided in this application; Figure 2 This is an exemplary flowchart for determining a multimodal feature matrix according to the present application; Figure 3 This is an exemplary flowchart for determining the microscopic evolution trend of defects in photoelectric detection targets, provided in this application. Figure 4 This is a module structure diagram of a target positioning photoelectric detection system based on multi-band fusion provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a target localization photoelectric detection system and method based on multi-band fusion. The core of the system is to acquire the time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies using a terahertz quantum pulse source, thereby obtaining a first input dataset. A dynamic bias current is applied to the photoelectric detection target to acquire its electroluminescence image, from which a second input dataset is extracted. The first and second input datasets are jointly preprocessed, followed by feature fusion to obtain a multimodal feature matrix. A multi-band deep fusion network based on a cross-domain feature interaction mechanism is used to perform feature monitoring and fusion on the multimodal feature matrix, obtaining a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, microscopic localization monitoring of defects in the photoelectric detection target is performed to obtain the microscopic evolution trend of the defects. The weight parameters of the multi-band deep fusion network are updated with negative feedback according to the microscopic evolution trend, and the updated multi-band deep fusion network outputs the defect results of the photoelectric detection target.

[0022] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown in the figure, this is an exemplary flowchart of a target positioning photoelectric detection method based on multi-band fusion according to this embodiment of the present application. The photoelectric detection method includes the following steps: In step S1, the time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies are obtained based on the terahertz quantum pulse source, thereby obtaining the first input dataset.

[0023] In practice, firstly, a terahertz quantum pulse source emits a pulse signal towards the photoelectric detection target, and a terahertz receiver detector receives the reflected time-domain signal to obtain time-domain reflection signals at different frequencies. Then, a fast Fourier transform is used to convert the acquired time-domain signal into a frequency-domain signal. Next, a polynomial baseline fitting algorithm is used to perform baseline correction on the frequency-domain signal, and the absorption peak is extracted using the first derivative method. Then, a phase unwrapping algorithm is used to unwrap the phase spectrum of the frequency-domain signal, and the phase delay is calculated using the phase difference calculation formula. Finally, a Hilbert transform is used to extract the amplitude decay curve of the time-domain signal on the time axis, and linear regression is used to calculate... The slope of the amplitude attenuation curve is used as a time-domain attenuation feature. Finally, the set of absorption peaks, phase delays, and time-domain attenuation features is used as a spectral fingerprint. Based on the above steps, pulse signals in the range of 0.1-10 THz are emitted to the photoelectric detection target to obtain the time-domain reflection signals and spectral fingerprints of the photoelectric detection target at different frequencies. The set of all time-domain reflection signals and all spectral fingerprints is then used as the first input dataset. Preferably, in this application, five consecutive pulse signals with equal range differences can be emitted to the photoelectric detection target, which is beneficial for analyzing and extracting the absorption peaks, phase delays, and time-domain attenuation features of the terahertz band signal.

[0024] It should be noted that the spectral fingerprint in this application is a set of characteristic spectral lines based on terahertz band signals. The set of characteristic spectral lines includes: absorption peaks, phase delays, and time-domain attenuation features. The absorption peak refers to the point where the amplitude of the reflected signal decreases. The phase delay refers to the amount of phase change that occurs after the signal penetrates the photoelectric detection target. The time-domain attenuation feature refers to the rate of amplitude decrease of the terahertz wave signal on the time axis due to multiple scattering, absorption, and energy dissipation. In addition, the phase unwrapping algorithm in this embodiment refers to obtaining the phase distribution curve by smoothly connecting abrupt change points after judging the continuity of the phase curve.

[0025] In step S2, a dynamic bias current is applied to the photoelectric detection target, and then the electroluminescence pattern of the photoelectric detection target is acquired, and the second input dataset is extracted from the electroluminescence pattern.

[0026] In practice, a dynamic bias current is applied to the photoelectric detection target through an adjustable current source, which is adjusted based on the intensity of the detected electroluminescence signal. In actual implementation, the adjustable DC current source can be connected to the positive and negative electrodes of the photoelectric detection target. The connection method can adopt a standard four-wire connection, that is, two wires are used for output current and two wires are used for voltage detection, which helps to reduce voltage errors caused by line resistance.

[0027] It should be noted that the adjustable current source in this application refers to an electronic instrument that can output a stable and controllable current and supports receiving remote control commands through an external control interface (such as GPIB, USB or LAN interface); the dynamic bias current is automatically adjusted according to the real-time feedback of the electroluminescence signal intensity during the test, which can compensate for the fluctuation of luminescence intensity caused by differences in device material characteristics, changes in contact resistance and changes in external ambient temperature.

[0028] In practice, an infrared imaging device is used to acquire the electroluminescence image of the photoelectric detection target. Preferably, the infrared imaging device can be a short-wave infrared camera based on an InGaAs sensor, which helps to ensure that the imaging covers the entire luminescent area of ​​the device under test. In actual implementation, an adjustable current source is started to apply a stable bias current to the photoelectric detection target. After the current and luminescence intensity stabilize, the surface area of ​​the photoelectric detection target is photographed by the short-wave infrared camera based on an InGaAs sensor to obtain the electroluminescence image.

[0029] It should be noted that the electroluminescence image in this application is an infrared light signal intensity distribution image, used to describe the electrical characteristics and defect distribution state inside the device. The bright areas usually correspond to normal areas with high carrier recombination efficiency, while the low-brightness or dark spot areas correspond to possible material defects, interface damage or leakage channels. The electroluminescence image is in an uncompressed format such as TIFF or RAW, which can prevent artifacts and information loss introduced by compression algorithms and is beneficial to improving the accuracy of feature extraction and deep learning model training.

[0030] In this embodiment, the extraction of the second input dataset from the electroluminescence image can be specifically carried out in the following manner: The electroluminescence image is subjected to noise reduction, brightness normalization, and background light interference suppression to obtain a defect mask; The second input dataset is formed by extracting brightness distribution features and geometric contour parameters from the defect mask using a deep learning segmentation network.

[0031] In specific implementation, firstly, a median filtering algorithm can be used to denoise the electroluminescent image. That is, median filtering replaces the center pixel value with the median gray value within a local window, thereby removing isolated high-frequency noise points (such as thermal noise and camera dark current noise). The average brightness value of the electroluminescent image is calculated, and then the brightness of each pixel is normalized according to the average brightness value to map it to the 0-1 interval. The normalization process can be implemented using Z-score. Then, an adaptive threshold segmentation algorithm (such as the Otsu algorithm) is used to extract the luminescent region of the electroluminescent image. The gray value of the background region is set to zero, which can suppress the false luminescent signal caused by the background light, and the extracted luminescent region is used as a defect mask. Then, a connected component labeling algorithm can be used to filter out regions that are too small or have insufficient brightness in the defect mask, resulting in multiple omnidirectional connected regions. These connected regions are then input into a deep learning segmentation network for pixel-level segmentation, yielding a binarized mask image of the defect mask. The average brightness, brightness standard deviation, and brightness histogram of the binarized mask image are then used as brightness distribution features, while the region area, aspect ratio, and centroid coordinates of the binarized mask image are used as geometric contour parameters. The structured feature vector combining the brightness distribution features and geometric contour parameters is then used as the second input dataset.

[0032] It should be noted that the defect mask in this application is a binary representation image of the defect area of ​​the photoelectric detection target, used to separate the luminous area from the background and clarify the spatial distribution location of the defect. The brightness distribution feature refers to the statistical characteristics of the pixel grayscale values ​​in the defect mask area, including average brightness, brightness standard deviation, and brightness histogram, used to measure the luminous energy distribution of the defect area. The geometric contour parameters describe the spatial morphological features of the defect and are used for defect classification, localization, and microscopic evolution trend analysis. In addition, the median filtering in this embodiment is a nonlinear filtering method that can be used to suppress bright spots or dark spots caused by random noise in optical imaging. High-precision pixel-level segmentation is achieved through a deep learning segmentation network, which is beneficial for defect recognition in complex backgrounds.

[0033] In step S3, the first input dataset and the second input dataset are jointly preprocessed and then a feature fusion operation is performed to obtain a multimodal feature matrix.

[0034] Preferably, in this embodiment, reference Figure 2 As shown, this figure is an exemplary flowchart of determining the multimodal feature matrix according to the present application. In this embodiment, after joint preprocessing of the first input dataset and the second input dataset, a feature fusion operation is performed to obtain the multimodal feature matrix. The specific steps are as follows: In step S31, synchronous noise suppression is performed on all data in the first input dataset and all data in the second input dataset based on the artifact elimination algorithm to obtain a noise-suppressed multimodal dataset. In step S32, spatiotemporal feature alignment and fusion are performed on the noise-suppressed multimodal dataset to obtain a multimodal feature matrix.

[0035] In specific implementation, firstly, wavelet denoising is applied to all data in the first input dataset. The Daubechies wavelet basis can be selected to decompose the signal into multiple layers. Preferably, the Savitzky-Golay smoothing filter (the fitting order of the Savitzky-Golay smoothing filter can be set according to the signal smoothing requirements) can be used to smooth the spectral fingerprint, which helps to eliminate small fluctuations and retain characteristic peaks. Then, median filtering is used to remove isolated bright spot noise from all data in the second input dataset, and Gaussian filtering is combined for overall smoothing. Then, principal component analysis algorithm is used to remove feature components with low energy contribution from the first and second input datasets. Noise unrelated to defects is identified and eliminated by detector jitter and electromagnetic interference. Finally, the set of all data in the first and second input datasets after the above processing is used as the noise-suppressed multimodal dataset. Then, in the time dimension, the sampling frequency of the terahertz time-domain signal can be unified to the same reference value. The frame sequence of the electroluminescent image is interpolated according to the timestamp using an interpolation algorithm. In the spatial dimension, the spatial coordinates of the terahertz detection point can be mapped to the pixel coordinate system of the electroluminescent image according to the physical size of the photoelectric detection target. The two-dimensional grid of the terahertz signal is adjusted to a spatial scale consistent with the resolution of the electroluminescent image using a bilinear interpolation algorithm. Through the above steps, the spatiotemporal feature-aligned and fused multimodal features can be obtained. Z-score normalization is used to convert the spatiotemporal feature-aligned and fused multimodal features into multiple unit variances. Then, all unit variances are used as a spatial vector composed of the image space of the electroluminescent image as the multimodal feature matrix.

[0036] It should be noted that the multimodal feature matrix in this application is a high-dimensional data matrix formed by spatiotemporally fusing the time-frequency features of the terahertz signal with the spatial brightness and geometric features of the electroluminescent image. The multimodal feature matrix is ​​used as the input of the subsequent deep learning network, which is beneficial for defect detection, classification and evolution trend prediction. In addition, the spatiotemporal feature alignment in this embodiment is the process of mapping the time-frequency features of the terahertz signal and the spatial features of the electroluminescent image to the same spatiotemporal reference frame. This can make the first input dataset and the second input dataset correspond in time and space, which is beneficial for integrating features of different modalities into a unified matrix input, providing standardized, multi-information-source input data for the deep learning network, thereby improving the accuracy of defect identification and prediction.

[0037] In step S4, a multi-band deep fusion network based on a cross-domain feature interaction mechanism is used to perform feature monitoring and fusion on the multi-modal feature matrix to obtain a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, the defects of the photoelectric detection target are microscopically located and monitored to obtain the microscopic evolution trend of the defects of the photoelectric detection target.

[0038] It should be noted that the cross-domain feature interaction mechanism in this application refers to the interactive fusion of multimodal features in the time-frequency domain and the spatial domain; cross-domain feature interaction is a technical process of normalizing, aligning, analyzing correlations and fusing data from different modalities, which is beneficial for preserving key information of each modality and achieving information complementarity; in addition, the multi-band deep fusion network includes a one-dimensional convolutional neural network and a two-dimensional convolutional neural network. The one-dimensional convolutional neural network is used to process the time-domain signal generated by terahertz detection, and can preserve the details of time changes through local convolution and frequency filtering. The two-dimensional convolutional neural network is used to process image data and can capture the texture, edge and overall morphological features of defects.

[0039] In this embodiment, a multi-band deep fusion network based on a cross-domain feature interaction mechanism is used to perform feature monitoring and fusion on the multimodal feature matrix to obtain the detection and discrimination matrix and the fused feature vector. Specifically, this can be achieved in the following manner: The terahertz band signal in the multimodal feature matrix is ​​extracted using a time-frequency convolutional sub-network in a multi-band deep fusion network to obtain a terahertz feature map. The texture pattern and brightness distribution of the electroluminescence map in the multimodal feature matrix are encoded using the image convolutional coding subnetwork in the multi-band deep fusion network to obtain the defect space feature map; The terahertz feature map and the defect space feature map are subjected to cross-domain interaction to obtain a fused feature vector; The multimodal feature matrix is ​​reconstructed by weighting based on the fused feature vector to obtain the detection and discrimination matrix.

[0040] In specific implementation, firstly, a one-dimensional convolutional neural network in the multi-band deep fusion network is used as a time-frequency convolutional sub-network. Then, the multi-modal feature matrix is ​​input into the one-dimensional convolutional layer of the time-frequency convolutional sub-network. Multi-scale features of the terahertz band signal are extracted by point-by-point scanning, and the multi-scale features extracted by scanning are converted into terahertz feature maps by the time-frequency convolutional sub-network. Then, a two-dimensional convolutional neural network in the multi-band deep fusion network is used as an image convolutional coding sub-network. Then, the multi-modal feature matrix is ​​input into the two-dimensional convolutional layer of the image convolutional coding sub-network. The image convolutional coding sub-network enhances the nonlinear expression of the electroluminescence map through the ReLU activation function and gradually reduces the resolution through downsampling. This allows the electroluminescence map to extract features step by step from low-level texture to high-level semantics in multi-layer convolutional blocks, and outputs a defect space feature map. The defect space feature map contains defect brightness distribution, boundary contour, and geometric morphology information.

[0041] Preferably, the number of channels in the terahertz feature map and the defect space feature map can be uniformly mapped using a 1×1 convolution. Then, the global average pooling value of each channel is calculated using a cross-channel attention mechanism. Attention weight coefficients are generated through a fully connected layer and a sigmoid activation function. The terahertz feature map and the defect space feature map are then weighted and fused using the attention weight coefficients. Finally, a fused feature vector is extracted from the weighted fused map by element-wise addition. This fused feature vector contains the dielectric structure information and spatial geometric features of the photoelectric detection target. Then, the fused feature vector is input into the fully connected layer of a multi-band deep fusion network for feature compression and reconstruction to obtain fusion weights. The multimodal feature matrix is ​​then weighted based on the fusion weights using a Softmax classifier to obtain a detection discrimination matrix. In this matrix, the rows correspond to the spatial location of the photoelectric detection target, and the columns represent the probability distribution of different defect types (such as cracks, voids, impurities, or no defects).

[0042] It should be noted that the detection and discrimination matrix in this application is a two-dimensional probability matrix. The rows of the detection and discrimination matrix correspond to the spatial location of the photoelectric detection target, and the columns represent the probability distribution of different defect types, which is beneficial for refined defect spatial mapping. The time-frequency convolutional sub-network is a multi-scale convolutional structure designed for terahertz signals, which can simultaneously extract local time-domain waveform features and global spectral mode information. The image convolutional coding sub-network includes two-dimensional convolution and U-Net encoder, which can capture the brightness mode, texture details and defect contour information of electroluminescent images layer by layer to make up for the lack of spatial resolution of terahertz signals. The fused feature vector is a unified feature representation after cross-domain interaction between the terahertz feature map and the defect spatial feature map. It contains both dielectric structure information and spatial geometric features of the photoelectric detection target, which is beneficial for defect localization and type discrimination.

[0043] Preferably, in this embodiment, reference Figure 3 As shown, this figure is an exemplary flowchart for determining the microscopic evolution trend of defects in photoelectric detection targets according to the present application. In this embodiment, the defects of photoelectric detection targets are microscopically located and monitored based on the fused feature vector and the detection discrimination matrix. The microscopic evolution trend of the defects in photoelectric detection targets can be obtained by the following steps: In step S41, candidate defect regions of the photoelectric detection target are determined; In step S42, the defect candidate region is located at the sub-pixel level to obtain the microscopic spatial coordinates of the defect of the photoelectric detection target; In step S43, the defect candidate region is subjected to microscopic defect type discrimination through the fused feature vector to obtain the defect type result; In step S44, the evolution prediction of the defect type in the microscopic spatial coordinates is performed based on the detection discrimination matrix and the defect type result to obtain the microscopic evolution trend of the photoelectric detection target defect.

[0044] In specific implementation, firstly, a connected component labeling algorithm can be used to extract dark spot regions from the image of the photoelectric detection target, and the extracted dark spot regions are used as candidate regions. Then, the Canny algorithm is used to perform edge detection on the candidate region boundaries, and the detected boundary points are refined using a sub-pixel interpolation algorithm (such as the gray-scale centroid method). Then, the spatial center of the defect is extracted using the Canny operator, and this spatial center is used as the coordinate center to obtain the microscopic spatial coordinates of the defect of the photoelectric detection target. Secondly, local features of the defect mask are extracted from the fused feature vector, including the position of the terahertz spectrum absorption peak, the electroluminescence brightness distribution, and the geometric shape of the defect mask. The extracted local features are input into a trained classifier (such as a support vector machine) to output the defect type result, which includes: cracks, cavities, impurities, or non-defect regions. Finally, the multi-frame detection results of the detection discrimination matrix are used to track the spatial coordinate changes of the defect type of the photoelectric detection target, and the area change of the defect center coordinates is obtained. This area change can be visualized based on a visualization algorithm to obtain the microscopic evolution trend of the defect of the photoelectric detection target.

[0045] It should be noted that the microscopic evolution trend of the photoelectric detection target defect in this application refers to the change law of the photoelectric detection target defect at the microscopic scale over time. It can be used to judge the development pattern of the defect over time, which is beneficial to predict the failure risk of the photoelectric detection target and provide a basis for subsequent feedback optimization. In this application, sub-pixel level localization refers to further refining the defect boundary position within the pixel grid through image interpolation algorithm, thereby enabling accurate localization of the defect at the microscopic scale. In addition, in this application, the classifier trained by supervised learning based on the labeled defect sample set can output the defect type result based on the local features of the defect mask, and output the discrimination bias of the defect type result. The discrimination bias refers to the difference between the defect type predicted by the classifier and the actual labeled defect type, including the difference between the predicted defect area and the actual defect area and the classification error between the predicted defect type and the actual defect type label. It can be quantified by confidence or cross-entropy loss value and used for subsequent adaptive update of model weights to continuously optimize the defect type discrimination accuracy.

[0046] In step S5, the weight parameters of the multi-band deep fusion network are updated with negative feedback according to the micro-evolution trend, and then the defect results of the photoelectric detection target are output based on the updated multi-band deep fusion network.

[0047] In this embodiment, the weight parameters of the multi-band deep fusion network are updated with negative feedback based on the micro-evolution trend, and the defect results of the photoelectric detection target are output based on the updated multi-band deep fusion network. Specifically, this can be achieved in the following manner: The weight adjustment factor is determined based on the discrimination bias of the micro-evolution trend and defect type results; The weight adjustment factor is used as the weight value for adaptive negative feedback update, and then the multi-band deep fusion network is updated adaptively with negative feedback to obtain the updated multi-band deep fusion network. The updated multi-band deep fusion network is used to locate and identify defects in photoelectric detection targets, and the defect results of the photoelectric detection targets are obtained.

[0048] In specific implementation, firstly, the difference between the predicted defect area and the actual defect area, as well as the classification error between the predicted defect type and the actual defect type label, are normalized and weighted to obtain a weight adjustment factor. Normalization can be achieved using Z-score, and the weights in the weighted summation are the proportions of the normalized errors. Then, this weight adjustment factor is used as the weight value for adaptive negative feedback updates, dynamically adjusting the learning rate and weight update amplitude of the multi-band deep fusion network during backpropagation. This allows the multi-band deep fusion network to adaptively optimize based on deviations in defect evolution trends, resulting in an updated multi-band deep fusion network. Finally, the updated multi-band deep fusion network is used to locate and identify defects in the photoelectric detection target, regenerating the fused feature vector and detection discrimination matrix, and extracting the defect location coordinates, type label, and evolution trend prediction to obtain the defect result for the photoelectric detection target.

[0049] It should be noted that, in this embodiment, the adaptive negative feedback update of the multi-band deep fusion network refers to the dynamic adjustment of the learning rate, weight parameters, and weight allocation of the feature extraction layer of the multi-band deep fusion network based on the discrimination bias of the defect evolution trend and defect type results. This enables the multi-band deep fusion network to automatically correct parameters according to the detection error and gradually approach the true defect features. This helps to reduce the overfitting risk of the multi-band deep fusion network in different detection scenarios, thereby improving the accuracy of defect micro-localization and evolution prediction.

[0050] Furthermore, it should be noted that in this application, multi-band fusion includes feature fusion of terahertz detection signals across multiple frequency bands and multi-modal feature fusion of terahertz detection signals and electroluminescence images. Feature fusion across multiple frequency bands refers to fusing features of terahertz signals from multiple frequency bands within the same detection system to cover different detection depth and sensitivity requirements, thus avoiding detection blind spots in a single frequency band. Multi-modal fusion refers to interacting and weighting terahertz signal and electroluminescence image data in the same feature space to form composite features that combine physical depth information and surface spatial information. This application, through the combination of multi-band fusion and multi-modal fusion, can achieve deep defect detection and fine-grained localization of surface defects in photoelectric detection, improving the accuracy of photoelectric detection.

[0051] In summary, the technical solution adopted in this application can realize microscopic defect localization and evolution prediction based on multimodal fusion detection, thereby improving the accuracy of photoelectric detection.

[0052] Example 2: This application provides a target positioning photoelectric detection system based on multi-band fusion, referring to... Figure 4 As shown, this figure is a modular structure diagram of a target positioning photoelectric detection system based on multi-band fusion according to this application. The photoelectric detection system includes: The terahertz feature acquisition module 100 is used to acquire the time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies based on the terahertz quantum pulse source, thereby obtaining the first input dataset. The electroluminescence imaging module 200 is used to apply a dynamic bias current to the photoelectric detection target, thereby acquiring the electroluminescence image of the photoelectric detection target and extracting a second input dataset from the electroluminescence image; The multimodal feature preprocessing module 300 is used to perform feature fusion operation after jointly preprocessing the first input dataset and the second input dataset to obtain a multimodal feature matrix; The cross-domain deep fusion analysis module 400 is used to perform feature monitoring and fusion on the multimodal feature matrix using a multi-band deep fusion network based on a cross-domain feature interaction mechanism to obtain a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, the module performs microscopic positioning and monitoring of defects in the photoelectric detection target to obtain the microscopic evolution trend of the defects in the photoelectric detection target. The adaptive feedback optimization module 500 is used to perform negative feedback updates on the weight parameters of the multi-band deep fusion network according to the micro-evolution trend, and then output the defect results of the photoelectric detection target based on the updated multi-band deep fusion network.

[0053] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0054] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0055] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A target localization photoelectric detection method based on multi-band fusion, characterized in that, The photoelectric detection method includes the following steps: The time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies are obtained based on the terahertz quantum pulse source, and then the first input dataset is obtained. A dynamic bias current is applied to the photoelectric detection target, and then the electroluminescence image of the photoelectric detection target is acquired. The second input dataset is extracted from the electroluminescence image. After joint preprocessing of the first input dataset and the second input dataset, a feature fusion operation is performed to obtain a multimodal feature matrix; The multi-band deep fusion network based on cross-domain feature interaction mechanism is used to perform feature monitoring and fusion on the multi-modal feature matrix to obtain a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, the defects of the photoelectric detection target are microscopically located and monitored to obtain the microscopic evolution trend of the defects of the photoelectric detection target. The microscopic evolution trend refers to the change law of the defects of the photoelectric detection target at the microscopic scale over time. The weight parameters of the multi-band deep fusion network are updated with negative feedback based on the micro-evolution trend, and then the defect results of the photoelectric detection target are output based on the updated multi-band deep fusion network.

2. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, The spectral fingerprint is a set of characteristic spectral lines based on terahertz band signals, which includes absorption peaks, phase delays, and time-domain attenuation features.

3. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, A dynamic bias current is applied to the photoelectric detection target by an adjustable current source, which is adjusted based on the intensity of the detected electroluminescence signal.

4. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, Electroluminescence images of photoelectric detection targets are acquired using infrared imaging equipment.

5. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, Extracting the second input dataset from the electroluminescence image specifically includes: The electroluminescence image is subjected to noise reduction, brightness normalization, and background light interference suppression to obtain a defect mask; The second input dataset is formed by extracting brightness distribution features and geometric contour parameters from the defect mask using a deep learning segmentation network.

6. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, After jointly preprocessing the first input dataset and the second input dataset, a feature fusion operation is performed to obtain the multimodal feature matrix, specifically including: Based on the artifact removal algorithm, synchronous noise suppression is performed on all data in the first input dataset and all data in the second input dataset to obtain a noise-suppressed multimodal dataset; Spatiotemporal feature alignment and fusion are performed on the noise-suppressed multimodal dataset to obtain a multimodal feature matrix; The multimodal feature matrix is ​​a high-dimensional data matrix formed by spatiotemporally fusing the time-frequency features of the terahertz signal with the spatial brightness and geometric features of the electroluminescent image.

7. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, The multi-band deep fusion network based on a cross-domain feature interaction mechanism is used to perform feature detection and fusion on the multi-modal feature matrix to obtain a detection discrimination matrix and a fused feature vector, specifically including: The terahertz band signal in the multimodal feature matrix is ​​extracted using a time-frequency convolutional sub-network in a multi-band deep fusion network to obtain a terahertz feature map. The texture pattern and brightness distribution of the electroluminescence map in the multimodal feature matrix are encoded using the image convolutional coding subnetwork in the multi-band deep fusion network to obtain the defect space feature map; The terahertz feature map and the defect space feature map are subjected to cross-domain interaction to obtain a fused feature vector; The multimodal feature matrix is ​​reconstructed by weighting based on the fused feature vector to obtain the detection and discrimination matrix; The detection and discrimination matrix is ​​a two-dimensional probability matrix.

8. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, Based on the fused feature vector and the detection discrimination matrix, microscopic localization and monitoring of defects in photoelectric detection targets are performed to obtain the microscopic evolution trend of defects in photoelectric detection targets, specifically including: Identify candidate defect regions for photoelectric detection targets; The defect candidate region is located at the sub-pixel level to obtain the microscopic spatial coordinates of the defect in the photoelectric detection target; The defect candidate region is used to determine the micro-defect type by the fused feature vector, and the defect type result is obtained. Based on the detection discrimination matrix and the defect type results, the evolution prediction of the defect type in the microscopic spatial coordinates is performed to obtain the microscopic evolution trend of the photoelectric detection target defect.

9. The target positioning photoelectric detection method based on multi-band fusion as described in claim 1, characterized in that, The cross-domain feature interaction mechanism refers to the interaction and fusion of multimodal features in the time-frequency domain and the spatial domain.

10. A target positioning photoelectric detection system based on multi-band fusion, used to execute the target positioning photoelectric detection method based on multi-band fusion as described in any one of claims 1 to 9, characterized in that, The photoelectric detection system include: The terahertz feature acquisition module is used to acquire the time-domain reflection signal and spectral fingerprint of the photoelectric detection target at different frequencies based on the terahertz quantum pulse source, thereby obtaining the first input dataset. An electroluminescence imaging module is used to apply a dynamic bias current to a photoelectric detection target, thereby acquiring an electroluminescence image of the photoelectric detection target and extracting a second input dataset from the electroluminescence image; The multimodal feature preprocessing module is used to perform feature fusion operation after jointly preprocessing the first input dataset and the second input dataset to obtain a multimodal feature matrix; The cross-domain deep fusion analysis module is used to perform feature monitoring and fusion on the multimodal feature matrix using a multi-band deep fusion network based on a cross-domain feature interaction mechanism, to obtain a detection discrimination matrix and a fused feature vector. Based on the fused feature vector and the detection discrimination matrix, the module performs microscopic positioning and monitoring of defects in the photoelectric detection target, and obtains the microscopic evolution trend of the defects in the photoelectric detection target. An adaptive feedback optimization module is used to perform negative feedback updates on the weight parameters of the multi-band deep fusion network based on the micro-evolution trend, and then output the defect results of the photoelectric detection target based on the updated multi-band deep fusion network.

Images