Polaroid quality detection method based on multi-modal imaging and fusion deep learning

Abstract

The invention relates to the technical field of polaroid quality detection, in particular to a polaroid quality detection method based on multi-modal imaging and fusion deep learning, and the method comprises the steps: collecting optical, intermediate infrared polarization and ultrasonic corresponding three-modal original data of a polaroid; the three-mode original data is preprocessed, and the preprocessed three-mode data is obtained; respectively extracting local defect features and global distribution features of the preprocessed three-mode data through a double-branch feature extraction network; fusing the local defect feature and the global distribution feature by using a dynamic weight to obtain a fused feature; and outputting a quality detection result of the polaroid based on the fusion features, and performing closed-loop feedback control on the production of the polaroid according to the quality detection result and production line process parameters. According to the scheme, comprehensive and high-precision detection of various defects on the surface and in the polaroid can be achieved, and meanwhile the real-time requirement of online detection of a production line is met.

Description

Technical Field

[0001] This invention relates to the field of polarizer quality inspection technology, specifically a polarizer quality inspection method based on multimodal imaging and fusion deep learning. Background Technology

[0002] Polarizing films are a core component of various display devices such as LCDs and monitors. Their quality directly determines the clarity, contrast, and stability of the displayed image, playing an irreplaceable role in high-end fields such as consumer electronics, automotive displays, and medical displays. As display technology rapidly advances towards higher resolution, higher refresh rates, and ultra-thin designs, various defects generated during polarizing film production, such as micro-scratches, transparency defects, internal bubbles, and delamination, are increasingly impacting display performance. The market demands stringent requirements for polarizing film quality inspection, including comprehensive defect detection, high-precision identification, and real-time production line response. High-quality testing technology has become a core support for ensuring the reliability of display products.

[0003] Currently, polarizer quality inspection methods are mainly divided into two categories, neither of which can solve the core contradiction of balancing comprehensiveness, high accuracy, and real-time performance: one category is the traditional single-modal inspection method, which relies heavily on optical imaging and can only capture visible surface defects, failing to effectively identify internal structural defects such as bubbles and delamination, resulting in insufficient comprehensiveness; the other category is some multimodal inspection schemes, which, although combining multiple imaging technologies, use fixed weights or simple stitching fusion methods, failing to dynamically adjust the contribution of each modality according to the defect type, resulting in limited defect recognition accuracy. Furthermore, traditional deep learning models have a large number of parameters and complex calculations, with single-image detection time generally exceeding 100 milliseconds, making it difficult to meet the real-time requirements of online inspection on production lines.

[0004] Therefore, how to achieve comprehensive and high-precision detection of various defects on and inside the polarizer surface while meeting the real-time requirements of online inspection on the production line is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a polarizer quality detection method based on multimodal imaging and fusion deep learning to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The present invention provides a polarizer quality detection method based on multimodal imaging and fusion deep learning, comprising: Acquire raw data of the three modes corresponding to the optical, mid-infrared polarization, and ultrasound of the polarizer; The original trimodal data is preprocessed to obtain preprocessed trimodal data; The local defect features and global distribution features of the preprocessed trimodal data are extracted using a dual-branch feature extraction network. By using dynamic weights, the local defect features and the global distribution features are fused to obtain the fused features; Based on the fusion features, the quality inspection result of the polarizer is output, and the production of the polarizer is controlled in a closed loop according to the quality inspection result and the production line process parameters.

[0007] The dual-branch feature extraction network includes a main branch network and an auxiliary branch network. The process of extracting local defect features and global distribution features of the preprocessed trimodal data using the dual-branch feature extraction network includes: The main branch network extracts multi-scale basic features from the preprocessed trimodal data through a first-stage asymmetric depthwise separable convolution combination to obtain first-stage features. It then extracts local features from the first-stage features through a second-stage asymmetric depthwise separable convolution combination to obtain second-stage features. Finally, it extracts defect features from the second-stage features through a third-stage asymmetric depthwise separable convolution combination to obtain third-stage features. The third-stage features are then normalized and activated to obtain the local defect features. The auxiliary branch network divides the surface of the polarizer into multiple grid nodes, calculates the feature similarity score between each grid node, normalizes the feature similarity score to generate attention coefficients, updates the graph attention calculation method of the features of the grid nodes based on the attention coefficients, and outputs the global distribution features. The features of each grid node include local defect features and coordinate information output by the main branch network.

[0008] The method of using dynamic weights to fuse the local defect features and the global distribution features to obtain the fused features includes: The global attention map corresponding to the global distribution features is upsampled to the same size as the local defect features using bilinear interpolation to obtain the upsampled attention map. The local defect features are concatenated with the upsampled attention map in the channel dimension to obtain the concatenated features; The spliced ​​features are input into a gated network consisting of a 1×1 convolutional layer, an activation function, and global average pooling to generate dynamic weight coefficients with values ​​ranging from 0 to 1. These dynamic weight coefficients are used to adjust the fusion ratio of the local defect features and the global distribution features. The local defect features and the global distribution features are fused according to the dynamic weighting coefficients to obtain the fused features.

[0009] The three-modal raw data includes optical raw data, mid-infrared polarization raw data, and ultrasonic raw data. The preprocessing of the three-modal raw data to obtain preprocessed three-modal data includes: The raw optical data is divided into multiple sub-blocks. Histogram equalization is performed on each sub-block, and the contrast enhancement factor is limited to obtain contrast-enhanced sub-block images. All sub-block images are stitched together to form a complete enhanced image. Gaussian filtering with a 3×3 kernel size and a standard deviation of 1.0 is used to eliminate structured light stripe noise in the enhanced image to obtain a denoised intermediate optical image. The denoised intermediate optical image is binarized by adaptive threshold calculation to generate a surface defect candidate region map. The surface defect candidate region map is used as the preprocessed optical data. The intensity values ​​of the four polarization directions of the original mid-infrared polarization data are decoded to obtain intensity data of the four polarization directions. Stokes vectors are calculated based on the intensity data of the four polarization directions. The degree of polarization and polarization angle are calculated based on the Stokes vectors to obtain polarization feature data. The thermal noise of the polarization feature data is removed by wavelet transform with three-level decomposition to obtain denoised polarization feature data. The denoised polarization feature data is aligned with the preprocessed optical data by feature point registration algorithm to obtain preprocessed mid-infrared polarization data. The environmental noise of the original ultrasound data is removed by wavelet transform with four-level decomposition to obtain denoised original ultrasound data. The denoised original ultrasound data is reconstructed by delay compensation algorithm to generate a two-dimensional thickness distribution map to obtain ultrasound thickness distribution data. The ultrasound thickness distribution data is then adaptively segmented by automatic threshold calculation algorithm to extract abnormal internal structural regions, resulting in preprocessed ultrasound data.

[0010] The step of outputting the quality detection result of the polarizer based on the fusion features includes: The fused features are used as a graph structure, with pixels as nodes and adjacent relationships as edges. The global importance of each node is calculated through a graph attention mechanism to obtain the node importance weight. The fused features are then globally pooled based on the node importance weight to generate a global description vector. The global description vector is input into a convolutional classification layer to output a multi-class defect probability distribution, thereby obtaining the defect type and confidence level. The defect type and confidence level are used as the defect classification result. The fused features are input into a sub-pixel convolution module with 4x upsampling to predict the coordinates and confidence of the defect bounding box, thus obtaining the original localization data containing coordinates and confidence. The duplicate predictions of the original localization data are removed by non-maximum suppression processing with preset confidence threshold and overlap threshold, and the coordinates, area, and aspect ratio of the defect are output to obtain the defect localization result. Based on the defect classification results and the defect location results, the quality inspection results of the polarizer are obtained, including the defect type, severity, and spatial distribution.

[0011] Before outputting the quality detection result of the polarizer based on the fusion feature, the method further includes: For the main branch network in the dual-branch feature extraction network, an optimization algorithm is used to reduce the number of multiply-accumulate operations of 3×3 convolution from 9 to 4, resulting in optimized convolution operation logic. Based on the optimized convolution operation logic, the same set of computing units is reused through multi-channel convolution operations, thereby increasing the utilization rate of computing units to above the preset utilization rate and reducing the computation time of local feature extraction process. For the graph attention calculation of the auxiliary branch network in the dual-branch feature extraction network, a parallel computing architecture is adopted. The features of the grid nodes are stored in the cache unit to obtain the cached node features. All attention scores of the cached node features are processed simultaneously by the parallel computing unit array. Each computing unit processes the attention score calculation of one node. Multiple grid nodes are processed in parallel through pipeline design to shorten the latency of global distributed feature extraction.

[0012] After outputting the quality detection result of the polarizer based on the fusion features, the method further includes: Based on the quality inspection results, statistical analysis is performed. When a new defect type is detected in three consecutive batches, or the defect classification accuracy rate is lower than the preset accuracy threshold for five consecutive periods, a pre-trained teacher model is obtained, and the current model used for feature extraction and fusion is used as the student model. The student model is optimized through the strong feature expression capability of the teacher model. Collect 5-10 new defect samples and label the defect type and location to obtain labeled new defect samples; The labeled new defect samples are input into the teacher model for reasoning to generate probability distributions of 10 types of defects as soft labels; The labeled new defect sample is input into the student model for reasoning to obtain the student model reasoning result; Calculate the classification cross-entropy loss between the student model inference result and the sample true label, and the difference loss between the student model inference result and the soft label, and calculate the comprehensive loss based on the classification cross-entropy loss and the difference loss; Based on the comprehensive loss, update the deep convolutional layer parameters of the main branch network of the dual-branch feature extraction network; By configuring an overload method, the updated deep convolutional layer parameters of the main branch network are written into the student model, thereby achieving online updating of model parameters.

[0013] The step of performing closed-loop feedback control on the production of the polarizer based on the quality inspection results and production line process parameters includes: A static rule base is obtained by pre-setting an association matrix between 10 types of defects and 5 types of process parameters. The association matrix includes defect type, main control parameter, adjustment direction, step size, and safety boundary. By using online reinforcement learning, the rule weights of the static rule base are adjusted based on the changes in the defect rate according to the quality inspection results, and an updated rule base for mapping defects to parameters is obtained. The quality inspection results are encapsulated into a fixed 64-byte message and encoded in binary format to obtain the encoded inspection data. The encoded detection data is transmitted to the production line control system via high-speed Ethernet, with the control communication link delay within 1 microsecond and the total end-to-end delay less than 5 milliseconds.

[0014] The process of performing closed-loop feedback control on the production of the polarizer based on the quality inspection results and production line process parameters further includes: Each polarizer is assigned a unique identifier, and the quality inspection results are linked to the process parameters of the corresponding production stage and stored in the database to establish a defect lifecycle traceability file. Based on the quality inspection results stored in the database, the defect rate of three consecutive batches of polarizers is statistically analyzed. When the defect rate exceeds the preset defect rate threshold, the historical process parameters of the corresponding production cycle are replayed through digital twin simulation to locate the process link that caused the defect and obtain the root cause analysis results. The root cause analysis results are used as new samples to input into the dynamic model predictive controller. The parameters of the dynamic model predictive controller are updated through incremental knowledge distillation to obtain an optimized dynamic model predictive controller. The optimized dynamic model predictive controller is used for closed-loop feedback control to improve the controller's control accuracy of process links.

[0015] The acquired raw data of the three modes corresponding to optical, mid-infrared polarization, and ultrasound of the polarizer includes: A 352MHz global clock signal is generated, and three types of trigger signals are generated based on the global clock signal. The three types of trigger signals include a level signal that controls the exposure timing of optical data acquisition and the 2-microsecond structured light pulse width, a pulse width modulation signal that controls the polarization direction switching of mid-infrared polarization data acquisition, and an independent trigger signal that controls the emission timing of ultrasonic data acquisition. Based on the level signal, the optical acquisition device is controlled to acquire an optical surface image of the polarizer under structured light illumination, and the raw optical data is obtained. Based on the pulse width modulation signal, the mid-infrared polarization acquisition device is controlled to acquire mid-infrared thermal images in four polarization directions to obtain raw mid-infrared polarization data. Based on the independent trigger signal, the ultrasonic acquisition device is controlled to acquire ultrasonic scanning images and obtain raw ultrasonic data. The raw optical data, mid-infrared polarization data, and ultrasonic data are cached in a temporary storage unit. The data in the temporary storage unit are time-calibrated using a timestamp alignment algorithm to keep the acquisition time difference within 50 microseconds.

[0016] The beneficial effects of this invention are as follows: This invention provides a polarizer quality inspection method based on multimodal imaging and fusion deep learning. It utilizes complementary acquisition through optical, mid-infrared polarization, and ultrasonic three-modal imaging. The optical modality captures visible surface defects, the mid-infrared polarization modality mines internal material characteristics and transparent defects, and the ultrasonic modality detects internal structural defects such as bubbles and layering, achieving comprehensive defect coverage and precise data alignment. Targeted preprocessing of the three-modal raw data effectively purifies the data and enhances defect features, laying the foundation for high-precision detection. The dual-branch feature extraction network employs an asymmetric depthwise separable convolution main branch to extract multi-scale local defect features, while a graph attention network auxiliary branch models global distribution correlations. The synergy between local and global features improves recognition accuracy, while asymmetric convolution and input pruning techniques significantly reduce computational load. The dynamic weight fusion module adaptively adjusts the feature fusion ratio through a lightweight gating network, focusing on effective information to improve detection accuracy. Combined with a lightweight classification and positioning head, the detection time for a single image is shortened, meeting the online real-time detection requirements of production lines, ultimately achieving the dual goals of high precision and real-time performance. Attached Figure Description

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of a polarizer quality detection method based on multimodal imaging and fusion deep learning in one embodiment of this application; Figure 2 This is a structural diagram of a polarizer quality detection system based on multimodal imaging and fusion deep learning in one embodiment of this application. Detailed Implementation

[0018] Figure 1 This is a flowchart of a polarizer quality detection method based on multimodal imaging and deep learning fusion according to an embodiment of this application, as shown below. Figure 1 As shown, the polarizer quality detection method based on multimodal imaging and fusion deep learning in this application includes: S110 acquires raw data of the three modes corresponding to the optical, mid-infrared polarization, and ultrasound of the polarizer; S120, preprocess the original trimodal data to obtain preprocessed trimodal data; S130, using a dual-branch feature extraction network, extract the local defect features and global distribution features of the preprocessed three-modal data respectively; S140, using dynamic weights, the local defect features and the global distribution features are fused to obtain the fused features; S150, based on the fusion features, output the quality inspection result of the polarizer, and perform closed-loop feedback control on the production of the polarizer according to the quality inspection result and the production line process parameters.

[0019] Trimodal imaging refers to the integration of an optical camera, a mid-infrared polarization camera, and an ultrasonic probe into a hardware combination, representing a typical hardware architecture in the field of multimodal detection. The optical camera captures visible surface defects, the mid-infrared polarization camera extracts internal material information through polarization characteristics, and the ultrasonic probe detects internal structural defects. These three components complement each other to achieve comprehensive detection.

[0020] Three-modal data preprocessing is a targeted purification and enhancement process for three types of raw data: optical, mid-infrared polarization, and ultrasound. It is a key step in optimizing the input of deep learning models. Different algorithms are used for each modality, such as adaptive histogram equalization enhancement and Gaussian filtering for optical data, Stokes vector decoding and polarization feature extraction for mid-infrared polarization data, and scan reconstruction and threshold segmentation for ultrasound data. Finally, a standardized data format that can be used for feature extraction is generated.

[0021] The dual-branch feature extraction network features a parallel network architecture with a main branch and an auxiliary branch, representing an innovative design in deep learning feature extraction. The main branch focuses on local defect details, while the auxiliary branch models global distribution correlations. By collaboratively extracting local and global features, it addresses the problem of insufficient representation by a single feature dimension, thereby improving the comprehensiveness of defect recognition.

[0022] By adaptively adjusting the feature fusion weights through a learnable gating network, unlike traditional fixed-weight fusion, dynamic weight coefficients in the [0,1] interval are generated. The contribution ratio of local and global features is flexibly allocated according to the defect type, thereby achieving efficient integration of feature information.

[0023] A lightweight classification and localization module replaces the traditional fully connected layer's efficient output module. The classification head uses global graph attention pooling, and the localization head uses sub-pixel convolution, significantly reducing computational load while maintaining accuracy. The classification head outputs the probability of 10 types of defects, and the localization head outputs defect coordinates with an accuracy of 0.1 pixels, providing core data for quality inspection results.

[0024] Closed-loop feedback control establishes a control mechanism that links detection results with production line parameters. Through a hybrid strategy of static rule engine and dynamic model predictive controller, defect information is transformed into process adjustment instructions, realizing the transformation of quality control from passive identification to proactive optimization.

[0025] In some embodiments, the dual-branch feature extraction network includes a main branch network and an auxiliary branch network. The step of extracting local defect features and global distribution features of the preprocessed trimodal data through the dual-branch feature extraction network includes: The main branch network extracts multi-scale basic features from the preprocessed trimodal data through a first-stage asymmetric depthwise separable convolution combination to obtain first-stage features. It then extracts local features from the first-stage features through a second-stage asymmetric depthwise separable convolution combination to obtain second-stage features. Finally, it extracts defect features from the second-stage features through a third-stage asymmetric depthwise separable convolution combination to obtain third-stage features. The third-stage features are then normalized and activated to obtain the local defect features. The auxiliary branch network divides the surface of the polarizer into multiple grid nodes, calculates the feature similarity score between each grid node, normalizes the feature similarity score to generate attention coefficients, updates the graph attention calculation method of the features of the grid nodes based on the attention coefficients, and outputs the global distribution features. The features of each grid node include local defect features and coordinate information output by the main branch network.

[0026] The main branch network is the core branch in the dual-branch network responsible for extracting local defect features. It adopts a three-stage asymmetric depthwise separable convolution structure. Asymmetric depthwise separable convolution refers to splitting the standard convolution into asymmetric convolution combinations such as 1×3+3×1 and 3×1+1×3. It belongs to lightweight convolution technology, which reduces the number of parameters and computational cost while maintaining feature extraction capability. The three-stage output channels are 64, 128, and 256, respectively, to progressively extract multi-scale local features.

[0027] Normalization and activation are optimization operations performed after convolutional layers. Normalization uses batch normalization layers to accelerate model training convergence and alleviate gradient vanishing. Activation uses the ReLU6 activation function to limit the output range, avoid gradient explosion, and enhance model stability. It is a common optimization technique in deep learning networks.

[0028] The auxiliary branch network is the branch in the dual-branch network responsible for extracting global distribution features, and its core is the graph attention network. The graph attention network is a deep learning model based on a graph structure. It models the relationship between nodes by calculating the attention scores between nodes. First, the polarizer surface is divided into 16×16 grid nodes. Each node integrates local features and coordinate information, and then the node features are updated through an attention mechanism to achieve global distribution modeling.

[0029] Feature similarity score and attention coefficient are the core parameters of graph attention calculation. Feature similarity score is obtained by calculating the inner product of node feature maps using a leaky linear rectified activation function, and is used to measure the strength of association between nodes. Attention coefficient is the result after normalizing the similarity score to ensure that the sum of weights is 1, thereby achieving focus on important associated nodes.

[0030] Input pruning is an optimization technique that uses the lasso regression algorithm to remove redundant grid nodes, belonging to network lightweighting techniques. Its function is to eliminate nodes that contribute minimally to global features, reducing computational load and memory consumption, while maintaining the integrity of global feature distribution and improving inference speed.

[0031] In this embodiment, by optimizing the specific structure of the dual-branch feature extraction network, the asymmetric depthwise separable convolution of the main branch significantly reduces the number of model parameters. The graph attention network and input pruning technique of the auxiliary branch control the computational overhead while modeling global correlations. The two work together to achieve efficient extraction of local defect details and global distribution features. This not only solves the problem of large number of parameters and computational complexity of traditional models, but also improves the expressive power of defect features, providing a high-quality feature foundation for subsequent fusion processing and helping to improve both detection accuracy and real-time performance.

[0032] In some implementations, the step of using dynamic weights to fuse the local defect features and the global distribution features to obtain fused features includes: The global attention map corresponding to the global distribution features is upsampled to the same size as the local defect features using bilinear interpolation to obtain the upsampled attention map. The local defect features are concatenated with the upsampled attention map in the channel dimension to obtain the concatenated features; The spliced ​​features are input into a gated network consisting of a 1×1 convolutional layer, an activation function, and global average pooling to generate dynamic weight coefficients with values ​​ranging from 0 to 1. These dynamic weight coefficients are used to adjust the fusion ratio of the local defect features and the global distribution features. The local defect features and the global distribution features are fused according to the dynamic weighting coefficients to obtain the fused features.

[0033] Bilinear interpolation upsampling is an image processing technique that enlarges the global attention map to the size of local defect features. It belongs to the interpolation scaling method. It calculates the interpolated pixel value by weighted averaging of adjacent pixels to ensure image smoothness. It is used to solve the problem of size mismatch between the global attention map and the local feature map and to ensure the feasibility of feature stitching.

[0034] Channel-dimensional concatenation is an operation that merges the local defect feature tensor with the upsampled attention map along the channel dimension, and it belongs to the feature fusion step. The number of channels of the concatenated feature is the sum of the number of channels of the two, realizing the initial integration of local features and global attention information, and providing complete input for dynamic weight calculation.

[0035] The gated network is a lightweight network module consisting of 1×1 convolutional layers, activation functions, and global average pooling. Its core function is to generate dynamic weight coefficients. The 1×1 convolutional layers are used for dimensionality compression and feature transformation, the ReLU6 activation function introduces non-linearity, and global average pooling outputs single-channel weight coefficients. The total number of parameters in the network is less than 1000, ensuring its lightweight characteristics.

[0036] The dynamic weight coefficient is a learnable parameter ranging from 0 to 1, used to adaptively adjust the fusion ratio of local defect features and global distribution features. Its value is dynamically generated by the gating network based on the input features, and the coefficient value varies depending on the defect type, achieving adaptive matching between defect type and fusion weight.

[0037] In this embodiment, a four-step process of dynamic weight fusion solves the problem of low information utilization caused by fixed weights or simple splicing in existing multimodal fusion. Bilinear interpolation upsampling ensures the consistency of feature size, channel splicing achieves preliminary information integration, the dynamic weight coefficients generated by the gating network can adaptively adjust the fusion ratio according to the defect type, and the final fusion formula achieves precise complementarity between local and global features. This not only improves the robustness and detection accuracy of defect recognition, but also ensures the real-time performance of the overall detection because the lightweight design of the gating network does not add extra computational burden.

[0038] In some embodiments, the trimodal raw data includes optical raw data, mid-infrared polarization raw data, and ultrasonic raw data. The preprocessing of the trimodal raw data to obtain preprocessed trimodal data includes: The raw optical data is divided into multiple sub-blocks. Histogram equalization is performed on each sub-block, and the contrast enhancement factor is limited to obtain contrast-enhanced sub-block images. All sub-block images are stitched together to form a complete enhanced image. Gaussian filtering with a 3×3 kernel size and a standard deviation of 1.0 is used to eliminate structured light stripe noise in the enhanced image to obtain a denoised intermediate optical image. The denoised intermediate optical image is binarized by adaptive threshold calculation to generate a surface defect candidate region map. The surface defect candidate region map is used as the preprocessed optical data. The intensity values ​​of the four polarization directions of the original mid-infrared polarization data are decoded to obtain intensity data of the four polarization directions. Stokes vectors are calculated based on the intensity data of the four polarization directions. The degree of polarization and polarization angle are calculated based on the Stokes vectors to obtain polarization feature data. The thermal noise of the polarization feature data is removed by wavelet transform with three-level decomposition to obtain denoised polarization feature data. The denoised polarization feature data is aligned with the preprocessed optical data by feature point registration algorithm to obtain preprocessed mid-infrared polarization data. The environmental noise of the original ultrasound data is removed by wavelet transform with four-level decomposition to obtain denoised original ultrasound data. The denoised original ultrasound data is reconstructed by delay compensation algorithm to generate a two-dimensional thickness distribution map to obtain ultrasound thickness distribution data. The ultrasound thickness distribution data is then adaptively segmented by automatic threshold calculation algorithm to extract abnormal internal structural regions, resulting in preprocessed ultrasound data.

[0039] The optical data preprocessing process involves a series of optimization operations on the raw optical data, with the core objective of enhancing surface defect contrast and removing noise. Specifically, the adaptive histogram equalization algorithm avoids noise amplification caused by overall equalization through sub-block histogram equalization, a 3×3 kernel Gaussian filter with a standard deviation of 1.0 specifically eliminates structured light stripe noise, and adaptive threshold binarization is used to filter candidate regions for surface defects, ultimately generating standardized data focusing on surface defects.

[0040] The mid-infrared polarization data preprocessing workflow revolves around the steps of polarization feature extraction and optimization. Stokes vector decoding is the process of converting the intensity values ​​of the four polarization directions (0°, 45°, 90°, 135°) into S0-S3 vectors, which is the basis for polarization feature calculation. The degree of polarization and the polarization angle are obtained through vector operations and are used to characterize the internal properties of the material. The three-layer decomposition of the Dobessi wavelet transform is specifically used to remove thermal noise, and the feature point registration algorithm ensures spatial alignment with the optical data.

[0041] The ultrasound data preprocessing process is a defect region extraction process for the raw ultrasound signal. The four-layer decomposition of the Dobesie wavelet transform is used to remove environmental noise, the delay compensation algorithm realizes the reconstruction of the scanned image and generates a two-dimensional thickness distribution map, and the Otsu algorithm automatically calculates the threshold for adaptive segmentation to accurately extract abnormal areas of internal structure (such as bubbles and delamination).

[0042] The adaptive histogram equalization algorithm avoids the problem of excessive noise amplification in traditional histogram equalization by limiting the contrast enhancement factor, and is suitable for contrast enhancement of transparent defects on the surface of polarizers.

[0043] The Stokes vector is a physical quantity that describes the polarization state of light. It contains four parameters: S0 (total light intensity), S1 (difference between horizontal and vertical polarization components), S2 (difference between 45° and 135° polarization components), and S3 (difference between circular polarization components). It is the core basis for calculating the degree of polarization and the polarization angle.

[0044] In this embodiment, a differentiated preprocessing workflow was designed based on the characteristics of the three-modal raw data. Optical data preprocessing enhanced contrast and denoised the data, highlighting visible and transparent surface defects. Mid-infrared polarization data preprocessing extracted polarization features and denoised the data to uncover internal material properties. Ultrasonic data preprocessing accurately located internal structural defects through reconstruction and threshold segmentation. The entire preprocessing workflow effectively purified the raw data, enhanced defect features, and achieved spatial alignment of multimodal data, providing high-quality input for subsequent bi-branch feature extraction and ensuring detection accuracy from the source. Simultaneously, the standardized processing workflow improved data consistency and the model's generalization ability.

[0045] In some implementations, outputting the quality inspection result of the polarizer based on the fusion features includes: The fused features are used as a graph structure, with pixels as nodes and adjacent relationships as edges. The global importance of each node is calculated through a graph attention mechanism to obtain the node importance weight. The fused features are then globally pooled based on the node importance weight to generate a global description vector. The global description vector is input into a convolutional classification layer to output a multi-class defect probability distribution, thereby obtaining the defect type and confidence level. The defect type and confidence level are used as the defect classification result. The fused features are input into a sub-pixel convolution module with 4x upsampling to predict the coordinates and confidence of the defect bounding box, thus obtaining the original localization data containing coordinates and confidence. The duplicate predictions of the original localization data are removed by non-maximum suppression processing with preset confidence threshold and overlap threshold, and the coordinates, area, and aspect ratio of the defect are output to obtain the defect localization result. Based on the defect classification results and the defect location results, the quality inspection results of the polarizer are obtained, including the defect type, severity, and spatial distribution.

[0046] Global graph attention pooling treats the fused feature map as a graph structure, using pixels as nodes and adjacency relationships as edges. It calculates the global importance weight of each node through a graph attention mechanism, and then performs global pooling to generate a global descriptive vector. This method replaces traditional fully connected layers, reducing the number of parameters while improving the expressive power of global features, making it suitable for accurate classification of defect types.

[0047] The convolutional classification layer is a 1×1 convolutional layer that takes a global description vector as input and outputs a probability distribution of multiple defect types. It has 10 output channels, corresponding to 9 types of defects and 1 type of background. The probability distribution is output through the Softmax activation function to achieve defect type classification and recognition.

[0048] The subpixel convolution module is a convolution module that uses a 4x upsampling factor. Its core principle is to map the pixels in the low-resolution feature map to the corresponding positions in the high-resolution grid, and improve the coordinate prediction accuracy through interpolation calculation. It is used for accurate prediction of the coordinates of defect bounding boxes, with an accuracy of up to 0.1 pixels.

[0049] Non-maximum suppression is a post-processing algorithm that removes duplicate predicted bounding boxes. By pre-setting a confidence threshold (0.5) and an overlap threshold (0.3), it retains the optimal defect box with the highest confidence and no duplication, thus avoiding the same defect being detected multiple times and improving the accuracy of the localization results.

[0050] The quality inspection results are a comprehensive inspection output that includes defect type (determined by classification results), severity (judged by defect area, confidence level, etc.) and spatial distribution (characterized by coordinates, aspect ratio, etc.), providing comprehensive defect information support for production line parameter feedback.

[0051] In this embodiment, a lightweight classification and localization head design is employed. Global graph attention pooling replaces fully connected layers, reducing the number of model parameters and the risk of overfitting. A 1×1 convolutional classification layer achieves efficient and accurate classification of defect types. Subpixel convolutional modules significantly improve the localization accuracy of defect bounding boxes, eliminating redundant predictions and ensuring the accuracy of the localization results. This approach achieves comprehensive output of defect type, severity, and spatial distribution, with a classification accuracy exceeding 99.6% and a localization accuracy of 0.1 pixels. It meets the requirements of high-precision detection while ensuring real-time performance due to its lightweight design. Furthermore, the detailed spatial distribution features output provide crucial data support for subsequent closed-loop feedback control of the production line.

[0052] In some implementations, before outputting the quality detection result of the polarizer based on the fusion feature, the method further includes: For the main branch network in the dual-branch feature extraction network, an optimization algorithm is used to reduce the number of multiply-accumulate operations of 3×3 convolution from 9 to 4, resulting in optimized convolution operation logic. Based on the optimized convolution operation logic, the same set of computing units is reused through multi-channel convolution operations, thereby increasing the utilization rate of computing units to above the preset utilization rate and reducing the computation time of local feature extraction process. For the graph attention calculation of the auxiliary branch network in the dual-branch feature extraction network, a parallel computing architecture is adopted. The features of the grid nodes are stored in the cache unit to obtain the cached node features. All attention scores of the cached node features are processed simultaneously by the parallel computing unit array. Each computing unit processes the attention score calculation of one node. Multiple grid nodes are processed in parallel through pipeline design to shorten the latency of global distributed feature extraction.

[0053] Among them, the number of multiplication and addition operations of 3×3 convolution is reduced from 9 to 4 through mathematical transformation, which greatly reduces the amount of computation without losing accuracy, and is used to accelerate the asymmetric convolution of the main branch.

[0054] By multiplexing multiple multiply-accumulate operations in the same group of digital signal processing units and rationally planning the operation timing (multiplexing cycle of 4 clock cycles), the utilization rate of digital signal processing units can be increased to over 90%, reducing hardware resource consumption and increasing computing throughput.

[0055] The features of 16×16 grid nodes are stored in a field-programmable gate array (FPGA) block random access memory. The attention scores of all nodes are processed simultaneously by a parallel computing unit array, thereby achieving parallel acceleration of graph attention computation and reducing the latency of the graph attention network layer to 14.8 microseconds.

[0056] An optimization strategy for attention calculation in auxiliary branch graphs is proposed. Each digital signal processing unit is responsible for calculating the attention score of one node. Multiple nodes are processed in parallel through pipeline design, which reduces the latency caused by serial calculation and improves the efficiency of global distributed feature extraction.

[0057] In this embodiment, a targeted field-programmable gate array (FPGA) acceleration optimization scheme significantly reduces the amount of convolution operations in the main branch's Vinogradad algorithm, while digital signal processing time multiplexing technology improves hardware resource utilization. The distributed-aggregate architecture and parallel computing optimization of the auxiliary branch enable efficient parallel processing of the graph attention network, keeping the layer latency of the graph attention network within 15 microseconds. This effectively solves the problems of computational complexity and high inference latency in dual-branch feature extraction networks, significantly reducing the time consumption of local feature extraction and global feature modeling. Combined with the overall lightweight design, the detection time for a single image is shortened to within 50 milliseconds, fully meeting the real-time requirements of online inspection on the production line.

[0058] In some implementations, after outputting the quality detection result of the polarizer based on the fusion features, the method further includes: Based on the quality inspection results, statistical analysis is performed. When a new defect type is detected in three consecutive batches, or the defect classification accuracy rate is lower than the preset accuracy threshold for five consecutive periods, a pre-trained teacher model is obtained, and the current model used for feature extraction and fusion is used as the student model. The student model is optimized through the strong feature expression capability of the teacher model. Collect 5-10 new defect samples and label the defect type and location to obtain labeled new defect samples; The labeled new defect samples are input into the teacher model for reasoning to generate probability distributions of 10 types of defects as soft labels; The labeled new defect sample is input into the student model for reasoning to obtain the student model reasoning result; Calculate the classification cross-entropy loss between the student model inference result and the sample true label, and the difference loss between the student model inference result and the soft label, and calculate the comprehensive loss based on the classification cross-entropy loss and the difference loss; Based on the comprehensive loss, update the deep convolutional layer parameters of the main branch network of the dual-branch feature extraction network; By configuring an overload method, the updated deep convolutional layer parameters of the main branch network are written into the student model, thereby achieving online updating of model parameters.

[0059] The criteria for triggering the online learning process include two situations: detecting new defect types in three consecutive batches and having a defect classification accuracy rate of less than 99% for five consecutive cycles. The system automatically triggers updates by monitoring the detection results in real time to ensure that the model can adapt to new scenarios in a timely manner.

[0060] The teacher-student model architecture is the core architecture of knowledge distillation. The teacher model is a pre-trained large network (such as residual network-50) with strong feature representation capabilities, used to generate high-quality soft labels. The student model is the lightweight model of this application, which optimizes its own parameters by learning the soft labels of the teacher model, thereby improving performance while maintaining lightweight design.

[0061] The loss function is a combined loss function consisting of classification cross-entropy loss and divergence loss. Classification cross-entropy loss measures the difference between the student model and the true label, while divergence loss measures the difference between the output distribution of the student model and the teacher model. The combination of the two ensures that the model can both classify accurately and learn the generalization ability of the teacher model.

[0062] The model update method that only updates the parameters of the deep convolutional layer in the main branch does not require retraining the entire dataset. Fine-tuning can be completed with only 5-10 new defect samples, which greatly reduces update time and data requirements.

[0063] The deployment method of online model parameter updates by configuring heavy loads of field-programmable gate arrays can complete model iteration without downtime, avoiding production line interruptions and improving system availability and continuity.

[0064] In this embodiment, the online adaptive learning mechanism ensures the model can respond promptly to new defects and accuracy degradation through triggering conditions. The teacher-student model architecture and comprehensive loss function enable efficient knowledge distillation with small sample sizes, requiring only 5-10 new samples to complete model optimization. Incremental learning strategies and field-programmable gate array (FPGA) hot-update technology avoid full-data retraining and production line downtime, significantly reducing maintenance costs. This effectively solves the problems of poor generalization ability and difficulty in online updates found in traditional models, enabling continuous optimization during production line operation, adapting to new defect types, and maintaining a detection accuracy of over 99.6% over the long term. This improves the system's adaptability and industrial applicability. In some embodiments, the step of performing closed-loop feedback control on the production of the polarizer based on the quality inspection results and production line process parameters includes: A static rule base is obtained by pre-setting an association matrix between 10 types of defects and 5 types of process parameters. The association matrix includes defect type, main control parameter, adjustment direction, step size, and safety boundary. By using online reinforcement learning, the rule weights of the static rule base are adjusted based on the changes in the defect rate according to the quality inspection results, and an updated rule base for mapping defects to parameters is obtained. The quality inspection results are encapsulated into a fixed 64-byte message and encoded in binary format to obtain the encoded inspection data. The encoded detection data is transmitted to the production line control system via high-speed Ethernet, with the control communication link delay within 1 microsecond and the total end-to-end delay less than 5 milliseconds.

[0065] Among them, the rule set of the association matrix between 10 types of defects and 5 types of process parameters (tension, temperature, pressure, speed, coating thickness) includes information such as defect type, main control parameter, adjustment direction, step size, and safety boundary. It is used to quickly respond to common defects and is the basic rule support for closed-loop control.

[0066] Based on the field-programmable gate array (FPGA) hard core user datagram protocol stack, and implemented using a hardware description language, the defect results are encapsulated into a 64-byte fixed message (JSON binary encoding). It connects to the 88E1111 network card through a gigabit media independent interface, with an Ethernet transmission rate of 1 gigabit per second. Combined with the time-sensitive network switch quality of service priority flag, the communication link latency is controlled to be <1 microsecond and the total end-to-end latency is <5 milliseconds.

[0067] Combining a static rule engine and a dynamic model prediction controller, the static rule engine directly queries the rule base based on the defect type to execute preset adjustments, quickly responding to common defects; the dynamic model prediction controller takes into account the defect spatial distribution heat map, current process parameters, and historical adjustment records, and aims to minimize the defect rate × weight + parameter fluctuation penalty term, generating a parameter adjustment sequence through rolling time-domain optimization, which is suitable for complex process parameter optimization.

[0068] Safety boundaries are preset limits on the adjustment range of process parameters. For example, the safety boundary for adjusting the tension of the coating machine for scratch defects is ±10%, which avoids excessive parameter adjustment that could lead to equipment failure and ensures the safety of the production process.

[0069] In this embodiment, through the design of a method to connect defect detection results with production line parameters, a static rule base enables rapid response to common defects, a dynamic model predictive controller solves the problem of collaborative optimization of complex process parameters, and a hybrid control strategy balances response speed and control accuracy. The real-time communication architecture controls end-to-end latency to within 5 milliseconds, meeting the real-time control requirements of the production line. A safety boundary design ensures the safety of parameter adjustments. This effectively solves the problems of missing linkage between detection results and production line parameters and passive quality control, enabling rapid conversion of detection results into process adjustments, proactively optimizing production processes, reducing defect rates, and improving the intelligence level of quality control.

[0070] In some embodiments, after performing closed-loop feedback control on the production of the polarizer based on the quality inspection results and production line process parameters, the method further includes: Each polarizer is assigned a unique identifier, and the quality inspection results are linked to the process parameters of the corresponding production stage and stored in the database to establish a defect lifecycle traceability file. Based on the quality inspection results stored in the database, the defect rate of three consecutive batches of polarizers is statistically analyzed. When the defect rate exceeds the preset defect rate threshold, the historical process parameters of the corresponding production cycle are replayed through digital twin simulation to locate the process link that caused the defect and obtain the root cause analysis results. The root cause analysis results are used as new samples to input into the dynamic model predictive controller. The parameters of the dynamic model predictive controller are updated through incremental knowledge distillation to obtain an optimized dynamic model predictive controller. The optimized dynamic model predictive controller is used for closed-loop feedback control to improve the controller's control accuracy of process links.

[0071] The traceability scheme assigns a unique RFID tag to each polarizer, links the test results with the corresponding process parameters of the production process, and stores them in a database to establish a full lifecycle traceability file for defects. This supports tracing back from finished product defects to key parameters in the production process. When the defect rate exceeds a preset threshold for three consecutive batches, a digital twin simulation analysis process is triggered. By replaying historical process parameters of the corresponding production cycle, the production process is simulated to locate the key process steps leading to defects, achieving reverse analysis of defect causes. The root cause analysis results are used as new samples to input into the dynamic model predictive controller. Through incremental knowledge distillation, the model predictive controller's parameters are updated, enabling the controller to continuously learn the correlation between process adjustments and defect rates, improving control accuracy. The virtual simulation model built based on production process data can accurately replay the production state corresponding to historical process parameters. This is an advanced technology for locating process root causes and simulating the effects of parameter adjustments, used for in-depth analysis of defect causes.

[0072] In this embodiment, through the design of quality traceability and closed-loop optimization, the unique identifier traceability system achieves full lifecycle traceability of defects, providing complete data support for root cause analysis. The root cause analysis process accurately locates key process links through digital twin simulation, solving the problem of difficulty in reverse analysis of defect causes. The model predictive controller continuously optimizes controller parameters through incremental knowledge distillation, improving long-term control performance. This achieves closed-loop quality control, reducing the defect rate of consecutive batches to <0.1%, improving product quality and production efficiency, enhancing the long-term stability and adaptability of the system, and promoting the transformation of quality control from passive response to proactive prevention.

[0073] In some embodiments, the acquisition of the three-mode raw data corresponding to the optical, mid-infrared polarization, and ultrasound of the polarizer includes: A 352MHz global clock signal is generated, and three types of trigger signals are generated based on the global clock signal. The three types of trigger signals include a level signal that controls the exposure timing of optical data acquisition and the 2-microsecond structured light pulse width, a pulse width modulation signal that controls the polarization direction switching of mid-infrared polarization data acquisition, and an independent trigger signal that controls the emission timing of ultrasonic data acquisition. Based on the level signal, the optical acquisition device is controlled to acquire an optical surface image of the polarizer under structured light illumination, and the raw optical data is obtained. Based on the pulse width modulation signal, the mid-infrared polarization acquisition device is controlled to acquire mid-infrared thermal images in four polarization directions to obtain raw mid-infrared polarization data. Based on the independent trigger signal, the ultrasonic acquisition device is controlled to acquire ultrasonic scanning images and obtain raw ultrasonic data. The raw optical data, mid-infrared polarization data, and ultrasonic data are cached in a temporary storage unit. The data in the temporary storage unit are time-calibrated using a timestamp alignment algorithm to keep the acquisition time difference within 50 microseconds.

[0074] The 352 MHz reference clock signal generated by the field-programmable gate array (FPGA) controller serves as the time reference for synchronous acquisition of the three modes of data, ensuring the timing accuracy of various trigger signals and laying the foundation for data time alignment. Targeted control signals generated based on the global clock include a transistor-to-transistor logic level signal (2 microsecond pulse width) controlling the exposure of the optical camera, a pulse width modulation signal (25% duty cycle, 40 microsecond period) controlling the polarization direction switching of the mid-infrared polarization camera, and an independent transistor-to-transistor logic level signal (5 microsecond period) controlling the emission of the ultrasonic probe, each adapted to the acquisition requirements of the three types of imaging equipment. The temporary storage operation of caching the raw optical, mid-infrared polarization, and ultrasonic data to the FPGA block random access memory, combined with a timestamp alignment algorithm, calibrates the acquisition time of the three types of data, keeping the acquisition time difference of the three modes of data within 50 microseconds, ensuring data consistency in the time dimension.

[0075] Transistor-to-transistor logic level signals have two states: high level (typically 5 volts) and low level (typically 0 volts). They are used to control the switching timing of optical camera exposure and ultrasonic probe emission, and have fast response speed and high stability.

[0076] The pulse width modulation signal controls the polarization direction switching speed of the mid-infrared polarization camera by adjusting the duty cycle of the pulse. Each switching step takes less than 10 microseconds, ensuring fast and accurate switching of the polarization direction.

[0077] In this embodiment, through the synchronous design of three-modal data acquisition, a 352 MHz global clock signal provides a high-precision time reference for various trigger signals. The three targeted trigger signals are adapted to the acquisition needs of different imaging devices, ensuring precise control of data acquisition. The data buffering and timestamp alignment algorithm controls the time difference between the three-modal data acquisitions to within 50 microseconds, solving the problem of difficult time alignment of multimodal data. This ensures precise temporal and spatial alignment of the three types of raw data: optical, mid-infrared polarization, and ultrasonic. This provides high-quality, highly consistent raw data for subsequent preprocessing and feature fusion, avoiding a decrease in detection accuracy due to data misalignment. This is the fundamental guarantee for the high-precision realization of the entire detection method.

[0078] This invention presents a polarizer quality inspection method based on multimodal imaging and fusion deep learning. It utilizes complementary acquisition through optical, mid-infrared polarization, and ultrasonic three-modal imaging. The optical modality captures visible surface defects, the mid-infrared polarization modality mines internal material characteristics and transparent defects, and the ultrasonic modality detects internal structural defects such as bubbles and delamination, achieving comprehensive defect coverage and precise data alignment. Targeted preprocessing of the three-modal raw data effectively purifies the data and enhances defect features, laying the foundation for high-precision detection. A dual-branch feature extraction network employs an asymmetric depthwise separable convolution main branch to extract multi-scale local defect features, while a graph attention network auxiliary branch models global distribution correlations. The synergy between local and global features improves recognition accuracy, while asymmetric convolution and input pruning techniques significantly reduce computational load. A dynamic weight fusion module adaptively adjusts the feature fusion ratio through a lightweight gating network, focusing on effective information to improve detection accuracy. Combined with a lightweight classification and positioning head, the detection time per image is shortened, meeting the online real-time detection requirements of production lines, ultimately achieving the dual goals of high precision and real-time performance.

[0079] like Figure 2 As shown, this application also provides a polarizer quality inspection system based on multimodal imaging and fusion deep learning, comprising: The acquisition module is used to acquire raw data of the three modes corresponding to the optical, mid-infrared polarization, and ultrasound of the polarizer; The preprocessing module is used to preprocess the original trimodal data to obtain preprocessed trimodal data. The feature extraction module is used to extract the local defect features and global distribution features of the preprocessed trimodal data through a dual-branch feature extraction network. The feature fusion module is used to fuse the local defect features and the global distribution features using dynamic weights to obtain fused features; The detection module is used to output the quality detection result of the polarizer based on the fusion features, and to perform closed-loop feedback control on the production of the polarizer according to the quality detection result and the production line process parameters.

[0080] This invention discloses a polarizer quality inspection system based on multimodal imaging and fusion deep learning. It utilizes complementary acquisition through optical, mid-infrared polarization, and ultrasonic three-modal imaging. The optical modality captures visible surface defects, the mid-infrared polarization modality mines internal material characteristics and transparent defects, and the ultrasonic modality detects internal structural defects such as bubbles and delamination, achieving comprehensive defect coverage and precise data alignment. Targeted preprocessing of the three-modal raw data effectively purifies the data and enhances defect features, laying the foundation for high-precision detection. A dual-branch feature extraction network employs an asymmetric depthwise separable convolution main branch to extract multi-scale local defect features, while a graph attention network auxiliary branch models global distribution correlations. The synergy between local and global features improves recognition accuracy, while asymmetric convolution and input pruning techniques significantly reduce computational load. A dynamic weight fusion module adaptively adjusts the feature fusion ratio through a lightweight gating network, focusing on effective information to improve detection accuracy. Combined with a lightweight classification and positioning head, the detection time per image is shortened, meeting the online real-time inspection requirements of production lines, ultimately achieving the dual goals of high precision and real-time performance.

Claims

1. A method for detecting the quality of polarizers based on multimodal imaging and fusion deep learning, characterized in that, include: Acquire raw data of the three modes corresponding to the optical, mid-infrared polarization, and ultrasound of the polarizer; The original trimodal data is preprocessed to obtain preprocessed trimodal data; The local defect features and global distribution features of the preprocessed trimodal data are extracted using a dual-branch feature extraction network. By using dynamic weights, the local defect features and the global distribution features are fused to obtain the fused features; Based on the fusion features, the quality inspection result of the polarizer is output, and the production of the polarizer is controlled in a closed loop according to the quality inspection result and the production line process parameters.

2. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, The dual-branch feature extraction network includes a main branch network and an auxiliary branch network. The dual-branch feature extraction network extracts local defect features and global distribution features of the preprocessed trimodal data, including: The main branch network extracts multi-scale basic features from the preprocessed trimodal data through a first-stage asymmetric depthwise separable convolution combination to obtain first-stage features. It then extracts local features from the first-stage features through a second-stage asymmetric depthwise separable convolution combination to obtain second-stage features. Finally, it extracts defect features from the second-stage features through a third-stage asymmetric depthwise separable convolution combination to obtain third-stage features. The third-stage features are then normalized and activated to obtain the local defect features. The auxiliary branch network divides the surface of the polarizer into multiple grid nodes, calculates the feature similarity score between each grid node, normalizes the feature similarity score to generate attention coefficients, updates the graph attention calculation method of the features of the grid nodes based on the attention coefficients, and outputs the global distribution features. The features of each grid node include local defect features and coordinate information output by the main branch network.

3. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, The method of using dynamic weights to fuse the local defect features and the global distribution features to obtain fused features includes: The global attention map corresponding to the global distribution features is upsampled to the same size as the local defect features using bilinear interpolation to obtain the upsampled attention map. The local defect features are concatenated with the upsampled attention map in the channel dimension to obtain the concatenated features; The spliced ​​features are input into a gated network consisting of a 1×1 convolutional layer, an activation function, and global average pooling to generate dynamic weight coefficients with values ​​ranging from 0 to 1. These dynamic weight coefficients are used to adjust the fusion ratio of the local defect features and the global distribution features. The local defect features and the global distribution features are fused according to the dynamic weighting coefficients to obtain the fused features.

4. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, The three-modal raw data includes optical raw data, mid-infrared polarization raw data, and ultrasonic raw data. The preprocessing of the three-modal raw data to obtain preprocessed three-modal data includes: The raw optical data is divided into multiple sub-blocks. Histogram equalization is performed on each sub-block, and the contrast enhancement factor is limited to obtain contrast-enhanced sub-block images. All sub-block images are stitched together to form a complete enhanced image. Gaussian filtering with a 3×3 kernel size and a standard deviation of 1.0 is used to eliminate structured light stripe noise in the enhanced image to obtain a denoised intermediate optical image. The denoised intermediate optical image is binarized by adaptive threshold calculation to generate a surface defect candidate region map. The surface defect candidate region map is used as the preprocessed optical data. The intensity values ​​of the four polarization directions of the original mid-infrared polarization data are decoded to obtain intensity data of the four polarization directions. Stokes vectors are calculated based on the intensity data of the four polarization directions. The degree of polarization and polarization angle are calculated based on the Stokes vectors to obtain polarization feature data. The thermal noise of the polarization feature data is removed by wavelet transform with three-level decomposition to obtain denoised polarization feature data. The denoised polarization feature data is aligned with the preprocessed optical data by feature point registration algorithm to obtain preprocessed mid-infrared polarization data. The environmental noise of the original ultrasound data is removed by wavelet transform with four-level decomposition to obtain denoised original ultrasound data. The denoised original ultrasound data is reconstructed by delay compensation algorithm to generate a two-dimensional thickness distribution map to obtain ultrasound thickness distribution data. The ultrasound thickness distribution data is then adaptively segmented by automatic threshold calculation algorithm to extract abnormal internal structural regions, resulting in preprocessed ultrasound data.

5. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, The step of outputting the quality detection result of the polarizer based on the fusion features includes: The fused features are used as a graph structure, with pixels as nodes and adjacent relationships as edges. The global importance of each node is calculated through a graph attention mechanism to obtain the node importance weight. The fused features are then globally pooled based on the node importance weight to generate a global description vector. The global description vector is input into a convolutional classification layer to output a multi-class defect probability distribution, thereby obtaining the defect type and confidence level. The defect type and confidence level are used as the defect classification result. The fused features are input into a sub-pixel convolution module with 4x upsampling to predict the coordinates and confidence of the defect bounding box, thus obtaining the original localization data containing coordinates and confidence. The duplicate predictions of the original localization data are removed by non-maximum suppression processing with preset confidence threshold and overlap threshold, and the coordinates, area, and aspect ratio of the defect are output to obtain the defect localization result. Based on the defect classification results and the defect location results, the quality inspection results of the polarizer are obtained, including the defect type, severity, and spatial distribution.

6. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, Before outputting the quality detection result of the polarizer based on the fusion features, the method further includes: For the main branch network in the dual-branch feature extraction network, an optimization algorithm is used to reduce the number of multiply-accumulate operations of 3×3 convolution from 9 to 4, resulting in optimized convolution operation logic. Based on the optimized convolution operation logic, the same set of computing units is reused through multi-channel convolution operations, thereby increasing the utilization rate of computing units to above the preset utilization rate and reducing the computation time of local feature extraction process. For the graph attention calculation of the auxiliary branch network in the dual-branch feature extraction network, a parallel computing architecture is adopted. The features of the grid nodes are stored in the cache unit to obtain the cached node features. All attention scores of the cached node features are processed simultaneously by the parallel computing unit array. Each computing unit processes the attention score calculation of one node. Multiple grid nodes are processed in parallel through pipeline design to shorten the latency of global distributed feature extraction.

7. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, After outputting the quality detection result of the polarizer based on the fusion features, the method further includes: Based on the quality inspection results, statistical analysis is performed. When a new defect type is detected in three consecutive batches, or the defect classification accuracy rate is lower than the preset accuracy threshold for five consecutive periods, a pre-trained teacher model is obtained, and the current model used for feature extraction and fusion is used as the student model. The student model is optimized through the strong feature expression capability of the teacher model. Collect 5-10 new defect samples and label the defect type and location to obtain labeled new defect samples; The labeled new defect samples are input into the teacher model for reasoning to generate probability distributions of 10 types of defects as soft labels; The labeled new defect sample is input into the student model for reasoning to obtain the student model reasoning result; Calculate the classification cross-entropy loss between the student model inference result and the sample true label, and the difference loss between the student model inference result and the soft label, and calculate the comprehensive loss based on the classification cross-entropy loss and the difference loss; Based on the comprehensive loss, update the deep convolutional layer parameters of the main branch network of the dual-branch feature extraction network; By configuring an overload method, the updated deep convolutional layer parameters of the main branch network are written into the student model, thereby achieving online updating of model parameters.

8. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, The closed-loop feedback control of the polarizer production based on the quality inspection results and production line process parameters includes: A static rule base is obtained by pre-setting an association matrix between 10 types of defects and 5 types of process parameters. The association matrix includes defect type, main control parameter, adjustment direction, step size, and safety boundary. By using online reinforcement learning, the rule weights of the static rule base are adjusted based on the changes in the defect rate according to the quality inspection results, and an updated rule base for mapping defects to parameters is obtained. The quality inspection results are encapsulated into a fixed 64-byte message and encoded in binary format to obtain the encoded inspection data. The encoded detection data is transmitted to the production line control system via high-speed Ethernet, with the control communication link delay within 1 microsecond and the total end-to-end delay less than 5 milliseconds.

9. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, After performing closed-loop feedback control on the production of the polarizer based on the quality inspection results and production line process parameters, the method further includes: Each polarizer is assigned a unique identifier, and the quality inspection results are linked to the process parameters of the corresponding production stage and stored in the database to establish a defect lifecycle traceability file. Based on the quality inspection results stored in the database, the defect rate of three consecutive batches of polarizers is statistically analyzed. When the defect rate exceeds the preset defect rate threshold, the historical process parameters of the corresponding production cycle are replayed through digital twin simulation to locate the process link that caused the defect and obtain the root cause analysis results. The root cause analysis results are used as new samples to input into the dynamic model predictive controller. The parameters of the dynamic model predictive controller are updated through incremental knowledge distillation to obtain an optimized dynamic model predictive controller. The optimized dynamic model predictive controller is used for closed-loop feedback control to improve the controller's control accuracy of process links.

10. The polarizer quality detection method based on multimodal imaging and fusion deep learning as described in claim 1, characterized in that, The acquired raw data of the three modes corresponding to optical, mid-infrared polarization, and ultrasound of the polarizer includes: A 352MHz global clock signal is generated, and three types of trigger signals are generated based on the global clock signal. The three types of trigger signals include a level signal that controls the exposure timing of optical data acquisition and the 2-microsecond structured light pulse width, a pulse width modulation signal that controls the polarization direction switching of mid-infrared polarization data acquisition, and an independent trigger signal that controls the emission timing of ultrasonic data acquisition. Based on the level signal, the optical acquisition device is controlled to acquire an optical surface image of the polarizer under structured light illumination, and the raw optical data is obtained. Based on the pulse width modulation signal, the mid-infrared polarization acquisition device is controlled to acquire mid-infrared thermal images in four polarization directions to obtain raw mid-infrared polarization data. Based on the independent trigger signal, the ultrasonic acquisition device is controlled to acquire ultrasonic scanning images and obtain raw ultrasonic data. The raw optical data, mid-infrared polarization data, and ultrasonic data are cached in a temporary storage unit. The data in the temporary storage unit are time-calibrated using a timestamp alignment algorithm to keep the acquisition time difference within 50 microseconds.

Images