Intelligent detection method and system for display device based on multi-modal perception

Abstract

The application discloses a kind of display device intelligent detection method and system based on multi-modal perception.The method constructs the closed-loop detection architecture driven by uncertainty: after initial acquisition multi-modal data, by multi-scale feature extraction and input evidence depth model, quantization classification confidence and cognitive uncertainty;When confidence is insufficient, based on pre-labeled modal mutual information matrix, select the target mode that can maximize reduce uncertainty from candidate mode to supplement collection, to cross-modal attention mechanism dynamically fuses new and old features after reasoning again, until meet confidence requirement.After detection is completed, mutual information matrix and model parameters are updated online using modal selection sequence and confidence convergence path.The scheme can compress the average modal collection times under the premise of maintaining defect recognition accuracy, consider detection accuracy and production line rhythm, and have continuous adaptation ability to batch process variation.

Description

Technical Field

[0001] This application relates to the field of automated testing technology for display devices, and more specifically, to an intelligent testing method and system for display devices based on multimodal sensing. Background Technology

[0002] In the manufacturing process of flat panel displays, flexible displays, and microdisplays, defect detection is a crucial step in ensuring yield and quality. As display devices develop towards higher resolution and more complex structures, the types and morphologies of defects are becoming increasingly diverse, encompassing pixel-level bright spots, dark spots, line defects, the Mura effect, and microcracks between film layers. Different types of defects have fundamentally different physical characteristics. A single sensing mode, such as relying solely on visible light imaging, cannot comprehensively and reliably capture the characteristic information of all defects, leading to a high risk of missed detections and false positives.

[0003] To improve the coverage and accuracy of detection, the industry has begun to introduce multimodal sensing solutions, which simultaneously or in a time-sharing manner utilize multiple sensing methods such as visible light imaging, infrared thermography, electroluminescence imaging, and laser interferometry to acquire complementary information. However, most mainstream multimodal detection methods currently follow a pre-defined, fixed process, that is, after acquiring several modal data in a predetermined order, data alignment, feature stitching, or decision-level fusion are performed. Such methods do not consider the real-time state differences of individual devices under test, perform the same modal acquisition sequence for all devices, and lack the dynamic adjustability of the detection strategy.

[0004] In real-world production environments, there are significant variations in process fluctuations and defect distribution between batches. Fixed multimodal acquisition strategies often face a dilemma of "insufficient acquisition" or "over-acquisition" when dealing with such variations: on the one hand, the discriminative modes required for certain sudden or rare defects are not included in the fixed sequence, resulting in missing key information and making it difficult to achieve a reliable classification confidence level; on the other hand, a large number of conventional devices are forced to complete full-modal acquisition, generating redundant data, severely slowing down the inspection cycle, and affecting overall production line efficiency. The root cause of these problems lies in the fact that existing multimodal inspection architectures lack a closed-loop decision-making mechanism based on real-time perceived quality feedback, and cannot adaptively schedule the optimal supplementary modes according to the source and degree of current uncertainty. Summary of the Invention

[0005] This application provides a method and system for intelligent detection of display devices based on multimodal sensing, which at least addresses the problems existing in the prior art.

[0006] A first aspect of this application provides a method for intelligent detection of display devices based on multimodal sensing, comprising the following steps: S1: Based on the type and batch information of the display device under test, select at least two sensing modes as the first mode group, and control them to synchronously acquire initial multimodal data, including spatial images and physical field distribution; S2: Perform multi-scale decomposition on the initial multimodal data, extract local texture features and global frequency features, and concatenate them into a defect response feature vector; S3: Input the defect response feature vector into the evidence depth defect classification model, output the evidence distribution of each defect category, and calculate the classification confidence and cognitive uncertainty; S4: Determine whether the classification confidence level exceeds the preset confidence threshold. If it does, proceed to S7; otherwise, determine that the cognitive uncertainty exceeds the limit, activate the supplementary modality selection and proceed to S5. S5: Based on the current evidence distribution and the pre-calibrated modal mutual information matrix, select the target modality that can maximize the reduction of cognitive uncertainty from the unselected candidate modalities; S6: Trigger target modality acquisition of supplementary data, use cross-modal attention mechanism to dynamically fuse the supplementary data with existing data, update the defect response feature vector, and return to execute S3; S7: Output the defect classification results that meet the confidence requirements, and update the modal mutual information matrix and the parameters of the evidence depth defect classification model based on the modality selection sequence and confidence convergence path of this detection.

[0007] In this embodiment, the detection process is constructed as a closed-loop decision-making architecture driven by uncertainty. After initial modal acquisition, the final result is not given directly. Instead, the current cognitive uncertainty regarding defect identification is quantified using an evidence depth model. Supplementary modal selection and acquisition are only activated when the confidence level is insufficient. The selection of supplementary modalities is not blind; rather, based on a pre-constructed modal mutual information matrix, the target modality with the greatest information gain for reducing current uncertainty is selectively chosen from candidate modalities. Then, a cross-modal attention mechanism dynamically fuses the new data with existing features before re-inference. This closed-loop mechanism of "perception-evaluation-decision-re-perception" ensures that each supplementary acquisition targets the weakest discrimination link, helping to avoid over-acquisition of conventional devices and under-acquisition of abnormal devices. It effectively reduces the average number of modal acquisitions while maintaining high defect identification accuracy, providing a feasible path for the dynamic balance between accuracy and cycle time in production line inspection. Meanwhile, the modality selection sequence and confidence convergence path formed during each inspection process are used to update the mutual information matrix and model parameters online, enabling the inspection strategy to gradually adapt to the process characteristics and defect distribution of the current batch as the production line runs.

[0008] In one or more embodiments of this application, step S3 further includes: S31: Input the defect response feature vector into the evidence depth defect classification model. The model outputs the original evidence value of each preset defect category through a fully connected layer and obtains the non-negative evidence value using a non-negative activation function. S32: Based on non-negative evidence values, construct the lumped parameters of the Dirichlet distribution and generate subjective opinions on defect categories accordingly; S33: Based on subjective opinions, calculate the quality of belief and the quality of uncertainty, take the maximum value of the quality of belief as the classification confidence, and take the quality of uncertainty as the cognitive uncertainty.

[0009] In this embodiment, the evidence depth defect classification model outputs subjective opinions based on the Dirichlet distribution, unifying the classification results with uncertainty in its modeling. Belief quality reflects the model's support for each category, while uncertainty quality quantifies hesitation arising from insufficient evidence. This output method provides a quantifiable and interpretable trigger for determining the necessity and direction of supplementary data collection, helping to avoid wasting data collection resources when the model is already sufficiently definitive, or hastily issuing high-confidence misjudgments when evidence is insufficient.

[0010] In one or more embodiments of this application, step S2 further includes: S21: For each modality in the initial multimodal data, local texture features are extracted in parallel using convolutional kernels of different scales, and global frequency features are extracted through wavelet transform to form a multi-scale feature map; S22: Concatenate the multi-scale feature maps of each modality along the channel dimension and input them into the channel attention module to obtain the weighted multi-modal fusion features; S23: Flatten the multimodal fusion features into a defect response feature vector.

[0011] In this embodiment, local texture features and global frequency features are extracted simultaneously. The former helps to characterize subtle pixel-level defects and edge anomalies, while the latter can capture large-area, slowly changing brightness unevenness. After adaptive weighting by channel attention, the fused features retain both detail discrimination and overall perception of large-area defects, providing richer input for subsequent evidence reasoning.

[0012] In one or more embodiments of this application, the modal mutual information matrix in step S5 is pre-calibrated in the following manner: S51: Collect the defect category labels and corresponding modal response data for each inspection under different modal combinations during historical inspection processes; S52: For any two modal pairs, calculate their conditional mutual information under different defect categories to characterize the amount of shared information between the two modal features when given a defect category, and construct a modal mutual information matrix accordingly.

[0013] In this embodiment, conditional mutual information is used to calibrate the amount of shared information between modal pairs, which can reflect the degree to which one modal feature explains another modal feature under a given defect category. This information-theory-based relational metric provides a data-driven basis for modality selection, enabling supplementary data collection to prioritize modal pairs that have a strong joint discriminative ability for the current high-uncertainty defect category.

[0014] In one or more embodiments of this application, step S5, which selects the target modality that can maximize the reduction of cognitive uncertainty, further includes: S53: Determine the contribution of cognitive uncertainty to each defect category based on the current evidence distribution, and calculate the expected information gain of each candidate modality in the candidate modality set for reducing the current cognitive uncertainty by combining the modality mutual information matrix; S54: Obtain the acquisition cost weight for each candidate mode, and select the optimal candidate mode as the target mode by minimizing the loss function that includes the expected information gain and acquisition cost.

[0015] In this embodiment, the selection of the target modality comprehensively weighs the expected information gain against the acquisition cost. The expected information gain is calculated based on the current evidence distribution and mutual information matrix, reflecting the potential value of each candidate modality in reducing uncertainty; the acquisition cost weight reflects the differences in time consumption, resources, etc., among different modalities. Through joint optimization, it is helpful to prioritize the more cost-effective modality while ensuring detection accuracy, avoiding paying excessive costs for a small increase in confidence.

[0016] In one or more embodiments of this application, step S6, which uses a cross-modal attention mechanism to dynamically fuse supplementary data with existing data, further includes: S61: Encode the supplementary data into supplementary features, linearly map the existing features corresponding to the existing data into query vectors, and linearly map the supplementary features into key vectors and value vectors; S62: Calculate the scaled dot product of the query vector and the key vector to obtain the attention weights, and use the weights to perform weighted aggregation on the value vectors to obtain enhanced supplementary features; S63: Perform residual connection and layer normalization between the enhanced supplementary features and the existing features, and replace the corresponding modal feature segments in the original defect response feature vector to form the updated defect response feature vector.

[0017] In this embodiment, cross-modal attention aggregation is performed using existing features as queries and supplementary features as keys, which can adaptively extract information valuable to the current decision from the supplementary data. Residual connections and layer normalization help to prevent the dilution of existing effective discriminative information while injecting new modal information, supporting smooth convergence of multi-round iterative inference.

[0018] In one or more embodiments of this application, step S7, updating the parameters of the modal mutual information matrix and the evidence depth defect classification model, further includes: S71: Store the modal selection sequence, confidence convergence path, and final defect category label of this test into the experience playback buffer; S72: When the number of samples in the experience replay buffer reaches the preset batch, incremental learning is performed on the evidence depth defect classification model using the samples in the buffer, and the conditional mutual information of modal pairs under different defect categories is re-statistically analyzed to update the modal mutual information matrix.

[0019] In this embodiment, the decision path and result of each detection are stored in an experience replay buffer and incremental learning is triggered. This allows the mutual information matrix to be dynamically adjusted as the production line process changes, and the model gradually adapts to newly emerging defect patterns and batch characteristics. This continuous evolution mechanism helps alleviate the model degradation problem caused by inconsistencies between the deployment environment and the training set distribution.

[0020] In one or more embodiments of this application, before step S1, the method further includes: S01: Collect prior information of the current batch of display devices under test, including process parameters and historical yield. S02: Based on prior information, select at least two modes associated with the most frequently occurring defect type from the modality library to form the first modality group.

[0021] In this embodiment, prior information such as process parameters and historical yield rates is used to predict common defect types in a batch, and the first mode group is determined accordingly. This initial mode selection based on prior knowledge makes the first round of acquisition more targeted, helps to increase the probability of reaching the confidence threshold in the first round, reduces the number of subsequent supplementary acquisitions, and thus improves overall efficiency in batch inspection scenarios.

[0022] In one or more embodiments of this application, step S4 further includes: acquiring the current production cycle signal, dynamically adjusting the preset confidence threshold and the upper limit of the number of iterations based on the production cycle signal; when the production cycle speeds up, increasing the confidence threshold to reduce the time consumed by single-piece inspection, and simultaneously activating the entropy-based information density evaluation mechanism to prioritize selecting the target mode with the largest information gain per unit time from the candidate modes; when the number of iterations in steps S3 to S6 reaches the adjusted upper limit of the number of iterations but still does not meet the confidence threshold, terminating the supplementary acquisition, outputting the defect category with the highest current confidence and marking it as the cycle constraint degradation result; and feeding back the time consumed by this inspection and the final confidence to the production line scheduling system to update the cycle-threshold mapping relationship.

[0023] In this embodiment, the production cycle signal is introduced into the decision-making process to dynamically adjust the confidence threshold and iteration upper limit. When the cycle speed increases, an information density evaluation mechanism is activated, prioritizing modes with high information gain per unit time. This mechanism, linked to production line scheduling, helps to complete the discrimination with reasonable confidence under tight cycle conditions. Feedback of detection time and confidence to the scheduling system provides a reference for the cycle-threshold mapping of subsequent devices.

[0024] In one or more embodiments of this application, the evidence depth defect classification model is a multi-task model, which also includes a defect segmentation branch. In step S3, the model simultaneously outputs a defect segmentation prediction map and the prediction entropy of each pixel in the defect segmentation prediction map. Regions with prediction entropy higher than a preset entropy threshold are marked as segmentation uncertainty regions, and spatial uncertainty masks are generated based on the segmentation uncertainty regions. In the cross-modal attention fusion in step S6, the spatial uncertainty mask is used as an attention bias term and added to the attention weight to enhance feature aggregation of the segmentation uncertainty regions. At the same time, the structural uncertainty coefficient is calculated using the area of ​​the segmentation uncertainty region and the average prediction entropy, and it is fed back as additional evidence quantity to the evidence distribution calculation in step S3 to improve the cognitive uncertainty estimation accuracy of the corresponding defect category.

[0025] In this embodiment, the model simultaneously outputs a segmentation prediction map and its prediction entropy, identifying spatially uncertain regions. The resulting spatial uncertainty mask guides cross-modal attention to these uncertain regions, and the structural uncertainty coefficient is fed back into the evidence distribution calculation to correct the cognitive uncertainty estimate. This allows the uncertainty assessments of the classification and segmentation layers to corroborate each other and synergistically improve.

[0026] In one or more embodiments of this application, multimodal data is acquired by a sensing module array in conjunction with a tunable illumination unit and a switchable polarization filter component; in step S5, while selecting the target mode, the optimal illumination wavelength, polarization angle and exposure parameters corresponding to the target mode are also determined; in step S6, when the target mode is triggered to acquire supplementary data, the tunable illumination unit is switched to the optimal illumination wavelength, and the switchable polarization filter component is adjusted to the polarization angle to maximize the contrast between the specific defect type and the background in the supplementary acquired image.

[0027] In this embodiment, when selecting a supplementary mode, the optimal illumination wavelength, polarization angle and exposure parameters are determined simultaneously. By actively adjusting the illumination and polarization conditions, the image contrast of a specific defect in the corresponding mode is enhanced. For example, a specific wavelength is used to penetrate the transparent film layer or surface glare is filtered out by polarization, thereby improving the signal quality of the supplementary acquisition from the data source.

[0028] In one or more embodiments of this application, the test pattern sent to the display device under test in step S1 or step S6 is generated in the following manner: obtaining the feature activation map or defect segmentation prediction map corresponding to the currently acquired data, locating the local area of ​​the suspected defect through gradient weighted class activation mapping; dynamically generating a local test pattern that matches the texture characteristics of the local area based on the main direction of the image gradient and the spatial frequency distribution in the local area, the local test pattern including sinusoidal stripes or checkerboard patterns with adaptive direction, frequency and contrast; and driving the local test pattern only to the corresponding pixels of the local area, controlling the corresponding modality to perform high-resolution scanning acquisition of the local area as part of the initial multimodal data or supplementary data.

[0029] In this embodiment, the test pattern is dynamically generated based on the feature activation map or segmentation prediction map of the collected data. It matches the texture characteristics of the suspected defect in terms of direction, frequency, and contrast, and is driven only to the corresponding pixels in the local area. This adaptive pattern helps to stimulate and highlight the response of local weak defects, improving the discriminative power of supplementary data for specific defects.

[0030] A second aspect of this application provides a display device intelligent detection system based on multimodal sensing, comprising: A multimodal sensing module array, including at least two of a visible light camera, an infrared thermal imager, an electroluminescent imaging device, and a laser speckle interferometer, is used to acquire multimodal data of the display device under test; The synchronous triggering and driving module is used to generate test pattern driving signals and generate synchronous trigger pulses to enable the multimodal sensing module array to be exposed synchronously. The feature extraction and fusion module is used to perform multi-scale feature extraction, defect response feature vector construction, and cross-modal attention fusion in the above methods. The evidence reasoning and decision-making module stores a deep evidence defect classification model, which is used to output the evidence distribution based on the defect response feature vector, calculate the classification confidence and cognitive uncertainty, and determine whether the confidence threshold has been reached. The modality selection and scheduling module stores a modality mutual information matrix, which is used to select the target modality from the candidate modalities when the cognitive uncertainty exceeds the limit, and send a supplementary acquisition command to the corresponding sensing unit. The online learning update module is used to update the modal mutual information matrix and the parameters of the evidence depth defect classification model based on the modality selection sequence, confidence convergence path and final defect category in the detection process.

[0031] The detection system provided in this application integrates multimodal perception, synchronous triggering, feature extraction and fusion, evidence reasoning and decision-making, modality selection scheduling, and online learning and updating into a closed-loop whole. Each module collaborates to complete the entire chain from data acquisition, uncertainty assessment, on-demand scheduling to continuous optimization, which facilitates the stable deployment and operation of the aforementioned methods in production line environments. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating an intelligent detection method for display devices based on multimodal sensing, provided as an embodiment of this application. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0034] In the field of display device manufacturing inspection, as products develop towards higher resolution and more complex structures, the types and forms of defects are becoming increasingly diverse. A single sensing modality is no longer sufficient to comprehensively capture the characteristic information of various defects, leading the industry to introduce multimodal sensing solutions. However, current mainstream multimodal inspection methods generally follow a fixed, preset process: collecting data from several modalities in a predetermined order, then performing data alignment, feature stitching, or decision-level fusion, applying the same modal acquisition sequence to all devices under test. This static inspection mode often faces a dilemma when dealing with batch-to-batch process fluctuations and changes in defect distribution: on the one hand, the discriminative modalities required for certain sudden or rare defects are not included in the fixed sequence, making it difficult to achieve a reliable classification confidence level; on the other hand, a large number of conventional devices are forced to complete full-modal acquisition, generating redundant data and severely slowing down the inspection cycle. The difficulty in balancing inspection accuracy and production line efficiency constitutes a prominent contradiction in existing technologies.

[0035] The inventors recognized in their actual R&D practice that the root of the aforementioned contradiction lies in the lack of a closed-loop decision-making mechanism based on real-time perceived quality feedback in existing solutions. Under a fixed process, the system lacks a quantitative understanding of whether the existing data is sufficient to make a reliable judgment, and it cannot selectively allocate supplementary information based on the source and degree of uncertainty. In other words, existing solutions treat multimodal detection as an open-loop data acquisition and fusion process, rather than a closed-loop reasoning process that can dynamically adjust strategies based on intermediate results. This architectural deficiency prevents the detection system from promptly terminating acquisition to save time when information is sufficient, and from accurately locating the modality types that need supplementation when information is insufficient.

[0036] Based on the above understanding, the inventors began to construct a solution from two levels: first, to introduce the ability to quantify the "uncertainty" of the detection system's own judgment, that is, to output not only the classification result, but also the confidence assessment of that result; second, to establish a decision-making mechanism that can selectively choose the supplementary modality from candidate modalities that best eliminates the current uncertainty when uncertainty exceeds the limit. Regarding the quantification of uncertainty, the inventors noted that the evidence deep learning framework can model classification results and cognitive uncertainty in a unified way by outputting evidence distribution and constructing subjective opinions, thereby providing interpretable triggering basis for subsequent decisions. Regarding modality selection decisions, the inventors drew on the concept of conditional mutual information in information theory, pre-labeling the information sharing relationship of each modality pair under different defect categories using historical data, so that when the judgment of a certain defect category has high uncertainty, it is possible to quantitatively calculate which candidate modality can bring the greatest information gain.

[0037] Therefore, this application proposes a closed-loop detection scheme driven by uncertainty and with modality scheduling on demand: after the initial modality acquisition, the results are not directly output, but are first extracted through multi-scale features and then input into the evidence depth model, simultaneously obtaining classification tendency and cognitive uncertainty; when the uncertainty exceeds the limit, based on the pre-calibrated modality mutual information matrix, the target modality that can maximize the reduction of the current uncertainty is selected for supplementary acquisition, and the new and old data are dynamically fused and re-inferred using a cross-modal attention mechanism, and this process is iterated until the confidence level meets the requirements; after the detection is completed, the modality selection sequence and confidence convergence path formed by this detection are used to update the mutual information matrix and model parameters online, enabling the detection strategy to gradually adapt to the current batch process characteristics and defect distribution as the production line runs.

[0038] To make the purpose, technical solution, and advantages of this application clearer, the following will be described in conjunction with the appendix. Figure 1 The following is an explanation using specific examples.

[0039] Please refer to Figure 1 , Figure 1 The first aspect of this application provides a method for intelligent detection of display devices based on multimodal sensing, comprising the following steps: S1: Based on the type and batch information of the display device under test, select at least two sensing modes as the first mode group, and control them to synchronously acquire initial multimodal data, including spatial images and physical field distribution; S2: Perform multi-scale decomposition on the initial multimodal data, extract local texture features and global frequency features, and concatenate them into a defect response feature vector; S3: Input the defect response feature vector into the evidence depth defect classification model, output the evidence distribution of each defect category, and calculate the classification confidence and cognitive uncertainty; S4: Determine whether the classification confidence level exceeds the preset confidence threshold. If it does, proceed to S7; otherwise, determine that the cognitive uncertainty exceeds the limit, activate the supplementary modality selection and proceed to S5. S5: Based on the current evidence distribution and the pre-calibrated modal mutual information matrix, select the target modality that can maximize the reduction of cognitive uncertainty from the unselected candidate modalities; S6: Trigger target modality acquisition of supplementary data, use cross-modal attention mechanism to dynamically fuse the supplementary data with existing data, update the defect response feature vector, and return to execute S3; S7: Output the defect classification results that meet the confidence requirements, and update the modal mutual information matrix and the parameters of the evidence depth defect classification model based on the modality selection sequence and confidence convergence path of this detection.

[0040] In this embodiment, the sensing modality mentioned in step S1 refers to the method of acquiring information about the display device under test through different physical principles or imaging techniques. The types of modalities include, but are not limited to, visible light imaging, infrared thermal imaging, electroluminescence imaging, laser speckle interferometry imaging, photoluminescence imaging, and X-ray imaging. Spatial images refer to image data reflecting the distribution of light intensity, color, reflectivity, etc., on the surface or interior of the display device under test in two-dimensional or three-dimensional space, such as photographs of the device surface taken by a visible light camera. Physical field distribution refers to data reflecting the spatial distribution of a certain physical quantity of the display device under test, such as the temperature field distribution reflected by infrared thermal imaging, the luminous intensity distribution reflected by electroluminescence images, and the surface deformation or stress field distribution reflected by laser speckle interferometry fringes. The first modal group refers to the set of modalities that are synchronously acquired in the first batch at the start of testing for the current display device under test; its selection can be based on device type and batch information. Synchronous acquisition refers to using a unified trigger signal to enable sensing units of different modalities to complete exposure or data acquisition at the same time or within a very short time window, so as to ensure the correspondence of data from each modality in the time dimension.

[0041] The multi-scale decomposition mentioned in step S2 refers to using multiple scales or resolutions to analyze the same modal data and extract feature information at different levels. Local texture features typically reflect the gray-level variation patterns of an image within a small neighborhood. They can be extracted using convolution kernels of different sizes, local binary mode operators, or Gabor filter banks, and are suitable for describing pixel-level bright spots, dark spots, microcracks, and other subtle defects. Global frequency features typically reflect the spectral distribution characteristics of an image over a whole or a large area. They can be extracted using wavelet transform, Fourier transform, or discrete cosine transform, and are suitable for describing large-area murmurs, brightness gradients, periodic stripes, and other defects. The defect response feature vector is a one-dimensional or multi-dimensional vector representation formed by splicing or fusing the aforementioned multi-modal and multi-scale features. It serves as the input to the subsequent classification model, and its dimension is related to the feature extraction method.

[0042] The evidence-based defect classification model mentioned in step S3 refers to a deep neural network classifier built based on evidence theory or subjective logic theory. Unlike traditional classifiers that output probability values ​​for each category, this model outputs the evidence distribution for each preset defect category. Evidence can be understood as the amount of information supporting that a sample belongs to a certain category. Preset defect categories refer to a set of defect types predefined based on production line experience and product specifications, which may include bright spots, dark spots, line defects, murras, scratches, foreign objects, bubbles, film peeling, etc. Classification confidence refers to the degree of certainty the model has regarding the classification result of the current sample, which can be derived from the evidence distribution. Cognitive uncertainty refers to the degree of hesitation or uncertainty the model exhibits regarding the classification result due to insufficient training data coverage, samples being at the category boundary, or the existence of contradictory evidence. For example, a high classification confidence and low cognitive uncertainty indicate that the model has sufficient and consistent evidence to support that the current sample belongs to a certain category; a low classification confidence and high cognitive uncertainty indicate that the model lacks sufficient information to make a reliable judgment and may need to supplement with new perceptual data. The specific structure of the evidence depth defect classification model can be built based on convolutional neural networks, Transformers or their variants. Its output layer generates evidence values ​​through non-negative activation functions and then maps them to the parameter space of the Dirichlet distribution.

[0043] The preset confidence threshold mentioned in step S4 refers to the confidence level threshold used to determine whether the current classification result is sufficient for acceptance. When the classification confidence exceeds this threshold, it can be considered that the model's judgment of the current sample has reached an acceptable level, and there is no need to continue supplementing data; the process can directly proceed to the result output stage. When the classification confidence does not exceed this threshold, it indicates that the model is not certain about the current judgment, and the cognitive uncertainty exceeds the limit. At this time, the supplementary modality selection mechanism is activated. It is understandable that the preset confidence threshold is not fixed and its value can be adjusted according to the different requirements of the production line for accuracy and cycle time: when the production line has a low tolerance for misjudgment, the confidence threshold can be set higher to trigger more supplementary data collection to improve the reliability of the judgment; when the production line cycle time is tight, the confidence threshold can be appropriately reduced to shorten the average detection time.

[0044] The modal mutual information matrix mentioned in step S5 refers to a pre-calibrated matrix used to characterize the information sharing relationship between any two perceptual modalities. The elements in the matrix can be calculated based on conditional mutual information, reflecting the amount of information shared between the features of the two modalities under a given defect category. A higher conditional mutual information value usually means that the two modalities have complementary or cooperative discriminative abilities in the corresponding defect category. Candidate modalities refer to other perceptual modalities configured in the system and available for scheduling, besides the currently acquired modalities. The target modality refers to the modality selected from the candidate modalities that is expected to be most valuable in reducing current cognitive uncertainty. The selection process can determine the uncertainty contribution of each defect category based on the current evidence distribution, calculate the expected information gain of each candidate modality in conjunction with the modal mutual information matrix, and select the modality with the largest information gain as the target modality.

[0045] The cross-modal attention mechanism mentioned in step S6 refers to a method of interacting and aggregating features from different modalities through attention computation. Specifically, supplementary data can be encoded as supplementary features, existing features corresponding to existing data can be used as queries, and supplementary features can be used as keys and values. Attention weights are obtained by calculating the similarity between the query and the key, and then these weights are used to weighted aggregate the values, thereby selectively extracting information relevant to the current decision from the supplementary features. Dynamic fusion means that the fusion process is not a simple feature concatenation or summation, but rather adaptively determines which parts of the supplementary features are emphasized or suppressed based on the specific content of the existing and supplementary features. Updating the defect response feature vector means replacing the corresponding modal feature segments in the original vector with the fused features or reconstructing a new feature vector for use in the next round of evidence reasoning.

[0046] The modality selection sequence mentioned in step S7 refers to the record of the modalities and their order selected and collected by the system sequentially from the initial modality group to the point where the confidence threshold is finally met during a complete detection process. The confidence convergence path refers to the trajectory of changes in classification confidence and cognitive uncertainty after each iteration in the above process, reflecting the degree of improvement in the discrimination results from each supplementary collection. Updating the modality mutual information matrix refers to recalculating the relevant conditional mutual information values ​​and correcting the corresponding elements in the matrix using the modality combinations and final defect category labels obtained in this detection. Updating the parameters of the evidence depth defect classification model refers to using the samples formed in this detection (including multimodal features and final defect categories) for online training or fine-tuning of the model, enabling the model to adapt to process drift or new defect morphologies that may occur during production line operation.

[0047] The steps mentioned above are not isolated, but rather work together and coordinate around the main thread of closed-loop reasoning driven by uncertainty.

[0048] First, the multi-scale decomposition and feature extraction in step S2 provide information-rich and discriminative input for the evidence reasoning in step S3. The combination of local texture and global frequency features enables the model to simultaneously perceive subtle defects and large-area anomalies. The cognitive uncertainty output in step S3 provides the triggering condition for the entire closed loop. This can be understood as follows: without the quantification of uncertainty in step S3, step S4 would be unable to determine whether the current data is sufficient for a reliable decision; the system would simply blindly execute a fixed process, unable to know when to terminate data collection or when supplementary information is needed.

[0049] Secondly, the modality selection decision in step S5 relies on the evidence distribution output from step S3 and the pre-calibrated mutual information matrix. The evidence distribution indicates the main sources of current uncertainty, i.e., which defect categories the model is hesitating between; the mutual information matrix provides information on the discrimination potential of different modalities for each defect category. The combination of these two factors makes the selection of supplementary modalities clearly targeted and computable, avoiding the inefficient practice of random trials or traversing all remaining modalities. This information gain-based targeted selection is a crucial bridge connecting uncertainty perception and precise scheduling.

[0050] Furthermore, the cross-modal attention fusion in step S6 is not simply a matter of stacking new and old data. Instead, it selectively absorbs information from the supplementary data that is beneficial for reducing current uncertainty through an attention mechanism, while retaining the confirmed effective parts of the existing features in a residual manner. This fusion method ensures that the defect response feature vector after each iteration contains more sufficient discriminative criteria than the previous iteration, thus potentially achieving higher classification confidence and lower cognitive uncertainty in the re-inference in step S3. Steps S4, S5, and S6, together with the return step S3, constitute a spiraling inference loop. Each round of supplementary data collection specifically enhances the information base based on the previous round until the confidence level crosses the threshold.

[0051] Finally, the online update in step S7 feeds back the experience from a single detection to the entire system. The modality selection sequence and confidence convergence path record the decision trajectory of this detection, and their correlation with the final defect category can be used to correct the information content estimates of each modality pair in the mutual information matrix, making the basis for subsequent modality selection more accurate. At the same time, the incremental update of model parameters allows the classifier to gradually adapt to the actual distribution of the production line, reducing the frequent exceedance of cognitive uncertainty caused by batch changes. This online update mechanism transforms each detection action itself into a driving force for system self-improvement, enabling the overall effectiveness of the solution to gradually improve with the accumulation of running time.

[0052] In one or more embodiments of this application, step S3 further includes: S31: Input the defect response feature vector into the evidence depth defect classification model. The model outputs the original evidence value of each preset defect category through a fully connected layer and obtains the non-negative evidence value using a non-negative activation function. S32: Based on non-negative evidence values, construct the lumped parameters of the Dirichlet distribution and generate subjective opinions on defect categories accordingly; S33: Based on subjective opinions, calculate the quality of belief and the quality of uncertainty, take the maximum value of the quality of belief as the classification confidence, and take the quality of uncertainty as the cognitive uncertainty.

[0053] In this embodiment, the original evidence value mentioned in step S31 refers to the numerical value directly output by the fully connected layer of the evidence depth defect classification model, before it has been processed by the activation function. This value can be positive, negative, or zero, reflecting the initial support strength of the model for each preset defect category. The non-negative activation function is a function that maps the original evidence value to the non-negative real number domain. Softplus, exponential linear unit, or modified linear unit functions can be used, etc., to ensure the non-negativity requirement of the lumped parameters when constructing the Dirichlet distribution. The non-negative evidence value is the numerical value greater than or equal to zero obtained after processing by the non-negative activation function. Its magnitude can be intuitively understood as the amount of evidence collected by the model supporting that the sample belongs to a certain defect category: the larger the value, the more sufficient the information supporting that category is considered by the model; the closer the value is to zero, the more supportive information the model has not yet observed.

[0054] The Dirichlet distribution mentioned in step S32 is a continuous multivariate probability distribution defined on a probability simplex, and the parameters of its probability density function are called lumped parameters. Constructing the lumped parameters of the Dirichlet distribution based on non-negative evidence values ​​can be achieved by adding the non-negative evidence values ​​to a preset basic prior value. The basic prior value is usually set to 1, representing a uniform prior when no evidence has been observed. For example, if the model outputs non-negative evidence values ​​for three defect categories as e1, e2, and e3, the corresponding lumped parameters of the Dirichlet distribution can be constructed as α1=e1+1, α2=e2+1, and α3=e3+1. Subjective opinion refers to a complete information structure about category attribution defined by the Dirichlet distribution. It not only includes estimates of the probability of each category but also a quantification of the uncertainty of the estimates themselves, thus extending the traditional probability output. The introduction of subjective opinion allows the model to express its own perception of the reliability of its judgments, in addition to category bias.

[0055] The belief quality mentioned in step S33 refers to the degree of support for the pre-defined defect category assignments in subjective opinions. It can be calculated by subtracting the basic prior value from the corresponding component in the lumped parameters and then dividing by the total amount of evidence. It reflects the strength of the model's reasonable belief that a sample belongs to each category given the existing evidence. Uncertainty quality refers to the remaining portion of subjective opinions that has not been assigned to any category. It can be calculated by dividing the total number of categories by the total amount of evidence. The smaller the total amount of evidence, the higher the uncertainty quality, indicating the degree of hesitation the model exhibits due to insufficient evidence. Using the maximum belief quality as the classification confidence means using the belief quality corresponding to the category the model most favors as a measure of confidence in the overall classification result. Using uncertainty quality as cognitive uncertainty means using the uncertain portion reserved by the model due to limited evidence as one of the decision bases for triggering supplementary data collection. It is understandable that when the overall non-negative evidence value is small, the total amount of evidence is low, the uncertainty quality is high, and the classification confidence is low. As more modal data is added and features are enhanced, the non-negative evidence value gradually increases, the total amount of evidence rises, the uncertainty quality decreases accordingly, and the classification confidence increases.

[0056] The steps S31 to S33 above together constitute a complete computational chain from feature vectors to classification confidence and cognitive uncertainty, and there is a close progression and dependency between each step.

[0057] First, step S31, generating nonnegative evidence values ​​from the defect response feature vector, is the starting point of the entire computational chain. Unlike traditional classifiers that directly output probability values, the introduction of evidence values ​​allows the model to express the strength of evidence support for each category, rather than merely the relative comparison results between categories. This design makes it possible to quantify uncertainty in subsequent steps, because in a model that only outputs probabilities, the sum of probability values ​​is always 1 regardless of whether the actual evidence is sufficient, making it difficult to distinguish between "a definite judgment with sufficient evidence" and "random guesses when there is a lack of evidence." Step S32, mapping nonnegative evidence values ​​to the lumped parameters of a Dirichlet distribution and constructing subjective opinions, plays a crucial role in connecting the preceding and following steps: on the one hand, the way the lumped parameters are constructed links the amount of evidence to the degree of concentration of the distribution; the more sufficient the evidence, the more concentrated the distribution is on a certain category; on the other hand, the framework of subjective opinions naturally incorporates classification support and uncertainty into a unified mathematical expression.

[0058] Step S33 separates belief quality and uncertainty quality from subjective opinions, providing clear two-dimensional information for the subsequent judgment in step S4. The maximum value of belief quality is directly used as the classification confidence score, which can be used for threshold comparison; uncertainty quality, as cognitive uncertainty, reflects the degree of information scarcity in the model. This two-dimensional output method makes the judgment in step S4 interpretable: when the uncertainty quality is high, it means that the total amount of evidence is insufficient, and the model lacks enough information to support a reliable judgment. In this case, regardless of whether the maximum value of belief quality is accidentally high, caution should be exercised, and supplementary data collection should be triggered. When the uncertainty quality is low, it means that the total amount of evidence is sufficient, and the model has enough information to make a judgment. In this case, the maximum value of belief quality has high credibility. Thus, the uncertainty quantification mechanism in steps S31 to S33 provides a stable, interpretable, and mathematically based triggering basis for the decision on whether supplementary data collection is needed in the entire closed-loop detection scheme.

[0059] In one or more embodiments of this application, step S2 further includes: S21: For each modality in the initial multimodal data, local texture features are extracted in parallel using convolutional kernels of different scales, and global frequency features are extracted through wavelet transform to form a multi-scale feature map; S22: Concatenate the multi-scale feature maps of each modality along the channel dimension and input them into the channel attention module to obtain the weighted multi-modal fusion features; S23: Flatten the multimodal fusion features into a defect response feature vector.

[0060] In this embodiment, the convolutional kernels of different scales mentioned in step S21 refer to convolutional filters with different receptive field sizes, such as 3×3, 5×5, and 7×7 kernels, or kernels that expand the receptive field while maintaining the number of parameters through dilated convolution. Small-sized convolutional kernels tend to capture fine-grained texture changes and are suitable for detecting small-area defects such as pixel-level bright spots, dark spots, and microcracks; large-sized convolutional kernels tend to capture gray-level fluctuation patterns over a larger range and are suitable for detecting defects such as regional brightness unevenness or blocky mura. Local texture features refer to the spatial distribution pattern of pixel gray-level values ​​in a small neighborhood of an image, which can reflect information such as the boundary direction, roughness, and directionality of defects. Wavelet transform is a time-frequency analysis method that decomposes the image into multiple levels to obtain sub-band coefficients of different scales and directions, thereby extracting global frequency features. Global frequency features can reflect the periodic or non-periodic variation patterns of the image over the entire range. For example, large-area, slowly changing mura is obvious in low-frequency sub-bands, while periodic stripe defects have a strong response in high-frequency sub-bands in specific directions. Multi-scale feature maps are a collection of feature maps extracted from the same modality data through convolutional kernels and wavelet transforms of different scales. The number of channels in a multi-scale feature map is equal to the sum of the number of output channels of each extraction branch.

[0061] The concatenation along the channel dimension mentioned in step S22 refers to connecting multiple multi-scale feature maps from different perceptual modalities along the channel dimension to form a composite feature tensor with the number of channels equal to the sum of the number of channels in each modality's feature map. The channel attention module is a computational unit that learns the dependencies between channels and assigns different weight coefficients to each channel. It can be implemented using SENet (Squeeze-and-Excitation Network) structure, ECA-Net (Efficient Channel Attention Network) structure, or other channel attention mechanisms. The channel attention module obtains global statistical information for each channel through global average pooling, and then generates scaling coefficients for each channel via fully connected layers or one-dimensional convolutional layers. These coefficients reflect the importance of the corresponding channel features in the current detection task. The weighted multimodal fusion feature refers to the result obtained by multiplying each channel of the composite feature tensor by its corresponding scaling coefficient, thereby enhancing features of modalities and scales more valuable for current defect detection, while suppressing features that contribute less.

[0062] Step S23, which mentions flattening the multimodal fusion features, refers to converting the weighted multimodal fusion feature tensor from a multidimensional array to a one-dimensional vector for input into subsequent fully connected layers for evidence reasoning. The flattening operation can arrange the elements into a one-dimensional sequence in either channel-first or space-first order. The defect response feature vector is the flattened one-dimensional vector, which integrates comprehensive feature information from multiple perceptual modalities, multiple scales, and adaptively weighted by channel attention. It serves as the direct input to the subsequent evidence-depth defect classification model.

[0063] The steps S21 to S23 described above together constitute a complete feature engineering chain from raw multimodal data to defect response feature vectors that can be used for model inference, with clear progressive and collaborative relationships between each step.

[0064] First, step S21 performs multi-scale decomposition in parallel for each modality of data. This design is based on the fact that defects in display devices exhibit significant scale differences; pixel-level bright spots may occupy only a few pixels, while large-area muras may span tens or even hundreds of pixels. If only a single-scale analysis is used, small defects may be overwhelmed by smoothing filters, or large defects may be difficult to perceive as a whole due to insufficient receptive field. Simultaneously extracting local texture features and global frequency features ensures that the feature representation retains both the discriminative power of microscopic details and the ability to perceive macroscopic anomalies, thus providing a multi-dimensional information foundation for subsequent classification.

[0065] Next, step S22 aggregates features from each modality and scale and applies channel attention weighting, which serves to filter and integrate information. Features from different modalities do not contribute equally to the discrimination of different defect types: for example, electroluminescent images are more sensitive to pixel-level luminescence anomalies, infrared thermography is more sensitive to local temperature rises caused by short circuits or leakage, and laser speckle interferometry is more sensitive to film stress anomalies. The channel attention module, by learning the weight coefficients of each channel, can adaptively strengthen feature channels highly correlated with the possible defect types of the current sample, while weakening irrelevant or redundant channels. This adaptive weighting helps reduce the interference of irrelevant modal noise on subsequent inference and improves the overall signal-to-noise ratio of the features.

[0066] Furthermore, step S23 flattens the weighted fused features into a defect response feature vector, which serves as the input to the evidence-depth defect classification model in step S3. The feature vector, after multi-scale decomposition, cross-modal concatenation, and channel attention weighting, contains richer and more targeted discriminative information compared to directly concatenating the original modal data or simple feature overlay. This helps the evidence-depth defect classification model more accurately estimate the evidence distribution of each defect category, thereby improving initial classification confidence, reducing initial cognitive uncertainty, and ultimately decreasing the probability of triggering supplementary data collection, thus improving detection efficiency.

[0067] In one or more embodiments of this application, the modal mutual information matrix in step S5 is pre-calibrated in the following manner: S51: Collect the defect category labels and corresponding modal response data for each inspection under different modal combinations during historical inspection processes; S52: For any two modal pairs, calculate their conditional mutual information under different defect categories to characterize the amount of shared information between the two modal features when given a defect category, and construct a modal mutual information matrix accordingly.

[0068] In this embodiment, the historical testing process mentioned in step S51 refers to the testing operation records completed before the current batch testing for display devices of the same or similar models. This may include offline testing data from the trial production stage, online testing data from previous batches, or specially organized modal calibration experimental data. Different modal combinations refer to the combination of two or more sensing modalities actually used in the historical testing process, such as a combination of visible light images and infrared thermal images, a combination of visible light images and electroluminescent images, or a combined acquisition of three modalities. Defect category labeling refers to the true defect category label assigned to each sample after manual review or confirmation by authoritative testing methods. For example, a sample may be labeled as a bright spot defect, a sample as a Mura defect, or a sample as defect-free. Corresponding modal response data refers to the raw data acquired by each participating modality on the same test sample, or the feature data extracted after preliminary processing, such as images captured by a visible light camera, temperature matrices recorded by an infrared thermal imager, or feature vectors extracted from each modal data. It is understandable that the scale and quality of historical data have a certain impact on the reliability of subsequent mutual information calculations. Generally, the more comprehensive the defect categories covered by the data and the more sufficient the number of samples, the better the calculated mutual information matrix can reflect the true intermodal relationships.

[0069] The modal pair mentioned in step S52 refers to an ordered or unordered combination of two modalities arbitrarily selected from all available perceptual modalities. Conditional mutual information is a concept in information theory used to measure the amount of information shared between two variables given a third variable. In the context of this step, conditional mutual information is used to characterize the amount of shared information between two modal features given a defect category. For example, if the conditional mutual information value between visible light image features and electroluminescent image features is high when the defect category is known to be a bright spot defect, it indicates that the two have strong consistency or complementary information in the bright spot defect category; conversely, if the conditional mutual information value is low, it indicates that the feature performance of the two in this defect category is relatively independent, and they may each capture different aspects of the defect. The modal mutual information matrix is ​​a matrix constructed with all available modalities as rows and columns. The element in the i-th row and j-th column of the matrix records the conditional mutual information value between modality i and modality j under different defect categories. The final value of this element can be the average, maximum, or weighted sum of each defect category. The diagonal elements of the matrix can be set to maximum or invalid values, representing the information relationship between the modality and itself. The modal mutual information matrix provides a pre-calculated data foundation for quickly querying and calculating the expected information gain of each candidate modality in subsequent online detection.

[0070] The above steps S51 and S52 together constitute the offline calibration process of the modal mutual information matrix, and there is a clear succession relationship between the two.

[0071] First, the historical data collected in step S51 forms the foundation of the entire calibration process. The coverage of the data directly impacts the reliability of the mutual information matrix: if the historical data contains only a small number of defect categories or some mode combinations are missing, the corresponding conditional mutual information estimation may not be stable enough. In practical applications, calibration experiments covering all major defect categories and multiple mode combinations can be consciously organized to accumulate more comprehensive calibration data. As the production line continues to operate, newly generated inspection data can also be continuously incorporated into the historical dataset, gradually enriching the calibration data.

[0072] Step S52, based on the data provided in Step S51, uses conditional mutual information as a metric to quantify the relationships between modalities into queryable and comparable numerical values. The choice of conditional mutual information over simple correlation coefficients or unconditional mutual information is based on specific considerations: the relationships between modalities may differ significantly under different defect categories. For example, visible light images and infrared thermal images may have a weak correlation under the bright spot defect category, but a strong correlation under the short circuit defect category. Calculating the overall mutual information without conditions would smooth out these inter-category differences, leading to a lack of specificity in subsequent modality selection. By calculating conditional mutual information by defect category, the modal mutual information matrix can more precisely characterize the relationships between modalities, providing the ability to selectively target modalities in Step S5 based on the current evidence distribution. This makes the selection of supplementary modalities more likely to effectively respond to the specific sources of uncertainty.

[0073] In one or more embodiments of this application, step S5, which selects the target modality that can maximize the reduction of cognitive uncertainty, further includes: S53: Determine the contribution of cognitive uncertainty to each defect category based on the current evidence distribution, and calculate the expected information gain of each candidate modality in the candidate modality set for reducing the current cognitive uncertainty by combining the modality mutual information matrix; S54: Obtain the acquisition cost weight for each candidate mode, and select the optimal candidate mode as the target mode by minimizing the loss function that includes the expected information gain and acquisition cost.

[0074] In this embodiment, the cognitive uncertainty contribution mentioned in step S53 refers to the uncertainty component corresponding to each defect category after decomposing the overall cognitive uncertainty calculated in step S3 according to the relative strength of the current evidence distribution across each preset defect category. For example, if the current evidence distribution shows that the model's evidence values ​​for both bright spot defects and Mura defects are low and similar, then the cognitive uncertainty is mainly contributed by these two categories, and their respective contribution ratios can be allocated according to the dispersion or entropy value of the evidence distribution. The candidate modality set refers to the set of modalities remaining after excluding the currently acquired modalities from all available perceptual modalities configured in the system. The expected information gain refers to the expected reduction in cognitive uncertainty after assuming that data from a candidate modality is acquired and used in fusion inference. The expected information gain can be calculated by combining the cognitive uncertainty contribution of each defect category with the modal mutual information matrix: for a candidate modality, query the conditional mutual information value between it and the currently collected modality in the defect category with a high cognitive uncertainty contribution. The higher the conditional mutual information value, the stronger the complementarity between the candidate modality and the existing information in the corresponding category. The greater the expected information increment after introducing the modality, the greater the expected contribution to reducing cognitive uncertainty.

[0075] The acquisition cost weight mentioned in step S54 refers to a pre-set quantitative indicator for each candidate mode, reflecting the resources consumed in its acquisition process. Acquisition cost can include one or more factors such as acquisition time, equipment occupancy time, energy consumption, data processing complexity, and equipment lifespan depletion, which are normalized or weighted summed to form a comparable value. For example, the cost weight for a visible light camera acquiring one frame of image can be set to a lower value, while the cost weight for a laser speckle interferometer, which requires multiple phase-shift acquisitions and phase calculations, can be set to a higher value. The loss function is a mathematical function that combines the expected information gain and acquisition cost into a scalar target value. Its design principle is usually to favor modes with lower costs when expected information gains are similar, or modes with higher expected information gains when costs are similar. The loss function can be a weighted difference, ratio, or other combination of expected information gain and acquisition cost. By minimizing this loss function, i.e., finding the candidate mode that minimizes the loss function value, a balance can be achieved between information gain and acquisition cost, selecting the optimal candidate mode as the target mode.

[0076] The above steps S53 and S54 work together within the modality selection framework of step S5 to transform the modality selection process, which may have previously relied on experience or preset rules, into a quantifiable and optimizable decision-making process.

[0077] Step S53 first utilizes the evidence distribution output from step S3 to refine the overall cognitive uncertainty down to the level of each defect category, clarifying the specific sources of the current uncertainty. This refinement forms the basis for subsequent targeted selection: if cognitive uncertainty is treated as a whole without differentiation, only a general selection of modalities can be made, making it difficult to specifically eliminate the most hesitant discriminative dimension. By combining the contribution of cognitive uncertainty with the modal mutual information matrix, step S53 can query which candidate modalities have strong conditional mutual information relationships with existing modalities in defect categories where the current uncertainty is concentrated, thereby calculating the expected value of each candidate modality in reducing the current specific uncertainty. This targeted calculation method helps avoid selecting modalities with limited information gain in the current context, improving the targeting and efficiency of supplementary data collection.

[0078] Step S54, based on the expected information gain provided in Step S53, introduces the acquisition cost dimension to further adapt the modality selection decision to the actual operational constraints of the production line. In engineering practice, the acquisition time and equipment resource consumption of different modalities may vary by orders of magnitude. If only maximizing information gain is the goal, high-cost modalities may be frequently invoked to pursue small information increments, which may slow down the overall detection cycle. By optimizing information gain and acquisition cost in a unified manner through a loss function, Step S54 can prioritize the selection of cost-effective modalities in most cases. However, when uncertainty is extremely high and the expected information gain is significantly greater than the cost difference, high-cost modalities can be invoked to prioritize accuracy. This trade-off mechanism makes the balance between accuracy and efficiency in the entire adaptive detection scheme more flexible and controllable.

[0079] In one or more embodiments of this application, step S6, which uses a cross-modal attention mechanism to dynamically fuse supplementary data with existing data, further includes: S61: Encode the supplementary data into supplementary features, linearly map the existing features corresponding to the existing data into query vectors, and linearly map the supplementary features into key vectors and value vectors; S62: Calculate the scaled dot product of the query vector and the key vector to obtain the attention weights, and use the weights to perform weighted aggregation on the value vectors to obtain enhanced supplementary features; S63: Perform residual connection and layer normalization between the enhanced supplementary features and the existing features, and replace the corresponding modal feature segments in the original defect response feature vector to form the updated defect response feature vector.

[0080] In this embodiment, the step S61 of encoding supplementary data into supplementary features refers to using the same or corresponding feature extraction network as in step S2 to perform multi-scale feature extraction on the supplementary data collected from the target modality, obtaining a feature representation that is aligned with the existing features in both spatial and channel dimensions. The existing features corresponding to the existing data refer to the set of modal features formed through previous collections and fusions before the current iteration. Linear mapping refers to a linear transformation of the input vector using a learnable weight matrix, typically implemented using matrix multiplication, and does not involve non-linear activation. The query vector is a vector representation obtained by linearly mapping the existing features, used to express what content is desired to be obtained from the supplementary data. The key vector is a vector representation obtained by linearly mapping the supplementary features, used to express what information the supplementary data can provide. The value vector is also a vector representation obtained by linearly mapping the supplementary features, used to carry the actual content to be extracted from the supplementary data. For example, if the dimension of the existing feature vector is d, the dimension of the query vector can be d_k, and the dimensions of the key vector and the value vector can be d_k and d_v respectively. The three sets of linear mappings each correspond to independent weight matrices, and the optimal mapping method can be learned through training.

[0081] The scaled dot product mentioned in step S62 refers to scaling the dot product similarity between the query vector and the key vector by dividing it by the square root of the key vector's dimension. This prevents the dot product value from becoming too large, which could cause the subsequent normalized exponential function (Softmax) to enter the saturation region. Attention weights are weight coefficients obtained by applying a normalized exponential function to the scaled dot product result. These coefficients range from 0 to 1 and sum to 1, reflecting the importance of each position or channel in the supplementary features to the response to the existing feature query. Weighted aggregation of the value vectors using these weights involves summing the value vectors using the attention weights as coefficients. This highlights value vector components highly relevant to the query while suppressing less relevant components. The enhanced supplementary features are the vectors obtained after weighted aggregation. They are not original copies of the supplementary features but rather information selectively extracted from the existing features based on their requirements.

[0082] The residual connection mentioned in step S63 refers to the operation of directly adding the enhanced supplementary features to the existing features at corresponding positions. The idea of ​​residual connections originated from Residual Networks (ResNet), which helps maintain smooth gradient propagation in deep networks and alleviates network degradation problems. In the context of this step, the role of residual connections is to allow confirmed valid information from existing features to be directly passed to the fusion result, while the enhanced supplementary features act as residual terms, superimposing incremental information extracted from the supplementary data on top of the existing information. Layer normalization is the operation of normalizing the mean and variance of the fused feature vector along its feature dimensions, which helps stabilize the feature distribution and accelerate subsequent inference. Replacing the corresponding modality feature segment in the original defect response feature vector means writing the fused and updated features into the corresponding modality position in the defect response feature vector, overwriting the original feature segment of that modality, while other unreplaced modality feature segments remain unchanged. The defect response feature vector can be divided into several feature segments according to modality, with each feature segment corresponding to a feature representation of a collected modality.

[0083] The steps S61 to S63 above together constitute a cross-modal attention fusion mechanism for supplementary data and existing features. The synergistic relationship between the steps is reflected in the three levels of information extraction, filtering and integration.

[0084] First, step S61 unifies the representation of the two types of information: encoding supplementary data into supplementary features, placing them in a comparable and interactive representation space with existing features; simultaneously, mapping existing features to query vectors, and supplementary features to key vectors and value vectors. These three sets of mappings endow the model with an asymmetric interactive capability—existing features can actively send queries to supplementary features, while supplementary features provide key-value pairs for query matching. This asymmetric design aligns with the actual logic of supplementary data collection: existing features are the current main body of information, supplementary data is the incremental information source, and fusion should be guided by the needs of existing features, extracting the parts useful for improving the current discrimination from the supplementary data.

[0085] Next, step S62 achieves selective extraction of supplementary features through the calculation and weighted aggregation of attention weights. Compared with simple feature concatenation or element-wise addition, the attention mechanism can adaptively determine which regions or channels in the supplementary features are worth paying attention to based on the similarity between the query vector and the key vector, thereby avoiding the mixing of redundant information or noise from the supplementary data into the existing features to the same extent. For example, if only a certain local area in the supplementary infrared thermal image data has a temperature anomaly related to the current suspected defect, the attention weight will produce a high response in that area. The enhanced supplementary features after weighted aggregation will focus on carrying the temperature anomaly information of that area, while information from other irrelevant background areas will be suppressed.

[0086] Step S63 integrates and stabilizes the new and old information through residual connections and layer normalization. The existence of residual connections ensures that even if the supplementary data does not bring significant incremental information, the fusion result will not be inferior to the existing features, essentially providing a safety net for information. When the supplementary data does contain useful information, the enhanced supplementary features are superimposed as residual terms, thus enhancing the information. Layer normalization regularizes the distribution of the fused features, helping to provide a numerically stable input for the evidence reasoning in the next round of step S3, enabling the multi-round iterative reasoning process to converge smoothly. Overall, this cross-modal attention fusion mechanism allows the defect response feature vector to systematically absorb useful information from the new modality after each supplementary acquisition, while maintaining the stable transmission of existing discriminative knowledge, supporting the entire detection loop to gradually approach a high-confidence state in multiple iterations.

[0087] In one or more embodiments of this application, step S7, updating the parameters of the modal mutual information matrix and the evidence depth defect classification model, further includes: S71: Store the modal selection sequence, confidence convergence path, and final defect category label of this test into the experience playback buffer; S72: When the number of samples in the experience replay buffer reaches the preset batch, incremental learning is performed on the evidence depth defect classification model using the samples in the buffer, and the conditional mutual information of modal pairs under different defect categories is re-statistically analyzed to update the modal mutual information matrix.

[0088] In this embodiment, the modality selection sequence mentioned in step S71 refers to the recording of the sequential acquisition of each modality and their order during the current complete detection process, starting from the initial modality group and continuing until the confidence threshold is met or the iteration limit is reached. For example, the modality selection sequence for a certain detection can be recorded as follows: first, visible light images and infrared thermal images are acquired as the initial modality group. Due to insufficient confidence, electroluminescent images are then selected as the target modality for the first supplementary acquisition. Still, the confidence threshold is not met, so laser speckle interferometry is selected as the target modality for the second supplementary acquisition, which finally meets the confidence requirement. The confidence convergence path refers to the trajectory of the classification confidence after each round of evidence reasoning during the progression of the above modality selection sequence. It can include the classification confidence value and cognitive uncertainty value after each iteration. For example, the classification confidence is 0.65 after the initial reasoning, rises to 0.78 after the first supplementary acquisition, and rises to 0.91 after the second supplementary acquisition. The final defect category label refers to the true defect category determined by the system output or subsequent manual review, such as confirming the sample as a Mura defect or confirming it as defect-free. The experience replay buffer is a data storage space used to temporarily store records of each detection. It can be implemented as a queue in memory, a circular buffer, or a table structure in persistent storage. Packing the modality selection sequence, confidence convergence path, and final defect category label of each detection into the experience replay buffer allows the decision-making process and final result of a single detection to be completely preserved, serving as training material for subsequent model and matrix updates.

[0089] The preset batch mentioned in step S72 refers to the threshold value of the number of samples in the experience replay buffer required to trigger incremental learning, such as 32, 64, or 128 records. The value of the preset batch can be adjusted according to the needs of production line operation speed, storage space, and model update frequency: a smaller value results in more frequent updates and a faster model response to process changes, but may introduce some fluctuations; a larger value results in longer update intervals and more stable model parameters, but may be slightly slower to adapt to new batches. Incremental learning refers to a training method that uses new samples to fine-tune or slightly update the parameters of an existing model without retraining the model from scratch. Incremental learning can be implemented using mini-batch stochastic gradient descent, online learning, or other continuous learning algorithms. Performing incremental learning on the evidence depth defect classification model means using the samples in the experience replay buffer as training data, starting from the current model parameters, and performing several rounds of iterative updates, allowing the model to gradually absorb the defect feature distribution and modal combination patterns in the new samples. Recalculating the conditional mutual information of modal pairs under different defect categories refers to recalculating the conditional mutual information values ​​of each modal pair under each defect category using newly added sample data from the experience replay buffer, combined with existing historical data, and updating the corresponding elements in the modal mutual information matrix accordingly. It can be understood that the recalculation can be performed by calculating using only the newly added samples and then weighting and fusing them with the original matrix, or by incorporating the newly added samples into the historical dataset and then recalculating the entire dataset. The specific method can be flexibly chosen based on the amount of data and computational resources.

[0090] The steps S71 and S72 described above together constitute a closed-loop update path for the detection system from single-time experience accumulation to global model optimization, with a clear temporal sequence and functional division between the two.

[0091] Step S71 plays a crucial role in experience acquisition and structured storage throughout the closed-loop update process. Compared to simply recording the final classification result, step S71 simultaneously saves the modality selection sequence and the confidence convergence path. These two pieces of information have unique value: the modality selection sequence records the decision-making path made by the system under uncertainty, reflecting the order in which the system tends to invoke modalities under the guidance of the current modality mutual information matrix; the confidence convergence path records the actual confidence improvement brought about by each supplementary acquisition, which can serve as post-hoc feedback for evaluating the actual information contribution of each modality. Linking and storing the decision path, confidence trajectory, and final true defect category provides structured learning samples for step S72.

[0092] Step S72 is triggered after the preset batch of samples is accumulated in step S71, completing two update tasks. First, these samples, containing decision paths and true labels, are used to incrementally learn the evidence-depth defect classification model, enabling the model to access the actual sample distribution on the production line, including defect morphologies or batch characteristics that may not be fully covered by the initial training set. This gradually reduces the frequent exceedances of cognitive uncertainty caused by data distribution drift. Second, the conditional mutual information of modal pairs under different defect categories is recalculated, allowing the mutual information matrix to be corrected using new samples. For example, if new samples show that the conditional mutual information between features from laser speckle interferometry and electroluminescence features is significantly higher than historical records when identifying a certain type of Mura defect, the updated mutual information matrix will correspondingly increase the mutual information value of that modal pair under the corresponding defect category. This makes it more likely that the system will prioritize this modal pair for supplementary acquisition when encountering similar uncertainties in subsequent detections. This continuous updating based on actual operational feedback allows the modal mutual information matrix and classification model to dynamically evolve in response to changes in production line process status and defect distribution. The decision-making quality of the entire detection system is expected to gradually improve with the accumulation of operating time.

[0093] In one or more embodiments of this application, before step S1, the method further includes: S01: Collect prior information of the current batch of display devices under test, including process parameters and historical yield. S02: Based on prior information, select at least two modes associated with the most frequently occurring defect type from the modality library to form the first modality group.

[0094] In this embodiment, the prior information mentioned in step S01 refers to background information related to the quality of the current batch of display devices that can be obtained before the current batch of devices formally enters the testing process. Process parameters may include key process parameters used in the production of this batch of devices, such as photolithography exposure time, etching rate, annealing temperature profile, film deposition thickness, and doping concentration. They may also include traceability information such as production equipment number, production line number, and production shift. Historical yield refers to the statistical results of defect detection for the same or similar model of display devices produced on the same production line within a similar time period. This may include overall yield, the frequency or proportion of each defect type, and the distribution trend of defect types. For example, if the proportion of Mura defects in the previous batch of the same model of products increases significantly, this information can be included in the prior information for reference in the current batch. The sources of prior information may include Manufacturing Execution System (MES) databases, process data management systems, yield analysis platforms, etc., and can be obtained through production line network interfaces or data buses. It is understandable that the richness and accuracy of prior information have a direct impact on the targeting of subsequent initial mode selection. In actual deployment, the available dimensions can be selected based on the level of information technology in the production line.

[0095] The modality library mentioned in step S02 refers to the collection of all available sensing modalities configured in the system. Each modality, in addition to recording its technical parameters and acquisition interface, also maintains data on the correlation between that modality and various defect types. This correlation can be obtained based on statistical analysis of historical detection data. For example, it can be used to statistically analyze the detection contribution or characteristic response intensity of each modality to a confirmed defect type in samples, thereby establishing a mapping table or correlation matrix from defect types to recommended modalities. The defect type with the highest frequency of occurrence refers to the defect category that occurs most frequently in the current batch or similar batches, based on historical yield data collected in step S01. The at least two associated modalities refer to the modalities with a high correlation to the defect type with the highest frequency of occurrence, retrieved from the modality library, and at least two selected to form the first modality group. When selecting modalities, the detection rate of each modality individually or in combination for that defect type in historical data can be referenced; modalities with higher detection rates can be prioritized. The first modality group is the set of modalities that are synchronously collected in the first batch in step S1. Its determination is supported by the prior information in step S01 and the defect-modal association query in step S02.

[0096] The above steps S01 and S02 introduce an initial modality configuration mechanism based on prior information before step S1. The two work together to complete the entire link from pre-information collection to the formulation of the first round of collection strategy.

[0097] Step S01, as a pre-processing step in the entire inspection process, aims to aggregate the scattered process data and yield statistics from the production line into structured prior knowledge that the inspection system can utilize. Without prior information, the initial mode selection often relies on fixed default configurations or random selection, making it difficult to differentiate based on the process characteristics and defect tendencies of different batches. By actively collecting the process parameters and historical yield of the current batch, step S01 allows the inspection system to gain a preliminary understanding of the quality profile of the batch of devices before formal inspection begins, providing data for subsequent targeted mode selection.

[0098] Step S02, based on the prior information provided in Step S01, transforms batch-level statistical characteristics into specific modal scheduling decisions. The logic is as follows: the differences in physical characteristics of different defect types determine their varying detectability under different modalities. If a batch experiences frequent occurrences of a certain type of defect due to process fluctuations, the modality with strong discriminative ability for that type of defect is prioritized in the first round of detection. This helps increase the probability of reaching the confidence threshold in the first round, thereby reducing the frequent triggering of supplementary acquisitions due to improper initial modality selection. At least two modalities associated with the most frequently occurring defect type are grouped into a first modality group. This considers both the matching degree between modality and defect and preserves the space for multimodal information complementarity, achieving a balance between efficiency and information richness in the first round of acquisition.

[0099] Overall, steps S01 and S02 connect production line-level production data with unit-level inspection decisions, extending the adaptive capability of the inspection system from the device level to the batch level. This preliminary step works in conjunction with the closed-loop reasoning mechanism formed by subsequent steps S3 to S7: the initial mode selection driven by prior information tends to allow inspection to start from a better information starting point, reducing unnecessary iterations; and when there is a deviation between prior information and the actual situation, or the initial selection fails to cover the actual defect types, the subsequent uncertainty-driven closed loop can still be corrected and remedied through supplementary data collection. The two together form a composite adaptive strategy that combines batch-level prior guidance with device-level real-time feedback.

[0100] In one or more embodiments of this application, step S4 further includes: acquiring the current production cycle signal, dynamically adjusting the preset confidence threshold and the upper limit of the number of iterations based on the production cycle signal; when the production cycle speeds up, increasing the confidence threshold to reduce the time consumed by single-piece inspection, and simultaneously activating the entropy-based information density evaluation mechanism to prioritize selecting the target mode with the largest information gain per unit time from the candidate modes; when the number of iterations in steps S3 to S6 reaches the adjusted upper limit of the number of iterations but still does not meet the confidence threshold, terminating the supplementary acquisition, outputting the defect category with the highest current confidence and marking it as the cycle constraint degradation result; and feeding back the time consumed by this inspection and the final confidence to the production line scheduling system to update the cycle-threshold mapping relationship.

[0101] In this embodiment, the production cycle signal mentioned in step S4 refers to a control signal or status indication issued by the production line scheduling system or manufacturing execution system, reflecting the current production line operating rhythm. The production cycle signal may include the target production cycle of the current production line, the allowed occupancy time of the current inspection station, or an identifier of the production line operating mode, such as normal mode, acceleration mode, or deceleration mode. It is understood that the production cycle signal can be acquired in real time through a production line network bus, digital input / output interface, or Ethernet communication. The preset confidence threshold and iteration limit refer to the confidence threshold value used in step S4 to determine whether the classification result is acceptable, and the maximum allowed number of execution rounds in steps S3 to S6. Dynamic adjustment means that these two parameters are no longer fixed but are adaptively modified according to the production line rhythm indicated by the production cycle signal. For example, when the production line is operating at a normal cycle time, the confidence threshold can be maintained at its default value, such as 0.90, and the maximum number of iterations can be set to 3. When the production line cycle time is accelerated and the available time for single-piece inspection is shortened, the confidence threshold can be appropriately increased to 0.88 or 0.85, while the maximum number of iterations can be reduced to 2 or 1, so as to complete the inspection decision within a limited time. The entropy-based information density evaluation mechanism refers to dividing the expected information gain of each candidate modality by its estimated acquisition time to obtain the expected information increment obtainable per unit time, and using this as the ranking criterion to prioritize the target modality with the largest information gain per unit time. Entropy here refers to the information quantity measurement in the sense of information theory, which is intrinsically related to the cognitive uncertainty in step S3. The cycle time constraint degradation result refers to the detection result output when the supplementary acquisition is terminated early due to the number of iterations reaching the upper limit after the cycle time adjustment. Although this result does not meet the original strict confidence requirement, it carries the corresponding label for downstream processes or quality management systems to identify and differentiate. The time taken and the final confidence level of this test are fed back to the production line scheduling system to update the cycle time-threshold mapping relationship. This means that the system will send back the actual performance data of each test, so that the production line scheduling system can accumulate the statistical regularity of the test confidence level under different cycle time conditions, thereby providing a reference for optimizing the cycle time-threshold mapping relationship of subsequent batches or similar devices.

[0102] The above technical solution introduces multiple functional features such as production line cycle time perception, dynamic parameter adjustment, modality selection strategy switching, degradation output marking, and feedback closed loop on the basis of the original confidence judgment in step S4. These features cooperate with each other around the main line of dynamic balance between detection accuracy and production line cycle time.

[0103] First, by acquiring production cycle time signals and dynamically adjusting the confidence threshold and iteration limit accordingly, the inspection system gains the ability to perceive and respond to the external production line rhythm. In traditional fixed-threshold schemes, the inspection system always pursues the same accuracy standard regardless of changes in the production line rhythm. This can lead to some devices exceeding the time limit due to the inability to complete the acquisition iteration in time during periods of high cycle time, resulting in production line shutdowns or forced removal from the line. By incorporating the cycle time signal into the decision variables, the inspection system can proactively and appropriately relax the confidence requirements for individual device inspections when the cycle time accelerates, in exchange for reducing inspection time and ensuring the smooth operation of the entire production line. This flexible decision-making mechanism, guided by the overall efficiency of the production line, helps the inspection system better integrate into the actual production environment.

[0104] Secondly, activating the entropy-based information density evaluation mechanism when the cycle time is accelerated further refines the mode selection strategy. Under normal cycle time, step S5 can select the target mode by comprehensively considering the expected information gain and the acquisition cost, which may include long-term cost factors such as equipment wear and tear and energy consumption. However, within the time window of a tight cycle time, the efficiency of information acquisition per unit time becomes the primary consideration. Therefore, the selection criterion shifts from comprehensive cost to information density, prioritizing the mode that can bring the maximum information gain in the shortest time. This strategy switching helps to maximize the value of supplementary acquisition within a limited time and improve the confidence level under degraded output conditions as much as possible.

[0105] When the iteration reaches the adjusted upper limit but still fails to meet the confidence threshold, the system terminates supplementary data collection and outputs a result with a downgraded flag indicating a cycle time constraint, rather than waiting indefinitely or halting the line with an error. This is a deterministic safety exit mechanism. The downgrade flag allows the test result to be treated differently in downstream processes, such as scheduling manual re-inspection, reducing shipment priority, or incorporating it into statistical quality analysis, thus creating a controllable trade-off between production line efficiency and quality risk. Finally, the time taken and confidence level of this test are fed back to the production line scheduling system, enabling the scheduling system to gradually optimize the cycle time-threshold mapping relationship in a data-driven manner. The decision-making quality of the entire test-scheduling joint system is expected to continuously improve with the accumulation of operational data.

[0106] In one or more embodiments of this application, the evidence depth defect classification model is a multi-task model, which also includes a defect segmentation branch. In step S3, the model simultaneously outputs a defect segmentation prediction map and the prediction entropy of each pixel in the defect segmentation prediction map. Regions with prediction entropy higher than a preset entropy threshold are marked as segmentation uncertainty regions, and spatial uncertainty masks are generated based on the segmentation uncertainty regions. In the cross-modal attention fusion in step S6, the spatial uncertainty mask is used as an attention bias term and added to the attention weight to enhance feature aggregation of the segmentation uncertainty regions. At the same time, the structural uncertainty coefficient is calculated using the area of ​​the segmentation uncertainty region and the average prediction entropy, and it is fed back as additional evidence quantity to the evidence distribution calculation in step S3 to improve the cognitive uncertainty estimation accuracy of the corresponding defect category.

[0107] In this embodiment, the evidence depth defect classification model is a multi-task model. This means that the model, while sharing a portion of the feature extraction network, simultaneously connects two output branches: one for defect classification and the other for defect segmentation. These two branches are jointly optimized during training and output their respective results synchronously during inference. The defect segmentation branch is the network branch used for pixel-level classification of the input image, outputting a defect segmentation prediction map. Each pixel in this map corresponds to a predicted value, representing the probability distribution of whether the pixel belongs to a certain type of defect or background. Prediction entropy is the information entropy value calculated based on the class probability distribution of each pixel in the defect segmentation prediction map, used to measure the uncertainty of the model's classification of that pixel. For example, if the predicted probability of a pixel between two classes is close to 0.5, its prediction entropy is high, indicating that the model is hesitant about classifying the pixel; if the predicted probability is close to 0.99 and 0.01, its prediction entropy is low, indicating that the model's classification of the pixel is relatively certain. A preset entropy threshold is a threshold value used to distinguish whether pixel-level predictions are certain; pixels exceeding this threshold are considered to indicate that the model's segmentation judgment is uncertain. The segmentation uncertainty region refers to the connected or disconnected region formed by all pixels in the defect segmentation prediction map whose predicted entropy is higher than a preset entropy threshold. These regions are typically locations where defect boundaries are blurred, defect morphology is abnormal, or multiple types of defects overlap. The spatial uncertainty mask is a two-dimensional weight map with the same spatial dimensions as the input image. Pixels located within the segmentation uncertainty region are assigned higher weight values, while other regions are assigned lower weights or zero values. This mask reflects which spatial locations require more attention from supplementary modalities.

[0108] The attention bias term refers to an additional bias value added to the scaled dot product result when calculating attention weights in step S62. Using the spatial uncertainty mask as the attention bias term means incorporating prior uncertainty about spatial location into the original attention weight calculation logic. This allows spatial locations within the segmentation uncertainty region to receive a higher initial score in the attention calculation, thus making them more likely to be assigned a larger attention weight in subsequent weighted aggregation. The structural uncertainty coefficient is a scalar value calculated based on the area of ​​the segmentation uncertainty region and the average prediction entropy within the region. It is used to quantify the degree of uncertainty of the current sample at the structural level from a spatial perspective. For example, the structural uncertainty coefficient can be the product of the proportion of the segmentation uncertainty region area to the overall image area and the average prediction entropy within the region, or it can be another combination function combining area and entropy. The additional evidence quantity refers to the interaction between the structural uncertainty coefficient as an additional input or correction term and the original evidence values ​​during the evidence distribution calculation process in step S3. For example, the structural uncertainty coefficient can be used as a virtual additional evidence component and assigned to the defect categories related to the segmentation uncertainty region according to certain rules, thereby improving the cognitive uncertainty estimate corresponding to these categories and enabling the uncertainty assessment at the classification level to reflect the fuzziness at the spatial segmentation level.

[0109] The above technical solution uses a multi-task model architecture to perform defect classification and defect segmentation tasks collaboratively on the basis of feature sharing. It also uses the spatial uncertainty information output by the segmentation branch to enhance the uncertainty perception at the classification level and the pertinence of cross-modal fusion, thus forming a collaborative mechanism for bidirectional information interaction between classification and segmentation.

[0110] First, the design of the multi-task model sharing the feature extraction network allows the classification and segmentation branches to mutually reinforce each other during training: the segmentation task requires the model to learn the precise boundaries of defects, which helps the feature extraction network gain more refined spatial discrimination capabilities—capabilities that are also beneficial for the classification task in distinguishing morphologically similar defect categories; the classification task requires the model to understand the global semantics of defects, which helps the feature extraction network gain more abstract semantic discrimination capabilities—capabilities that also aid in contextual reasoning when handling locally ambiguous regions in the segmentation task. During the inference phase, the two branches output results synchronously, providing temporal alignment and semantic association for subsequent collaborative processing.

[0111] Secondly, the predicted entropy output by the segmentation branch and the spatial uncertainty mask generated accordingly provide clear spatial focus guidance for the cross-modal attention fusion in step S6. Without the guidance of the spatial mask, the attention mechanism only assigns weights based on query-key similarity at the feature level, which may evenly distribute attention between uncertain and non-uncertain regions, resulting in valuable supplementary modality information not being focused on further exploration of uncertain regions. By using the spatial uncertainty mask as an attention bias term, the data processing of the supplementary modality can be guided to consciously focus on regions that the segmentation model itself is uncertain about, allowing the information gain from the supplementary acquisition to be more effectively allocated and utilized in the spatial dimension.

[0112] Meanwhile, the structural uncertainty coefficient, as additional evidence, is fed back to the evidence distribution calculation in step S3, realizing the transfer of uncertainty information from the segmentation level to the classification level. In single-task classification models, the estimation of cognitive uncertainty relies solely on the evidence distribution of the classification branch itself, which may not fully perceive the discrimination difficulties caused by factors such as irregular spatial morphology and blurred boundaries of defects. After introducing the structural uncertainty coefficient, the cognitive uncertainty estimation of the classification branch can integrate spatial uncertainty signals. When there are large-area or high-entropy segmentation uncertainty regions in the image, even if the evidence distribution of the classification branch itself is acceptable, the cognitive uncertainty will be moderately increased due to the addition of the structural uncertainty coefficient, thus more accurately reflecting the true uncertainty state of the model and providing a more reliable basis for judging whether supplementary data collection is needed. This bidirectional collaboration between classification and segmentation improves the entire multimodal detection system in both uncertainty perception and fusion efficiency.

[0113] In one or more embodiments of this application, multimodal data is acquired by a sensing module array in conjunction with a tunable illumination unit and a switchable polarization filter component; in step S5, while selecting the target mode, the optimal illumination wavelength, polarization angle and exposure parameters corresponding to the target mode are also determined; in step S6, when the target mode is triggered to acquire supplementary data, the tunable illumination unit is switched to the optimal illumination wavelength, and the switchable polarization filter component is adjusted to the polarization angle to maximize the contrast between the specific defect type and the background in the supplementary acquired image.

[0114] In this embodiment, the tunable illumination unit refers to a light source device capable of dynamically selecting and switching the wavelength or spectral composition of the emitted light. It can be implemented as an array of multiple narrowband light-emitting diodes (LEDs), with different wavelengths selected by an electronic switch; it can also be a broadband light source combined with an acousto-optic tunable filter or a liquid crystal tunable filter, selecting the center wavelength of transmission by changing the driving signal; or it can be a supercontinuum light source combined with a grating beam splitter. For example, the tunable illumination unit can output wavelengths covering the ultraviolet, visible, and near-infrared bands, with wavelength switching step accuracy reaching the nanometer level. The switchable polarization filter component refers to a device installed in the light-receiving optical path of the sensing module that can dynamically change the polarization state of the incident light. It can be implemented as an electrically driven rotating polarizer, with a stepper motor controlling the rotation angle of the polarizer to select the polarization direction of the transmission; or it can be a liquid crystal polarization modulator, adjusting the orientation of the liquid crystal molecules by changing the applied voltage, thereby controlling the polarization state of the transmitted light, possessing the advantages of no mechanical moving parts and fast response speed. The sensing module array refers to the imaging or sensing units for each modality in the system, such as visible light cameras, infrared thermal imagers, and electroluminescent imaging devices. The optimal illumination wavelength refers to the wavelength of illumination light that, under the current target modality and for defect types with high cognitive uncertainty in the current detection context, maximizes the contrast between the defect area and the surrounding background in the acquired image. For example, for microcrack defects under a certain type of transparent film layer in a display device, near-infrared illumination may yield a clearer crack image than visible light due to its better penetration. The polarization angle refers to the angle between the transmission polarization direction set by the switchable polarization filter assembly and the reference direction. By adjusting the polarization angle, reflected glare from specific directions or enhanced polarization state change responses caused by defects can be selectively filtered out. Exposure parameters may include the integration time, gain value, and aperture size of the imaging sensor, used to adjust the brightness and signal-to-noise ratio of the image in conjunction with the illumination conditions.

[0115] The above technical solution introduces active optical control capability at the sensing hardware level and links the optimization of optical parameters with modal selection decision-making. The synergistic relationship between various features is reflected in the closed-loop process of decision-making, control and acquisition.

[0116] First, step S5 determines the optimal illumination wavelength, polarization angle, and exposure parameters simultaneously with selecting the target mode. This means that the decision-making process for mode selection has expanded from choosing which sensing mode to select to configuring the appropriate optical acquisition conditions for that mode. The rationale behind this expansion is that the same sensing mode can exhibit significantly different imaging effects for different defect types under different illumination and polarization conditions. For example, a visible light camera may struggle to distinguish edge defects on a transparent electrode under ordinary white light illumination, but the contrast of that defect may be significantly enhanced under specific wavelength narrowband illumination and orthogonal polarization conditions. Therefore, simply selecting a mode without configuring its optical parameters may not fully realize its detection potential.

[0117] Secondly, in step S6, when triggering the acquisition of supplementary data for the target mode, the illumination wavelength switching and polarization angle adjustment are performed simultaneously, ensuring the timing consistency between the optical configuration and the acquisition action. The coordinated operation of the tunable illumination unit and the switchable polarization filter component enables a single supplementary acquisition to obtain an image with an optimized specific spectral-polarization combination. This active optical control improves signal quality from the physical source of data generation, allowing subsequent feature extraction and evidence reasoning to be based on the original data with a higher signal-to-noise ratio and higher defect-background contrast. Compared to methods that rely solely on post-processing image algorithms to enhance contrast, this approach offers a more direct and stable effect.

[0118] Overall, this technical solution extends the concept of adaptive detection from the signal processing level to the physical acquisition level. The tunable optical configuration at the front end and the modal mutual information matrix decision-making at the back end work together: the modal mutual information matrix guides the selection of which mode to use from an information theory perspective, while the optimal optical parameters guide the conditions under which that mode's acquisition is performed from a physical optics perspective. Both work together to improve the quality of the supplementary data. This linkage between decision-making, control, and acquisition helps to obtain more discriminative information within a limited number of supplementary acquisitions, thereby improving the efficiency and confidence convergence speed of the entire closed-loop detection process.

[0119] In one or more embodiments of this application, the test pattern sent to the display device under test in step S1 or step S6 is generated in the following manner: obtaining the feature activation map or defect segmentation prediction map corresponding to the currently acquired data, locating the local area of ​​the suspected defect through gradient weighted class activation mapping; dynamically generating a local test pattern that matches the texture characteristics of the local area based on the main direction of the image gradient and the spatial frequency distribution in the local area, the local test pattern including sinusoidal stripes or checkerboard patterns with adaptive direction, frequency and contrast; and driving the local test pattern only to the corresponding pixels of the local area, controlling the corresponding modality to perform high-resolution scanning acquisition of the local area as part of the initial multimodal data or supplementary data.

[0120] In this embodiment, the test pattern sent to the display device under test in step S1 or S6 refers to the pattern signal of the sensing mode for image acquisition when the display device under test displays a specific image through a driving circuit, so that the sensing mode is in an active light-emitting or display state. A feature activation map refers to the weighted combination of feature maps of a certain intermediate layer or each channel feature map during forward propagation of input data in a feature extraction network, reflecting the network's response intensity to different spatial locations in the input data. For example, a feature activation map can be obtained by equally weighting or weighted summing the feature maps of each channel output from the last convolutional layer, or it can be generated using gradient-weighted class activation mapping (Grad-CAM). The defect segmentation prediction map is the pixel-level defect prediction result output by the segmentation branch in the multi-task model. Gradient-weighted class activation mapping (Grad-CAM) is a visualization and localization technique that uses the gradient information of the classification or segmentation output to the target layer, and weights and sums the feature maps of each channel of that layer to obtain a heatmap of spatial importance. In this step, gradient-weighted class activation mapping is used to locate suspected defect local areas with high current model attention or high prediction uncertainty from feature activation maps or defect segmentation prediction maps, providing spatial coordinate references for the subsequent generation of local test patterns.

[0121] A local region refers to a rectangular, elliptical, or arbitrarily shaped sub-region of an image containing suspected defects, located by gradient-weighted activation mapping. Its location can be determined by peak coordinates or threshold segmentation from a heatmap. The principal direction of the image gradient refers to the direction in which the pixel grayscale values ​​change most drastically within the local region. It can be obtained by calculating the gradient structure tensor and its eigenvector, reflecting the directional characteristics of the texture within the region. Spatial frequency distribution refers to the spectral characteristics of the local region in the spatial frequency domain. It can be obtained by performing a Fourier transform or discrete cosine transform on the local region, including the frequency values ​​and directional distribution of the principal frequency components, reflecting the coarseness and periodicity of the texture within the region. Dynamic generation means that the test pattern is not selected from a pre-set fixed pattern library, but is calculated and synthesized in real time based on the specific texture characteristics of the current local region. A sinusoidal fringe with adaptive direction, frequency, and contrast refers to a sinusoidal intensity distribution pattern whose fringe direction forms a specific angle with the principal direction of the local region's gradient, whose fringe spatial frequency matches the spatial frequency distribution of the local region, and whose fringe brightness and contrast are dynamically adjusted according to the signal-to-noise ratio of the local region. A checkerboard pattern refers to a pattern composed of alternating light and dark rectangular blocks, the size and orientation of which can be adjusted according to the texture characteristics of the local area. Local test patterns only drive the corresponding pixels in the local area, meaning that only the suspected defect area and its adjacent display pixels are subjected to the test pattern signal. The remaining display areas can remain normally bright, normally dark, or have a neutral gray background, thus focusing the attention and resolution of the perceptual modality acquisition on the suspected defect area. High-resolution scanning acquisition refers to acquiring images of local areas using a sampling density higher than the conventional detection resolution. This can be achieved, for example, by adjusting the camera's region of interest setting or optical zoom, so that the local area occupies more pixels in the acquired image to obtain richer detail information.

[0122] The above technical solution upgrades the generation of test patterns from static preset to adaptive generation based on real-time perception feedback, and achieves spatial focusing linkage with modal acquisition. The synergistic relationship between the features is reflected in the closed-loop connection of the three links of feedback perception, pattern generation and region focusing.

[0123] First, by utilizing the feature activation maps or defect segmentation prediction maps from the collected data, gradient-weighted class activation mapping is used to locate local areas suspected of defects, thus completing the transfer from macroscopic data to microscopic points of interest. In traditional methods, the test pattern is usually a uniform solid color, standard stripes, or checkerboard pattern covering the entire screen. Regardless of whether defects exist in different areas of the screen, all areas are subjected to the same excitation signal. This approach is insufficient for exciting local, small defects, or defects at the detection boundary, easily causing the defect signal to be submerged in the background. By extracting heatmaps of model interest from the collected data, local areas requiring further investigation can be automatically identified without manual intervention, allowing subsequent detection resources to be allocated to these areas.

[0124] Secondly, based on the principal direction and spatial frequency distribution of the image gradient within a local area, a local test pattern matching the texture characteristics of the area is dynamically generated, demonstrating targeted excitation of the physical properties of defects. Different defect types exhibit different response characteristics on the displayed image: for example, line defects extending along a certain direction are more easily exposed under stripe excitation orthogonal to that direction, while periodic non-uniform defects may produce more significant visual differences under stripe excitation with a spatial frequency close to that of the defect. By generating adaptive test patterns by matching local texture characteristics, specific defects can generate stronger response signals in the acquired image, helping to separate weak defect signals from background noise and improving the ability of subsequent feature extraction and evidence reasoning to discriminate defects.

[0125] By driving the local test pattern only to the corresponding pixels in a specific area and performing high-resolution scanning acquisition of that area in conjunction with the corresponding modality, precise coordination between display driving and perception acquisition in the spatial dimension is achieved. Local driving means that the energy of the test stimulus is concentrated in the suspected defect area, avoiding interference with normal areas, and also reducing the adverse effects of power consumption and thermal effects caused by full-screen driving on detection. High-resolution scanning acquisition improves the information acquisition density of the local area from the perception end, allowing the supplementary data to contain richer defect details. This spatially focused acquisition strategy echoes the spatial attention mechanism of cross-modal attention fusion in step S6, enhancing the actual contribution of supplementary acquisition to reducing cognitive uncertainty from the data source. Overall, this scheme uses the perception information acquired in real time during the detection process to react on the tested object itself, forming an adaptive detection closed loop of device response-system perception-stimulus optimization, which helps to improve the detection capability of local weak and complex defects.

[0126] A second aspect of this application provides a display device intelligent detection system based on multimodal sensing, comprising: A multimodal sensing module array, including at least two of a visible light camera, an infrared thermal imager, an electroluminescent imaging device, and a laser speckle interferometer, is used to acquire multimodal data of the display device under test; The synchronous triggering and driving module is used to generate test pattern driving signals and generate synchronous trigger pulses to enable the multimodal sensing module array to be exposed synchronously. The feature extraction and fusion module is used to perform multi-scale feature extraction, defect response feature vector construction, and cross-modal attention fusion in the above methods. The evidence reasoning and decision-making module stores a deep evidence defect classification model, which is used to output the evidence distribution based on the defect response feature vector, calculate the classification confidence and cognitive uncertainty, and determine whether the confidence threshold has been reached. The modality selection and scheduling module stores a modality mutual information matrix, which is used to select the target modality from the candidate modalities when the cognitive uncertainty exceeds the limit, and send a supplementary acquisition command to the corresponding sensing unit. The online learning update module is used to update the modal mutual information matrix and the parameters of the evidence depth defect classification model based on the modality selection sequence, confidence convergence path and final defect category in the detection process.

[0127] In this embodiment, the detection system integrates multimodal sensing hardware, synchronous trigger control, feature extraction and fusion, evidence reasoning and decision-making, online modality scheduling, and continuous learning and updating into a closed-loop whole. Each module collaborates to complete the entire chain from data acquisition, uncertainty assessment, on-demand scheduling to model evolution, which helps to stably implement a multimodal adaptive detection process in a production line environment, balancing defect identification accuracy and detection cycle time.

[0128] A third aspect of this application provides an electronic device, including: one or more processors, one or more input devices, one or more output devices, and one or more memories. The processors, input devices, output devices, and memories communicate with each other via a communication bus. The memories store computer programs, including program instructions. The processors execute the program instructions stored in the memories. The processors are configured to invoke the program instructions to execute the aforementioned intelligent detection method for display devices based on multimodal perception.

[0129] It should be understood that in one or more embodiments of this application, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0130] Input devices may include touchpads, fingerprint sensors (for collecting the user's fingerprint information and fingerprint orientation information), microphones, etc., while output devices may include displays (LCDs, etc.), speakers, etc.

[0131] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0132] In specific implementations, the processor, input device, and output device described in one or more embodiments of this application may execute the implementation method described in any embodiment of the intelligent detection method for display devices based on multimodal perception provided in the embodiments of this application, or may execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0133] In another embodiment of this application, an electronic device is provided. The electronic device stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the above-described intelligent detection method for display devices based on multimodal perception. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in an electronic device, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0134] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A smart detection method for display devices based on multimodal sensing, characterized in that, Includes the following steps: S1: Based on the type and batch information of the display device under test, select at least two sensing modes as the first mode group, and control them to synchronously collect initial multimodal data, the initial multimodal data including spatial images and physical field distribution; S2: Perform multi-scale decomposition on the initial multimodal data, extract local texture features and global frequency features, and concatenate them into a defect response feature vector; S3: Input the defect response feature vector into the evidence depth defect classification model, output the evidence distribution of each defect category, and calculate the classification confidence and cognitive uncertainty; S4: Determine whether the classification confidence level exceeds the preset confidence threshold. If it does, proceed to S7; otherwise, determine that the cognitive uncertainty exceeds the limit, activate the supplementary modality selection and proceed to S5. S5: Based on the current evidence distribution and the pre-calibrated modal mutual information matrix, select the target modality that can maximize the reduction of cognitive uncertainty from the unselected candidate modalities; S6: Trigger the acquisition of supplementary data for the target modality, dynamically fuse the supplementary data with the existing data using a cross-modal attention mechanism, update the defect response feature vector, and return to execute S3; S7: Output the defect classification results that meet the confidence requirements, and update the modal mutual information matrix and the parameters of the evidence depth defect classification model based on the modality selection sequence and confidence convergence path of this detection.

2. The method according to claim 1, wherein, Step S3 further includes: S31: Input the defect response feature vector into the evidence depth defect classification model. The model outputs the original evidence value of each preset defect category through a fully connected layer and obtains the non-negative evidence value using a non-negative activation function. S32: Based on the non-negative evidence values, construct the lumped parameters of the Dirichlet distribution and generate a subjective opinion on the defect category accordingly; S33: Based on the subjective opinion, calculate the quality of belief and the quality of uncertainty, take the maximum value of the quality of belief as the classification confidence, and take the quality of uncertainty as the cognitive uncertainty.

3. The method according to claim 1, wherein, Step S2 further includes: S21: For each modality of the initial multimodal data, local texture features are extracted in parallel using convolutional kernels of different scales, and global frequency features are extracted through wavelet transform to form a multi-scale feature map; S22: Concatenate the multi-scale feature maps of each modality along the channel dimension and input them into the channel attention module to obtain the weighted multi-modal fusion features; S23: Flatten the multimodal fusion features into the defect response feature vector.

4. The method according to claim 1, wherein, The modal mutual information matrix mentioned in step S5 is pre-calibrated in the following way: S51: Collect the defect category labels and corresponding modal response data for each inspection under different modal combinations during historical inspection processes; S52: For any two modal pairs, calculate their conditional mutual information under different defect categories to characterize the amount of shared information between the two modal features when given a defect category, and construct the modal mutual information matrix accordingly.

5. The method according to claim 4, wherein, Step S5, which involves selecting the target modality that maximizes the reduction of cognitive uncertainty, further includes: S53: Determine the cognitive uncertainty contribution of each defect category based on the current evidence distribution, and calculate the expected information gain of each candidate mode in the candidate mode set for reducing the current cognitive uncertainty in combination with the modal mutual information matrix; S54: Obtain the acquisition cost weight for each candidate mode, and select the optimal candidate mode as the target mode by minimizing the loss function that includes the expected information gain and acquisition cost.

6. The method according to claim 1, wherein, Step S6, which describes dynamically fusing supplementary data with existing data using a cross-modal attention mechanism, further includes: S61: Encode the supplementary data into supplementary features, linearly map the existing features corresponding to the existing data into query vectors, and linearly map the supplementary features into key vectors and value vectors; S62: Calculate the scaled dot product of the query vector and the key vector to obtain the attention weights, and use the weights to perform weighted aggregation on the value vectors to obtain enhanced supplementary features; S63: Perform residual connection and layer normalization between the enhanced supplementary features and the existing features to replace the corresponding modal feature segments in the original defect response feature vector, forming an updated defect response feature vector.

7. The method according to claim 1, wherein, Step S7, which updates the modal mutual information matrix and the parameters of the evidence depth defect classification model, further includes: S71: Store the modal selection sequence, confidence convergence path, and final defect category label of this test into the experience playback buffer; S72: When the number of samples in the experience playback buffer reaches a preset batch, incremental learning is performed on the evidence depth defect classification model using the samples in the buffer, and the conditional mutual information of modal pairs under different defect categories is re-statistically analyzed to update the modal mutual information matrix.

8. The method according to claim 1, wherein, Before step S1, the following is also included: S01: Collect prior information of the current batch of display devices under test, including process parameters and historical yield; S02: Based on the prior information, select at least two modes associated with the most frequently occurring defect type from the mode library to form the first mode group.

9. The method according to claim 1, wherein, Step S4 further includes: acquiring the current production cycle signal, dynamically adjusting the preset confidence threshold and the upper limit of the number of iterations based on the production cycle signal; when the production cycle speeds up, increasing the confidence threshold to reduce the time consumed by single-piece inspection, and simultaneously activating the entropy-based information density evaluation mechanism to prioritize selecting the target mode with the largest information gain per unit time from the candidate modes; when the number of iterations in steps S3 to S6 reaches the adjusted upper limit of the number of iterations but still does not meet the confidence threshold, terminating the supplementary acquisition, outputting the defect category with the highest current confidence and marking it as the cycle constraint degradation result; and feeding back the time consumed by this inspection and the final confidence to the production line scheduling system to update the cycle-threshold mapping relationship.

10. A display device intelligent detection system based on multimodal sensing, characterized in that, include: A multimodal sensing module array, including at least two of a visible light camera, an infrared thermal imager, an electroluminescent imaging device, and a laser speckle interferometer, is used to acquire multimodal data of the display device under test; The synchronous triggering and driving module is used to generate a test pattern driving signal and generate a synchronous trigger pulse to enable the multimodal sensing module array to be exposed synchronously. The feature extraction and fusion module is used to perform multi-scale feature extraction, defect response feature vector construction and cross-modal attention fusion in the method as described in any one of claims 1 to 9; The evidence reasoning and decision-making module stores the evidence depth defect classification model, which is used to output the evidence distribution based on the defect response feature vector, calculate the classification confidence and cognitive uncertainty, and determine whether the confidence threshold has been reached. The modality selection and scheduling module stores a modality mutual information matrix, which is used to select the target modality from the candidate modalities when the cognitive uncertainty exceeds the limit, and send a supplementary acquisition command to the corresponding sensing unit. The online learning update module is used to update the parameters of the modal mutual information matrix and the evidence depth defect classification model based on the modality selection sequence, confidence convergence path and final defect category in the detection process.

Images