A method for detecting the quality of a display screen

Abstract

This application discloses a quality inspection method for display screens, relating to the field of quality inspection technology, comprising: obtaining image data and extracting defect areas to generate a defect image dataset; generating a sensor dataset based on environmental data; aligning the defect image dataset and the sensor dataset by timestamp to construct a multimodal time series dataset; obtaining defect instances based on the defect image dataset, calculating the spatial distance, time interval, morphological similarity, and co-occurrence probability between any two defect instances as relational features, constructing a defect association graph model and performing relational feature analysis, determining whether preset early warning conditions are met based on the analysis results, and triggering an early warning including at least one combination of regional early warning, time series early warning, and rule degradation early warning; extracting defect evolution feature vectors from the multimodal time series dataset, inputting the defect evolution feature vectors into a prediction model, and outputting the probability of defect occurrence.

Description

Technical Field

[0001] This invention relates to the field of quality inspection technology, and in particular to a method for quality inspection of display screens. Background Technology

[0002] In the field of display screen manufacturing quality inspection, existing methods generally face problems such as insufficient detection accuracy, weak environmental adaptability, and lagging early warning mechanisms. In typical industrial environments, factors such as lighting, temperature, humidity, vibration, and noise are complex and variable, making it difficult for existing technologies to reliably identify hidden defects in screens. Furthermore, they cannot effectively characterize the spatiotemporal correlations and evolution patterns among multiple defects, resulting in frequent missed defects and false alarms. This fails to meet manufacturers' core needs for high detection accuracy and controllable costs, and also makes it difficult to provide production line quality inspectors with stable, timely, and non-frequently manual decision support, thus hindering the improvement of quality control and intelligentization throughout the entire screen manufacturing process.

[0003] Currently widely used automated inspection technologies largely rely on single image analysis methods: acquiring images of the screen surface, performing binary segmentation with a fixed threshold, and then extracting defect features. This method exhibits significant limitations in complex industrial environments: the detection results are easily affected by external interference such as fluctuations in ambient light, equipment vibration, and temperature changes, leading to unstable image quality and consequently false positives or false negatives; analysis based solely on morphological features makes it difficult to distinguish between genuine process defects and pseudo-defects caused by environmental noise, resulting in insufficient robustness; these methods are post-inspection, only able to detect and classify defects after they have fully formed, unable to provide early warnings or probabilistic predictions of defect initiation and development, thus failing to support proactive process adjustments and quality control. Summary of the Invention

[0004] This application provides a quality inspection method for display screens, which solves the problems of insufficient detection accuracy, weak environmental adaptability, lagging early warning mechanism, and lack of correlation analysis between defects in the prior art. It achieves the technical effect of high accuracy, early warning capability, and ability to reveal the evolution law of defects.

[0005] This application provides a method for quality testing of a display screen, including: S1: Obtain image data, extract defect areas to generate a defect image dataset; generate a sensor dataset based on environmental data; S2: Align the defect image dataset with the sensor dataset by timestamp to construct a multimodal time series dataset; S3: Obtain defect instances based on the defect image dataset, calculate the spatial distance, time interval, morphological similarity and co-occurrence probability between any two defect instances as relational features, construct a defect association graph model and perform relational feature analysis, and determine whether the preset early warning conditions are met based on the analysis results, triggering an early warning including at least one combination of regional early warning, time-series early warning and rule degradation early warning; The relational feature analysis includes spatiotemporal clustering analysis, frequency analysis, and rule confidence analysis; S4: Extract defect evolution feature vectors from the multimodal time series dataset, input the defect evolution feature vectors into the prediction model, and output the probability of defect occurrence; The defect evolution feature vector includes the defect area change rate, average temperature gradient, vibration energy, and sound pressure level change rate.

[0006] Furthermore, the area change rate is calculated by statistically analyzing the defect area through a sliding window, and the defect area change rate between adjacent frames is calculated using the difference method: , in, This represents the rate of change of the defect area. Let be the defect area at the current window time t. The defect area at window time t-1 is the area at the current window time t. The time interval between two windows; The average temperature gradient is extracted using linear regression to determine the temperature change trend. , in, The normalized temperature value. The average temperature gradient, For time window, The intercept is the temperature value at the start of the time window; The vibrational energy is the root mean square within each time window: , in, The root mean square, i.e., vibrational energy. Let be the amplitude reading acquired by the vibration sensor at time point i. For calculation The total number of data points contained within the time window used to calculate the value. To all within the time window Sum the squares of the amplitudes of each data point. The average power of the vibration signal within the time window; The rate of change of sound pressure level was obtained by calculating the slope through linear regression.

[0007] Further, the extraction of the defect region includes: calculating the overall grayscale histogram of the image data; marking pixel regions with grayscale values ​​greater than the segmentation threshold as candidate defect regions and pixel regions with grayscale values ​​less than the segmentation threshold as background regions; converting the image into a binary image; performing edge detection and contour extraction on the binary image to identify and locate the contour of the defect; calculating the minimum bounding rectangle of each contour to obtain the coordinates and size information of the region where the defect is located; and cropping the rectangular region corresponding to the defect from the original image based on the coordinates and size information to generate defect image data. The segmentation threshold is calculated by analyzing the grayscale histogram to maximize the separation between the defect candidate region and the background region.

[0008] Furthermore, the defect instance includes: extracting morphological features, texture features, and color features of each defect region from the defect image dataset to generate a defect feature vector; the timestamp of each defect occurrence and the center point position of the defect on the screen; The morphological features include defect area, perimeter, roundness, and minimum circumscribed rectangle size; The texture features are represented by texture histograms extracted from local binary patterns; The color features include the mean and variance of the hue based on the HSV color space.

[0009] Furthermore, the spatial distance is calculated using Euclidean distance based on the coordinates of the center points of the two defect instances on the screen; The time interval is determined based on the absolute value of the difference between the timestamps of the occurrence of the two defect instances; The morphological similarity is obtained by calculating the cosine similarity between the morphological feature vectors of two defective instances; The co-occurrence probability is determined based on the frequency of two defect instances appearing together within a time window, according to historical data.

[0010] Furthermore, the spatiotemporal clustering analysis is based on a defect association graph model. If the area of ​​the region formed by the defect clustering exceeds a preset area threshold, a regional warning is triggered. The frequency analysis is based on the defect association graph model to count the frequency of defect occurrence. If the increase in defect frequency exceeds a preset frequency threshold within a certain time window, a time-series warning is triggered. The rule confidence analysis is based on extracting defect association rules from historical data. If the confidence of a rule decreases by more than a preset confidence threshold within a certain time window, a rule degradation warning is triggered. The preset area threshold, preset frequency threshold, and preset confidence threshold are calibrated based on historical data.

[0011] Furthermore, the preset warning conditions include: performing tiered warning operations based on the trigger combination of regional warnings, time-series warnings, and rule degradation warnings; triggering a level one warning when only one warning dimension is triggered; triggering a level two warning when any two warning dimensions are triggered; and triggering a level three warning when all three warning dimensions are triggered.

[0012] Furthermore, the prediction model is a model trained based on a historical multimodal time series dataset. The training process includes: generating corresponding real labels for each time window based on actual defect occurrence records in historical production data; using the defect evolution feature vectors extracted from the multimodal time series dataset and the corresponding real labels as training samples, inputting them into a neural network with time series modeling capabilities, and training the model by optimizing the prediction error to obtain the prediction model.

[0013] Furthermore, the method also includes: dividing the screen surface into several sub-regions by collecting ambient light data, adjusting the image acquisition parameters and performing local image compensation based on the ambient light data corresponding to each sub-region, and generating image data after ambient light compensation. The adjustment of image acquisition parameters includes adjusting the exposure value according to the light intensity based on the real-time ambient light data of each sub-region, and correcting the contrast deviation caused by color temperature changes. The local image compensation includes using histogram equalization to improve local brightness and contrast in areas with an incident angle > 60°, in order to suppress shadow interference.

[0014] Furthermore, the division of the screen surface into several sub-regions includes a three-level division: the first-level division, based on the screen's physical structure, divides the screen surface into a display area, an edge area, and an interface area; the second-level division, based on the historical defect incidence rate, further divides the first-level area into high-frequency, medium-frequency, and low-frequency defect areas; and the third-level division, based on the ambient light incident angle, divides the screen area into high-sensitivity, medium-sensitivity, and low-sensitivity areas. Among these, the first-level division has a higher priority than the second-level division, and the second-level division has a higher priority than the third-level division.

[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages: By integrating industrial cameras with multiple types of sensors to perform multimodal data fusion, early and accurate prediction of hidden defects in screens was achieved. By introducing an ambient light compensation mechanism and multi-level region division, interference from complex environments was effectively suppressed, and detection stability was improved. By constructing a defect correlation graph model and a three-level early warning mechanism, the evolution law of defects was discovered and proactive hierarchical control was achieved. Attached Figure Description

[0016] Figure 1This is a flowchart of a display screen quality detection method according to an embodiment of the present invention. Detailed Implementation

[0017] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0019] Example 1: As Figure 1 As shown, a method for quality inspection of a display screen.

[0020] S1: Obtain image data, extract defect areas to generate a defect image dataset; generate a sensor dataset based on environmental data; Specifically, multimodal data is acquired in real time using industrial cameras and sensors. The industrial cameras continuously capture screen images at a frequency of 10 frames per second to obtain image data, including the brightness distribution, color features, and texture information of the screen surface. The sensors include temperature sensors, vibration sensors, and acoustic sensors. Among them, the temperature sensor monitors environmental thermal changes to obtain temperature data including temperature fluctuation rate and extreme point distribution; the vibration sensor detects abnormal vibrations of the equipment to obtain vibration data including amplitude, phase information, and vibration intensity; and the acoustic sensor captures material stress sound to obtain sound data including the temporal sequence of sound pressure level changes and spectral distribution characteristics.

[0021] The overall grayscale histogram of the image data is calculated. The optimal threshold is determined using the maximum inter-class variance method. Regions with pixel grayscale values ​​greater than the threshold are designated as white defect candidate regions, and regions with values ​​less than the threshold are designated as black background regions, converting the image into a binary image. After obtaining the binary image, an edge detection algorithm is used to identify defect contours. Continuous boundaries are extracted using contour approximation techniques, and the minimum bounding rectangle of each contour is calculated to obtain the coordinates and dimensions of the defect region. The rectangular region containing the defect is cropped from the original image, generating defect image data. The extracted defect image data from each frame are arranged chronologically to construct an image dataset. Each data point includes: a timestamp, defect image data, and metadata.

[0022] The raw sensor data is filtered to remove environmental noise. The filtered sensor data is then normalized, scaling each sensor reading to the [0,1] range. Temperature data is dynamically scaled using a historical extreme value sliding window method; vibration data is normalized using statistically obtained peak acceleration values ​​as the benchmark; and sound data is standardized based on the historical distribution of decibel reference values. The processed sensor data is stored by timestamp, generating a sensor dataset. Each data point includes: timestamp, temperature value, vibration intensity, and sound pressure level.

[0023] S2: Align the defect image dataset with the sensor dataset by timestamp to construct a multimodal time series dataset; Specifically, the timestamps of the sensor dataset and the image dataset are matched: the timestamp of each data point is extracted, and the data is aligned using a linear interpolation algorithm. For example, if the timestamp of the image data is t, but the sensor data is missing at time t, it is filled in by interpolation based on the sensor readings at adjacent timestamps. After matching, each time point corresponds to a complete set of multimodal data, resulting in a multimodal time series dataset. The dataset is stored in time series format, and each record contains: timestamp, defect image data, and sensor data.

[0024] The defect area change rate is calculated based on the defect image data in the defect image dataset. The time series is divided into fixed-length sliding windows with a window size of 5 seconds. The total number of white pixels in the binary defect image is counted as the defect area. The defect area change rate between adjacent frames is calculated using the difference method. , in, This represents the rate of change of the defect area. Let be the defect area at the current window time t. The defect area at window time t-1 is the area at the current window time t. This represents the time interval between two windows. A positive value for the rate of change of defect area indicates defect expansion, while a negative value indicates contraction; the absolute value reflects the intensity of the change.

[0025] Temperature characteristics are extracted from temperature sensor data, and temperature change trends are extracted using linear regression. , in, The normalized temperature value. The average temperature gradient, For time window, is the intercept, and is the theoretical temperature value at the start of the time window. Temperatures are trending upward. The temperature remained relatively stable within the time window. The temperature is trending downwards.

[0026] Using a peak detection algorithm, local maximum and local minimum values ​​are extracted from the temperature sequence. The number of extreme points within the time window is calculated as the extreme point frequency, and the difference between an extreme point and its adjacent valley is calculated as the average amplitude of the extreme point.

[0027] Vibration characteristics are extracted from vibration sensor data, and the root mean square (RMS) is calculated for each time window. , in, The root mean square, i.e., vibrational energy. Let be the amplitude reading acquired by the vibration sensor at time point i. For calculation The total number of data points contained within the time window used to calculate the value. To all within the time window Sum the squares of the amplitudes of each data point. The average power of the vibration signal within the time window is the average of the sum of squares. The rate of change of the RMS values ​​between adjacent windows is calculated using the finite difference method as the vibration rate of change.

[0028] Sound features are extracted from sound sensor data, and the slope of the linear regression is used as the sound pressure level change rate. For the spectral data of each time window, the ratio of key frequency band energy to total energy is calculated as the sound energy proportion. The rate of change of the sound energy proportion between adjacent windows is calculated using the difference method as the sound change rate.

[0029] The image, temperature, vibration and sound features within the same time window are combined into a defect evolution feature vector. The defect evolution feature vector includes the defect area change rate, average temperature gradient, extreme point frequency, extreme point average amplitude, vibration energy, vibration change rate, sound pressure level change rate and sound change rate.

[0030] S4: Extract defect evolution feature vectors from the multimodal time series dataset, input the defect evolution feature vectors into the prediction model, and output the probability of defect occurrence; Specifically, the defect evolution feature vector is input into a pre-trained prediction model, which outputs the probability of defect occurrence. A Long Short-Term Memory (LSTM) deep learning network is used as the prediction model, receiving the defect evolution feature vector as input, with each feature vector corresponding to a 5-second time window. The input data is organized as a time series, forming a sample sequence. Each sample contains feature vectors from five consecutive time windows, used to predict the probability of defects within the next 5 minutes. A Sigmoid activation function is used to map the output defect occurrence probability to the range [0,1]. The closer the probability value is to 1, the higher the likelihood of a latent defect appearing within the next 5 minutes.

[0031] The prediction model training process employs batch gradient descent. Training data is divided into multiple batches, each containing 64 sample sequences. Each sequence consists of defect evolution feature vectors representing five consecutive time windows. Historical data is chronologically divided into a training set (70%), a validation set (15%), and a test set (15%). The training set is used for model learning, the validation set for hyperparameter tuning, and the test set for evaluating generalization ability. The training cycle is determined using early stopping. After each cycle, cross-entropy loss is calculated on the validation set, and an L2 regularization term is added to penalize excessively large weights. , in, For defect prediction loss value, This represents the number of samples in the current training batch. This is a true label, with a value of 0 or 1, where 1 indicates a defect has occurred and 0 indicates no defect has occurred. It is obtained based on actual defect occurrence records from historical production data. To predict probabilities for the model, The regularization strength hyperparameter is used to control model complexity. For model weights, This is the sum of squares of all weight parameters in the model, used to suppress overfitting. It is minimized using gradient descent. This allows the model to more accurately predict the probability of defects occurring.

[0032] If the validation set loss no longer decreases for 5 consecutive epochs, training is terminated early, and the model parameters with the lowest validation set loss value are saved as the final model.

[0033] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: This application integrates industrial cameras and multiple sensors to acquire images, temperature changes, vibrations, and sound signals from the screen in real time, fusing them to form a multimodal time-series dataset. Dynamic evolution indicators such as defect area change rate, temperature trend, vibration energy, and acoustic spectrum characteristics are extracted from the dataset to construct feature vectors characterizing the defect development process. Long Short-Term Memory (LSTM) networks are used to model and learn the temporal features, enabling accurate prediction of the probability of latent screen defects occurring within the next few minutes. The complementary use of multi-source information enhances the robustness and early identification capabilities of the detection system, providing a complete intelligent solution for quality monitoring and process optimization in screen manufacturing.

[0034] Example 2: Example 1 uses industrial cameras and sensors to collect multimodal data in real time and uses dynamic feature extraction and LSTM model to predict defect probability. However, in actual production, it is easily affected by ambient light fluctuations, which leads to a decrease in image quality and an increase in false detection rate. This example further improves and explains the content of Example 1.

[0035] The method further includes: dividing the screen surface into several sub-regions by collecting ambient light data, adjusting the image acquisition parameters and performing local image compensation according to the ambient light data corresponding to each sub-region, and generating image data after ambient light compensation. Specifically, the ambient light sensor collects ambient light intensity, color temperature, and incident angle data in real time at a frequency of 10 frames per second. The collected ambient light data is timestamped with the image data and sensor data. A linear interpolation algorithm is used to match the timestamps of the ambient light data with those of the image data, and the ambient light data is then added to the multimodal time series dataset.

[0036] Establish the relationship between ambient light data and image acquisition parameters. Generate a light intensity versus exposure curve through historical data regression analysis. Color temperature data is used to adjust contrast; mapping is achieved by consulting laboratory calibration data tables. Incident angle data is used to identify areas of uneven illumination.

[0037] The screen surface is divided into multiple sub-regions, and a three-level region division system is established. The first-level division is based on the screen's physical structure, dividing the screen surface into display area, edge area, and interface area. Defect distribution characteristics serve as the second-level division standard, further dividing each region into high-frequency defect areas (>5%), medium-frequency defect areas ([1,5]%), and low-frequency defect areas (<1%) based on historical defect incidence rates. The degree of ambient light influence serves as the third-level division basis, dividing the screen area into high-sensitivity areas (>60°), medium-sensitivity areas ([30,60]°), and low-sensitivity areas (<30°) based on the incident angle. When region division conflicts occur, the first-level region has higher priority than the second-level region, which in turn has higher priority than the third-level region. Ambient light data from each sub-region is time-stamped with image data and sensor data. An independent ambient light dataset is generated for each sub-region, including timestamps, local illumination intensity, color temperature, and incident angle. Calculate the variance and range of illumination data for each sub-region. If the difference in illumination intensity between sub-regions exceeds 50 Lux, it is marked as an area of ​​uneven illumination, triggering a local compensation mechanism.

[0038] The exposure parameters of the industrial camera are dynamically adjusted based on the light intensity of the sub-regions. When the light intensity is below 100 Lux, the exposure value is automatically increased to the maximum value to prevent the image from being too dark; when the light intensity is above 1000 Lux, the exposure value is reduced to the minimum value to prevent overexposure. Based on the color temperature data of the sub-regions, local contrast adjustment is applied. In low color temperature warm light environments (color temperature < 4000K), the contrast of the blue channel is increased to compensate for color shift; in high color temperature cool light environments (color temperature > 6000K), the contrast of the red channel is enhanced. For shadow areas caused by high incident angles > 60°, histogram equalization technology is used for local brightness compensation to maintain the contrast stability between defects and the background. After local compensation, the image data of each sub-region is normalized, scaling parameters such as brightness and contrast to the [0,1] range to ensure consistent inspection standards across the entire screen.

[0039] The image after ambient light compensation is converted to a grayscale image. For each pixel in the grayscale image, a 3×3 neighborhood window is taken centered on that pixel. The grayscale values ​​of the eight neighboring pixels are compared with the grayscale value of the center pixel. If a neighboring pixel value is greater than the center pixel value, the corresponding position is marked as 1; otherwise, it is marked as 0. The comparison results are read clockwise to generate an 8-bit binary sequence. The binary sequence is converted into a decimal value and used as the LBP (Local Binary Pattern) feature value of the center pixel. The LBP value distribution of all pixels in the entire image is statistically analyzed to generate an LBP histogram, which serves as the texture feature of the image.

[0040] The image, after ambient light compensation processing, is converted from the RGB color space to the HSV color space. Because hue values ​​are relatively insensitive to changes in illumination—even with altered lighting conditions, the hue values ​​of the same object remain relatively stable—the mean of the hue components within the HSV color space is directly extracted to quantify the overall color distribution stability of the image. The mean value reflects the color consistency between defective areas and the background, and fluctuations in the mean can indicate color shifts caused by changes in illumination. RGB values ​​are directly affected by light intensity; the same color will have completely different RGB values ​​under different lighting conditions. The Gray-World assumption is used to calculate the mean values ​​of each color channel (R, G, B). By comparing the differences in the mean values ​​of the three channels, the channel gain is adjusted to make the overall color average tend towards gray, thereby compensating for illumination color shifts. The normalized RGB color features are then fused with the HSV hue mean through feature concatenation to form color features. Texture features and color features are combined into an illumination-invariant feature vector.

[0041] The ambient light compensated image was converted to grayscale, and a multi-scale image pyramid was constructed to generate three image levels with different resolutions: the original image scale (100% resolution), the 50% downsampled scale (half resolution), and the 25% downsampled scale (quarter resolution). The original image scale retains complete details and is used to extract subtle texture features. The half-downsampled scale uses bilinear interpolation to downsample the original image, reducing its size by half, and is used to extract medium-scale shape features of defect contours. The quarter-downsampled scale further downsamples the half-scale image for coarse localization of the approximate defect area. Gabor filters were applied to perform texture feature analysis on the images at each scale level, configuring different frequency scale parameters in four directions (0°, 45°, 90°, and 135°) to generate texture response maps with directionality and frequency selectivity. The original image scale uses high-frequency parameters of 0.3-0.5 periods / pixel, the 1 / 2 downsampling scale uses mid-frequency parameters of 0.1-0.3 periods / pixel, and the 1 / 4 downsampling scale uses low-frequency parameters of 0.05-0.1 periods / pixel. These frequency parameters are calibrated based on historical data. Defect edges are identified using the Canny edge detection algorithm, generating a binary edge map. Hu moment invariant features are calculated based on the binary edge map to characterize the overall shape and orientation of the defect, ensuring the stability of shape features under scale changes. Each scale outputs a multi-scale feature vector, including a texture response map, a binary edge map, and Hu moment values.

[0042] The prediction model adds a separate branch for predicting the intensity of ambient light influence, sharing the same LSTM feature extractor as the defect prediction branch, but using a separate output layer. The illumination-invariant feature vector and multi-scale feature vector are concatenated into a global feature vector, which is then input into the prediction model. Ambient light prediction is treated as a regression task, employing the mean squared error loss function. , in, This represents the predicted loss value for ambient light. The number of samples, i.e., the number of data points contained in a training batch. For sample indexing, from arrive This is used to iterate through each sample in the batch; The ambient light influence intensity value predicted by the model is obtained by using a linear activation function output from the ambient light prediction branch. The ambient light influence intensity value is normalized to [0,1], where 0 represents no light interference and 1 represents the strongest interference. To obtain a true label for the intensity of ambient light influence, the variances of light intensity, color temperature, and incident angle are extracted from ambient light sensor data, normalized to [0,1], and the average value is calculated. This represents the squared error of a single sample.

[0043] The total loss function is defined by weighted summation of the defect prediction loss value and the ambient light prediction loss value: , in, This is the total loss value. For defect prediction loss value, This represents the predicted loss value for ambient light. These are the weighting coefficients, initially... Calculate the loss ratio every 5 training cycles. ,like ,but Increase by 0.1, if ,but Reduce by 0.1.

[0044] The overall loss function can simultaneously supervise the performance of the prediction model on both defect identification and ambient light interference suppression tasks. During training, the optimizer is based on... Backpropagation is performed to update the weights of the LSTM feature extractor and its two output branches, enabling the model to adaptively compensate for image quality fluctuations caused by changes in illumination while maintaining high defect detection accuracy. This is achieved through weight coefficients. The model employs a dynamic adjustment mechanism that allows it to automatically balance the learning focus based on the ratio of the two types of losses: if the ambient light interference is strong, it enhances the supervision of the ambient light branch; if the interference is weak, it focuses more on optimizing the defect recognition task itself.

[0045] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: This application achieves adaptive perception and compensation for complex lighting environments by introducing an ambient light sensor and a multi-level region segmentation mechanism; by combining illumination-invariant feature extraction and multi-scale texture analysis, it effectively suppresses the impact of illumination fluctuations on image quality and improves the stability and accuracy of defect detection; and by employing a dual-task learning LSTM prediction model, it simultaneously optimizes defect recognition and ambient light interference suppression, enhancing robustness in variable scenarios.

[0046] Example 3: Example 2 achieved the detection of a single defect and suppression of ambient light interference, but still lacked the ability to identify the correlation and synergistic development patterns among multiple defects. This example further supplements and explains the content of Example 2.

[0047] S3: Obtain defect instances based on the defect image dataset, calculate the spatial distance, time interval, morphological similarity and co-occurrence probability between any two defect instances as relational features, construct a defect association graph model and perform relational feature analysis, determine whether the preset early warning conditions are met based on the analysis results, and trigger an early warning including at least one combination of regional early warning, time-series early warning and rule degradation early warning; Specifically, a defect feature vector containing morphological, texture, and color features is extracted for each defect instance. The morphological features include defect area, perimeter, roundness, and minimum bounding rectangle size; the texture features are extracted using LBP to create a texture histogram, enhancing the distinction between bright and dark spots; and the color features are calculated using HSV hue analysis to determine the hue mean and variance of the color cast. The timestamp and center point location of each defect are recorded to form a defect instance dataset.

[0048] For each pair of defect instances, calculate the spatial distance, time interval, morphological similarity, and co-occurrence probability as relational features. Calculate the Euclidean distance between the center points of two defects; the smaller the distance, the stronger the correlation. , in, The spatial distance between defect instance A and defect instance B. For defect instance A Axis coordinates For defect instance B Axis coordinates For defect instance A Axis coordinates For defect instance B Axis coordinates.

[0049] Calculate the absolute value of the time difference between the occurrence of defects; the shorter the interval, the stronger the correlation. , in, The time interval between defect instance A and defect instance B. This is the timestamp of when defect instance A occurred. This is the timestamp of when defect instance B occurred.

[0050] Compare the morphological feature vectors of defects using cosine similarity: , in, Let be the similarity between defect instance A and defect instance B. and These are the morphological feature vectors of defect instances A and B, respectively. and These are the Euclidean norms of eigenvectors A and B, respectively, used for normalization. The closer the value is to 1, the more similar the two values ​​are.

[0051] Based on historical data, the frequency of simultaneous occurrence of two types of defects within a 5-second time window is used as the co-occurrence probability. A relational feature dataset is generated within each 5-second window, containing defect pairs and their relational values. A multi-defect association graph model is constructed by assigning weights to edges based on similarity and co-occurrence probability, with nodes representing defect instances and edges representing relationships between defects.

[0052] Using a spatiotemporal clustering algorithm, defect instances are clustered based on spatial coordinates and timestamps. The spatial neighborhood radius is set to 5% of the screen size, and the time window is set to 5 seconds. For each cluster, the cluster center, the number of defects, and the average frequency of occurrence are calculated. The coordinates of the cluster centers are calculated using the mean. , in, As cluster center, The number of defects in the cluster. For defect indexing, The coordinates of the defect instances are displayed. The clustering results are output, marking high-frequency defect areas and anomalous time points.

[0053] Strong association rules from defect relationships are used to predict defect co-occurrence. Each 5-second time window represents a set of defect instances, and the support and confidence scores for each defect relationship are calculated. Support is used to measure the generality of the rule. , in, For defects To defects Support For defects to occur simultaneously and defects Number of time windows This represents the total number of time windows.

[0054] Confidence level is used to measure the reliability of a rule: , in, For defects To defects Confidence level, For defects To defects Support For defects Support , For defects Number of time windows This represents the total number of time windows.

[0055] Based on historical data, a minimum support threshold of 0.05 and a minimum confidence threshold of 0.7 were set to filter weak rules. After filtering, all rules meeting the criteria were aggregated into a defect rule set. For example, if bright spot defects frequently co-occur with vibration sensor anomalies, and the support is >0.1 and the confidence is >0.8, it indicates that device vibration may cause bright spots on the screen, and this rule is included in the defect rule set. The rule confidence is used as a feature input to the LSTM prediction model to enhance prediction accuracy. When the rule confidence decreases, model retraining is triggered.

[0056] The conditions for triggering alerts are defined according to association rules: when the defect cluster radius exceeds 10% of the screen area, a regional alert is triggered; when the frequency of defect occurrence increases by 50% within three consecutive windows, a time-series alert is triggered; when the confidence of a defect rule decreases by more than 20% within three consecutive windows, a rule degradation alert is triggered.

[0057] When any one warning dimension is triggered alone, and other dimensions do not reach the threshold, a Level 1 warning is triggered, the log is recorded, and the abnormal dimension data is marked; when any two warning dimensions are triggered simultaneously, a Level 2 warning is triggered, an audible and visual alarm is triggered, quality inspectors are notified to intervene and verify, and multi-dimensional data is automatically associated to generate an anomaly analysis report; when all three warning dimensions are triggered, a Level 3 warning is triggered, the production line is immediately suspended, historical rule sets are called for defect tracing, and the model retraining mechanism is triggered.

[0058] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: This application performs spatiotemporal correlation analysis on multiple defect instances, extracts morphological, texture, and color features, and constructs a defect correlation graph model, achieving a structured representation and quantitative analysis of the intrinsic relationships between defects. By combining spatiotemporal clustering and association rule mining, it can identify high-frequency defect regions, abnormal time points, and defect co-occurrence patterns, thereby revealing the potential mechanisms and propagation paths of defect generation. Based on a multi-dimensional early warning mechanism, it achieves hierarchical early warning from regional anomalies, temporal anomalies to rule degradation, enhancing the proactiveness of defect detection.

[0059] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for quality inspection of a display screen, characterized in that, include: S1: Obtain image data, extract defect areas, and generate a defect image dataset; Generate sensor datasets based on environmental data; S2: Align the defect image dataset with the sensor dataset by timestamp to construct a multimodal time series dataset; S3: Obtain defect instances based on the defect image dataset, calculate the spatial distance, time interval, morphological similarity and co-occurrence probability between any two defect instances as relational features, construct a defect association graph model and perform relational feature analysis, determine whether the preset early warning conditions are met based on the analysis results, and trigger an early warning including at least one combination of regional early warning, time-series early warning and rule degradation early warning; The relational feature analysis includes spatiotemporal clustering analysis, frequency analysis, and rule confidence analysis; S4: Extract defect evolution feature vectors from the multimodal time series dataset, input the defect evolution feature vectors into the prediction model, and output the probability of defect occurrence; The defect evolution feature vector includes the defect area change rate, average temperature gradient, vibration energy, and sound pressure level change rate.

2. The method for quality inspection of a display screen as described in claim 1, characterized in that, The area change rate is calculated by statistically analyzing the defect area through a sliding window and using the difference method to calculate the defect area change rate between adjacent frames: ， in, This represents the rate of change of the defect area. Let be the defect area at the current window time t. The defect area at window time t-1 is the area at the current window time t. The time interval between two windows; The average temperature gradient is extracted using linear regression to determine the temperature change trend. ， in, The normalized temperature value. The average temperature gradient, For time window, The intercept is the temperature value at the start of the time window; The vibrational energy is the root mean square within each time window: ， in, The root mean square, i.e., vibrational energy. Let be the amplitude reading acquired by the vibration sensor at time point i. For calculation The total number of data points contained within the time window used to calculate the value. To all within the time window Sum the squares of the amplitudes of each data point. The average power of the vibration signal within the time window; The rate of change of sound pressure level was obtained by calculating the slope through linear regression.

3. The method for quality inspection of a display screen as described in claim 1, characterized in that, The defect region extraction process includes: calculating the overall grayscale histogram of the image data; marking pixel regions with grayscale values ​​greater than a segmentation threshold as candidate defect regions and pixel regions with grayscale values ​​less than a segmentation threshold as background regions; converting the image into a binary image; performing edge detection and contour extraction on the binary image to identify and locate the contours of the defects; calculating the minimum bounding rectangle of each contour to obtain the coordinates and size information of the defect region; and cropping the rectangular region corresponding to the defect from the original image based on the coordinates and size information to generate defect image data. The segmentation threshold is calculated by analyzing the grayscale histogram to maximize the separation between the defect candidate region and the background region.

4. The method for quality inspection of a display screen as described in claim 1, characterized in that, The defect instance includes: extracting the morphological features, texture features, and color features of each defect region from the defect image dataset, generating a defect feature vector; the timestamp of each defect's occurrence and the center point position of the defect on the screen; The morphological features include defect area, perimeter, roundness, and minimum circumscribed rectangle size; The texture features are represented by texture histograms extracted from local binary patterns; The color features include the mean and variance of the hue based on the HSV color space.

5. The method for quality inspection of a display screen as described in claim 1, characterized in that, The spatial distance is calculated using Euclidean distance based on the coordinates of the center points of the two defect instances on the screen. The time interval is determined based on the absolute value of the difference between the timestamps of the occurrence of the two defect instances; The morphological similarity is obtained by calculating the cosine similarity between the morphological feature vectors of two defective instances; The co-occurrence probability is determined based on the frequency of two defect instances appearing together within a time window, according to historical data.

6. The method for quality inspection of a display screen as described in claim 1, characterized in that, The spatiotemporal clustering analysis is based on the defect association graph model. If the area of ​​the region formed by the defect clustering exceeds the preset area threshold, a regional warning is triggered. The frequency analysis is based on the defect association graph model to count the frequency of defect occurrence. If the increase in defect frequency exceeds a preset frequency threshold within a certain time window, a time-series warning is triggered. The rule confidence analysis is based on extracting defect association rules from historical data. If the confidence of a rule decreases by more than a preset confidence threshold within a certain time window, a rule degradation warning is triggered. The preset area threshold, preset frequency threshold, and preset confidence threshold are calibrated based on historical data.

7. The method for quality inspection of a display screen as described in claim 1, characterized in that, The preset early warning conditions include: executing tiered early warning operations based on the trigger combination of regional early warning, time-series early warning, and rule degradation early warning; triggering a level one early warning when only one early warning dimension is triggered; triggering a level two early warning when any two early warning dimensions are triggered; and triggering a level three early warning when all three early warning dimensions are triggered.

8. The method for quality inspection of a display screen as described in claim 1, characterized in that, The prediction model is trained based on a historical multimodal time series dataset. The training process includes: generating corresponding real labels for each time window based on actual defect occurrence records in historical production data; using the defect evolution feature vectors extracted from the multimodal time series dataset and the corresponding real labels as training samples, inputting them into a neural network with time series modeling capabilities, and training the model by optimizing the prediction error to obtain the prediction model.

9. The method for quality inspection of a display screen as described in claim 1, characterized in that, The method further includes: dividing the screen surface into several sub-regions by collecting ambient light data, adjusting the image acquisition parameters and performing local image compensation according to the ambient light data corresponding to each sub-region, and generating image data after ambient light compensation. The adjustment of image acquisition parameters includes adjusting the exposure value according to the light intensity based on the real-time ambient light data of each sub-region, and correcting the contrast deviation caused by color temperature changes. The local image compensation includes using histogram equalization to improve local brightness and contrast in areas with an incident angle > 60°, in order to suppress shadow interference.

10. The method for quality inspection of a display screen as described in claim 9, characterized in that, The process of dividing the screen surface into several sub-regions includes three levels of division: Level 1 division, based on the screen's physical structure, divides the screen surface into a display area, an edge area, and an interface area; Level 2 division, based on the historical defect incidence rate, further divides the Level 1 area into high-frequency, medium-frequency, and low-frequency defect areas; Level 3 division, based on the ambient light incident angle, divides the screen area into high-sensitivity, medium-sensitivity, and low-sensitivity areas; wherein, Level 1 division has a higher priority than Level 2 division, and Level 2 division has a higher priority than Level 3 division.

Images