A fault detection method and system for display screen production based on data feedback

Abstract

The application relates to the technical field of fault monitoring, in particular to a display screen production fault detection method and system based on data feedback. The method comprises the following steps: acquiring multi-modal image data and environmental light intensity of a to-be-detected display screen in a production process; performing data checking processing on the multi-modal image data and the environmental light intensity to obtain checked multi-modal image data and environmental light intensity; comparing the checked environmental light intensity with preset light intensity to obtain an environmental light coefficient; adjusting a preset image detection model by using the environmental light coefficient to confirm an adjusted image detection model; and detecting the multi-modal image data by using the adjusted image detection model to obtain a detection result of the to-be-detected display screen in production. The application aims to solve the problem that the existing detection method is easily disturbed by the environment, resulting in low detection precision of the display screen.

Description

Technical Field

[0001] This invention relates to the field of fault monitoring technology, and more specifically, to a fault detection method and system for display screen production based on data feedback. Background Technology

[0002] Display screens are devices used for outputting visual information and are widely used in electronic products such as computers, televisions, and mobile phones. With technological advancements, display screens have transitioned from CRT screens to more advanced LCD and LED screens.

[0003] Currently, displays are prone to various defects during production, such as screen flickering, white screens, color cast, bright spots, dark spots, uneven brightness, scratches, and cracks. Fault detection during production is crucial for ensuring product quality, reducing production costs, meeting industry standards, and improving user experience. However, current display fault detection technologies rely on contrast ratios, making them susceptible to environmental interference and unable to detect low-contrast objects, such as black defects on a black background, resulting in significantly reduced detection accuracy. Furthermore, changes in ambient lighting and reflection interference also severely impact detection accuracy. Additionally, existing optical imaging systems struggle to identify minute defects, further contributing to low detection precision.

[0004] Therefore, it is necessary to design a fault detection method and system for display screen production based on data feedback to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes a fault detection method and system for display screen production based on data feedback, aiming to solve the problem that existing detection methods are easily affected by environmental interference, resulting in low detection accuracy of display screens.

[0006] In one aspect, the present invention proposes a fault detection method for display screen manufacturing based on data feedback, comprising:

[0007] Acquire multimodal image data and ambient light intensity of the display screen under test during the production process;

[0008] The multimodal image data and ambient light intensity are subjected to data verification processing to obtain the verified multimodal image data and ambient light intensity.

[0009] The ambient light intensity after testing is compared with the preset light intensity to obtain the ambient light coefficient;

[0010] The preset image detection model is adjusted using the ambient light coefficient, and the adjusted image detection model is confirmed.

[0011] The adjusted image detection model is used to detect multimodal image data to obtain the detection results of the display screen under test during production.

[0012] Furthermore, the step of acquiring multimodal image data and ambient light intensity of the display screen under test during the production process includes:

[0013] Acquire initial multimodal image data of the display screen under test during the production process, including visible light image data, infrared thermal imaging data, and multispectral image data; and acquire external light intensity, screen reflectance coefficient, and backlight intensity of the display screen under test.

[0014] Based on the external light intensity, screen reflectance, and backlight intensity, the initial ambient light intensity is determined;

[0015] The initial data of the multimodal image and the initial intensity of the ambient light are time-aligned to obtain the multimodal image data and the ambient light intensity.

[0016] Further, the step of performing data verification processing on the multimodal image data and ambient light intensity to obtain verified multimodal image data and ambient light intensity includes:

[0017] The multimodal image data and ambient light intensity are verified separately to obtain verified multimodal image data and verified ambient light intensity.

[0018] The validated multimodal image data and validated ambient light intensity are denoised and standardized respectively to obtain the validated multimodal image data and ambient light intensity.

[0019] Further, the step of comparing the tested ambient light intensity with a preset light intensity to obtain the ambient light coefficient includes:

[0020] The tested ambient light intensity is compared with the preset light intensity to confirm the light intensity ratio between the tested ambient light intensity and the preset light intensity;

[0021] After verifying the light intensity ratio, the qualified light intensity ratio is taken as the ambient light coefficient.

[0022] If the light intensity ratio verification fails, the sensor that acquires the ambient light intensity of the display screen under test is verified, and the multimodal image data and ambient light intensity of the display screen under test during the production process are reacquired, and a fault prompt message is generated.

[0023] Furthermore, if the light intensity ratio verification fails, then after verifying the sensor that acquires the ambient light intensity of the display screen under test, reacquire the multimodal image data and ambient light intensity of the display screen under test during the production process, and generate a fault prompt message;

[0024] If the recalibrated light intensity ratio fails again, a result indicating that the display screen under test has failed the test will be generated.

[0025] Furthermore, the preset image detection model is adjusted using the ambient light coefficient, and the steps for confirming the adjusted image detection model include:

[0026] The ambient light coefficient is used to adjust the preset image detection model to obtain a preliminary image detection model;

[0027] The preliminary image detection model was validated using multimodal image sample data to obtain the validation results of the preliminary image detection model.

[0028] If the verification result of the preliminary image detection model indicates that it is qualified, the qualified preliminary image detection model will be used as the adjusted image detection model.

[0029] If the verification result of the preliminary image detection model is unqualified, the training sample size of the preliminary image detection model is increased, and a new preliminary image detection model is obtained and then re-verified.

[0030] Furthermore, the step of using the adjusted image detection model to detect multimodal image data and obtaining the detection results of the display screen under test during production includes:

[0031] The multimodal image data is fused to obtain fused data;

[0032] The adjusted image detection model is used to detect the fused data to obtain the detection results of the display screen under test during production.

[0033] Further, the step of fusing the multimodal image data to obtain fused data includes:

[0034] Based on the multimodal image data, the data type in the multimodal image data is identified;

[0035] Based on the data type and the preset fusion scheme, determine the data fusion scheme;

[0036] The data fusion scheme is used to fuse the multimodal image data to obtain fused data.

[0037] Furthermore, after the step of using the adjusted image detection model to detect multimodal image data and obtaining the detection results of the display screen under test during production, the method further includes:

[0038] If the test result indicates that the display screen is in normal production, an indicator that the display screen under test is in normal production is generated, and the test result is fed back to the display screen production equipment;

[0039] If the test result indicates a production abnormality in the display screen, an identifier for the production abnormality of the display screen under test is generated, and the test result and the corresponding preset improvement plan are fed back to the display screen production equipment.

[0040] Compared with existing technologies, the beneficial effects of this invention are as follows: By acquiring multimodal image data and ambient light intensity of the display screen under test during the production process, a comprehensive number of multimodal images are collected, providing a foundation for subsequent inspection. Data verification processing is performed on the multimodal image data and ambient light intensity to obtain verified multimodal image data and ambient light intensity, thereby ensuring data accuracy and consistency and improving the reliability of subsequent inspections. The verified ambient light intensity is compared with a preset light intensity to obtain an ambient light coefficient; the ambient light coefficient is used to quantify the impact of current illumination on inspection. The preset image detection model is adjusted using the ambient light coefficient, and the adjusted image detection model is confirmed; then, the ambient light coefficient is used to provide data feedback to the model, ensuring that the model can accurately detect under different illuminations. The multimodal image data is then detected using a more precise adjusted image detection model to obtain the inspection results of the display screen under test during production. This enables accurate identification of display screen defects and ensures the quality of the produced display screens. Simultaneously, by using ambient light intensity for data feedback, the preset image detection model adapts to different display screen inspection environments, improving inspection accuracy and efficiency.

[0041] On the other hand, this application also provides a data feedback-based fault detection system for display screen production, used to apply the data feedback-based fault detection method for display screen production as described in any of the above claims, including:

[0042] The acquisition module is used to acquire multimodal image data and ambient light intensity of the display screen under test during the production process;

[0043] The verification module is used to perform data verification processing on the multimodal image data and ambient light intensity to obtain the verified multimodal image data and ambient light intensity.

[0044] The comparison module is used to compare the tested ambient light intensity with a preset light intensity to obtain the ambient light coefficient;

[0045] The confirmation module is used to adjust the preset image detection model using the ambient light coefficient and confirm the adjusted image detection model.

[0046] The detection module is used to detect multimodal image data using the adjusted image detection model to obtain the detection results of the display screen under test during production.

[0047] It is understandable that the above-mentioned fault detection method and system for display screen production based on data feedback have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0048] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0049] Figure 1 A flowchart of a fault detection method for display screen production based on data feedback provided in an embodiment of the present invention;

[0050] Figure 2 A flowchart of S100 in the fault detection method for display screen production based on data feedback provided in an embodiment of the present invention;

[0051] Figure 3 A flowchart of step S300 in the data feedback-based fault detection method for display screen production provided in an embodiment of the present invention;

[0052] Figure 4 A flowchart of step S400 in the fault detection method for display screen production based on data feedback provided in an embodiment of the present invention;

[0053] Figure 5 This is a functional block diagram of a fault detection system for display screen production based on data feedback, provided in an embodiment of the present invention. Detailed Implementation

[0054] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] In some embodiments of this application, see Figure 1 As shown, a fault detection method for display screen production based on data feedback includes the following steps:

[0056] S100: Acquires multimodal image data and ambient light intensity of the display screen under test during the production process. The multimodal image data is collected from images using different types of sensors, covering the display screen's state information under various physical fields. Specifically, data is acquired through a multi-sensor array and a light intensity sensor. The multi-sensor array includes a visible light camera, an infrared thermal imager, and a multispectral camera, simultaneously acquiring multimodal image data and ambient light intensity during the display screen's production process. Ambient light intensity refers to the intensity of light surrounding the display screen during production, reflecting the potential impact of illumination on display screen inspection. By acquiring multimodal data, complementary information can be obtained, thus comprehensively depicting the display screen's state and improving the coverage of defect detection data. For example, infrared detection can detect hidden defects, visible light can identify surface scratches, and multispectral imaging overcomes contrast limitations, enabling the identification of low-contrast defects, such as black scratches on a black background. Ambient light intensity provides benchmark data for subsequent inspections, avoiding image distortion caused by excessively strong or dim lighting.

[0057] S200: Perform data verification processing on the multimodal image data and ambient light intensity to obtain verified multimodal image data and ambient light intensity. The data verification processing involves operations such as denoising and standardization on the multimodal image data and ambient light intensity to eliminate noise interference, standardize data format, and ensure data quality.

[0058] S300: Compare the tested ambient light intensity with the preset light intensity to obtain the ambient light coefficient. The preset light intensity is a pre-defined standard ambient light intensity value, such as the recommended light intensity value for existing production workshops. The ambient light coefficient is an adjustment factor used to quantify the impact of current illumination on detection. By determining the ambient light coefficient, missed or false detections due to sudden changes in illumination can be avoided.

[0059] S400: Adjust the preset image detection model using the ambient light coefficient, and confirm the adjusted image detection model. The preset image detection model is a deep learning model trained under standard lighting conditions, such as the YOLO model or the Faster R-CNN model. Adjusting the image detection model involves optimizing the model based on the ambient light coefficient to adapt to actual lighting conditions and avoid performance degradation due to changes in lighting. For example, if the ambient light is too strong or too dark, the preset image detection model is adjusted using the ambient light coefficient, such as by modifying the threshold or weights in the preset image detection model, to ensure accurate detection under different lighting conditions, thereby achieving more precise detection of the display screen.

[0060] S500: The adjusted image detection model is used to detect multimodal image data to obtain the detection results of the display screen under test during production. Multimodal fusion detection combines temperature anomalies from infrared thermography, surface scratches from multispectral imaging, and appearance defects from visible light images. For example, internal cracks in the display screen can be detected through infrared thermography. The detection results accurately identify the type, location, and severity of defects in the display screen. Specifically, defect types include cracks, stains, and overheating, ensuring the quality of display screen production. Simultaneously, by inspecting each display screen under test, rework costs due to missed detections are reduced, increasing overall production capacity.

[0061] In this embodiment, multimodal image data and ambient light intensity of the display screen under test during the production process are acquired. By collecting comprehensive multimodal image data, a foundation for subsequent inspection is provided. Data verification processing is performed on the multimodal image data and ambient light intensity to obtain verified multimodal image data and ambient light intensity, thereby ensuring data accuracy and consistency and improving the reliability of subsequent inspections. The verified ambient light intensity is compared with a preset light intensity to obtain an ambient light coefficient; this coefficient is used to quantify the impact of current illumination on inspection. The preset image detection model is adjusted using the ambient light coefficient, and the adjusted model is confirmed. The ambient light coefficient is then used to provide data feedback to the model, ensuring accurate detection under different lighting conditions. The adjusted image detection model is used to inspect the multimodal image data, obtaining the inspection results of the display screen under test during production. This enables accurate identification of display screen defects, ensuring the quality of produced display screens. Simultaneously, by using ambient light intensity as data feedback, the preset image detection model adapts to different display screen inspection environments, thereby improving the accuracy and efficiency of display screen inspection.

[0062] Please refer to some embodiments of this application as well. Figure 2 Step S100: The step of acquiring multimodal image data and ambient light intensity of the display screen under test during the production process includes:

[0063] S110: Acquire initial multimodal image data of the display screen under test during the production process, including visible light image data, infrared thermal imaging data, and multispectral image data, as well as acquire external light intensity, screen reflectance coefficient, and backlight intensity of the display screen under test. The initial multimodal image data is the unprocessed raw image containing multi-dimensional information such as visible light, infrared thermal imaging, and multispectral data. External light intensity is acquired through a light intensity sensor. Multimodal data acquisition involves simultaneously capturing images of the display screen using multiple sensors, such as a visible light camera recording the appearance; an infrared thermal imager monitoring temperature distribution; and a multispectral camera capturing material properties. Visible light is used to identify surface defects of the display screen, such as scratches or stains. Infrared thermal imaging can detect temperature anomalies in the display screen, such as short circuits or overheating. Multispectral data can analyze material composition or microstructure, such as fluorescence reactions or interlayer delamination. Utilizing multimodal data acquisition can cover both the visible surface and hidden internal features of defects, improving the comprehensiveness of display screen inspection. Simultaneously, the reflectance coefficient is measured by illuminating the screen with a standard light source, determining the ratio of reflected light intensity to incident light intensity. The screen reflectance coefficient is the proportion of incident light reflected by the screen material, ranging from 0 to 1, to determine the impact of ambient light on the screen display. Backlight intensity is the light intensity emitted by the display itself, such as the brightness of an LED backlight module, measured in nits. Backlight intensity is measured directly using a photometer to measure the screen's luminance. By obtaining the reflectance coefficient and backlight intensity, we provide the screen's own optical characteristic parameters for ambient light calculations, avoiding the limitations of relying solely on external lighting.

[0064] S120: Based on the external light intensity, screen reflectivity, and backlight intensity, determine the initial ambient light intensity. The external light intensity is the ambient light independently measured in the workshop where the displays are produced, such as the brightness of the workshop's overhead lights. The initial ambient light intensity comprehensively considers the equivalent ambient light intensity of the screen reflection and the independently measured ambient light in the workshop. By accurately quantifying the ambient light, errors caused by using only external light intensity are avoided, such as preventing strong reflection interference from highly reflective screens in low light conditions. Furthermore, determining the initial ambient light intensity is applicable to different screen types, such as high-brightness screens and matte screens, as well as production scenarios such as cleanrooms and open production lines. The formula for calculating the initial ambient light intensity is as follows:

[0065] D = Z × (1 - X) + Y;

[0066] Where D represents the initial intensity of ambient light, Z represents the intensity of external light, X represents the reflectance coefficient, and Y represents the backlight intensity.

[0067] S130: The initial multimodal image data and the initial ambient light intensity are time-aligned to obtain multimodal image data and ambient light intensity. Time alignment is achieved through timestamp synchronization or interpolation algorithms to ensure that the multimodal data and ambient light intensity are acquired at the same point in time. This ensures that the multimodal image and ambient light intensity input to the detection model correspond to the same production moment, avoiding analysis misalignment and ensuring data consistency. Simultaneously, the time-aligned data better reflects the actual production process of the display screen, improving the accuracy of model training and real-time detection.

[0068] In some embodiments of this application, the step of performing data verification processing on the multimodal image data and ambient light intensity to obtain verified multimodal image data and ambient light intensity includes:

[0069] The multimodal image data and ambient light intensity are verified separately to obtain verified multimodal image data and verified ambient light intensity. The verified multimodal image data and verified ambient light intensity are data that have passed integrity, consistency, and rationality checks. Integrity checks are performed to ensure the data is complete, such as checking for missing image files and empty light intensity values. Consistency checks are performed to ensure the multimodal data is synchronized in time and space, such as checking if the time difference between the infrared and visible light images exceeds a threshold. Pixel value range checks are performed, such as ensuring that 8-bit images should be 0-255 and checking for dead pixels or stripe noise. Rationality checks are performed to ensure the data is within a physically feasible range, such as ensuring that backlight intensity cannot be negative. Data cleaning is achieved by filtering out invalid or erroneous data to reduce interference in subsequent processing. Continuous data verification allows for monitoring of sensor status; for example, if a camera consistently outputs abnormal pixels, it indicates a hardware fault, thus providing a fault warning. This prevents data anomalies, such as missing pixels or sudden changes in light intensity values, that may occur in the production environment due to hardware failure, electromagnetic interference, or transmission errors. Data validation ensures the integrity and reliability of input data.

[0070] The validated multimodal image data and validated ambient light intensity are denoised and standardized respectively to obtain the validated multimodal image data and ambient light intensity. Denoising is performed to eliminate random noise and preserve the true signal. Adaptive filtering, such as Wiener filtering or deep learning denoising, is used for image data denoising. Denoising improves the signal-to-noise ratio; for example, after denoising infrared thermal imaging, the temperature resolution is improved from ±2℃ to ±0.5℃. Ambient light intensity is denoised using moving average or wavelet thresholding. Standardization maps the data to a uniform dimension or distribution, such as normalizing to the [0,1] interval. Image data is normalized using pixel values, such as subtracting the mean and dividing by the standard deviation; ambient light intensity is normalized using Min-Max, such as mapping 0-1000 Lux to 0-1. Standardization effectively eliminates dimensional differences.

[0071] Please refer to some embodiments of this application as well. Figure 3 Step S300: The step of comparing the tested ambient light intensity with the preset light intensity to obtain the ambient light coefficient includes:

[0072] S310: Compare the tested ambient light intensity with the preset light intensity to confirm the light intensity ratio between the tested ambient light intensity and the preset light intensity. The formula for calculating the light intensity ratio is as follows:

[0073]

[0074] Where K represents the light intensity ratio, I represents the ambient light intensity after testing, and M represents the preset light intensity.

[0075] S320: After verifying the light intensity ratio, the verified light intensity ratio is used as the ambient light coefficient. Verification involves logically and physically confirming the light intensity ratio's reasonableness, such as whether the ratio falls within the range of 0.8-1.5, to determine if it conforms to the changing patterns of workshop lighting. The light intensity ratio directly reflects the degree of deviation between the current ambient light intensity and the standard light intensity. For example, K=1.2 indicates that the current ambient light intensity exceeds the standard by 20%, thus quantifying the environmental difference. Furthermore, the threshold or weight of the detection model is dynamically adjusted based on the light intensity ratio.

[0076] S330: If the light intensity ratio verification fails, the sensor acquiring the ambient light intensity of the display screen under test is re-acquired, and the multimodal image data and ambient light intensity of the display screen under test during the production process are re-acquired, generating a fault prompt message. The sensor acquiring the ambient light intensity of the display screen under test is verified by restarting the sensor or checking its status, such as checking sensor calibration deviation, power supply calibration, and communication calibration. The fault prompt message includes the error type, occurrence time, suggested handling measures, or alarm signals. Error types include sensor failure and data exceeding limits. In this embodiment, if the ratio is not between 0.8 and 1.5, it is marked as verification failure. Furthermore, by observing the ratio change, it is possible to determine whether it conforms to the workshop lighting variation pattern. Specifically, if the verification fails, a sensor self-test operation is performed, a diagnostic command is sent, and the hardware status is verified. This allows for rapid location of sensor problems, avoiding continuous data errors due to hardware failure. Ambient light intensity and multimodal image data are re-acquired, triggering the complete process of steps S100 and S200. By re-acquiring data, the process automatically resumes when the sensor experiences a brief malfunction, reducing the need for manual intervention. Simultaneously, alarm information, such as light intensity sensor malfunction (code E003), is pushed through the human-machine interface or IoT platform. The fault message includes the specific sensor number and error code, shortening maintenance response time. By filtering out transient interference from sensors, such as abnormal ratios caused by flickering lights, the system automatically adjusts the detection model when natural light enters the workshop through windows, causing sudden changes in light intensity, thus avoiding misjudgments. It also prevents erroneous model adjustments triggered by incorrect ratios, leading to missed detections or false alarms.

[0077] In some embodiments of this application, if the light intensity ratio verification fails, the process proceeds after verifying the sensor that acquires the ambient light intensity of the display screen under test, re-acquiring the multimodal image data and ambient light intensity of the display screen under test during the production process, and generating a fault prompt message.

[0078] If the recalculated light intensity ratio fails again, a result indicating the display screen under test is deemed unqualified is generated. Specifically, if the sensor self-test detects a fault, such as a calibration offset exceeding a threshold, a prompt message is generated, indicating that the light intensity sensor needs calibration. Simultaneously, if the recalculated light intensity ratio still exceeds a reasonable range (e.g., less than 0.8, greater than 1.5, or not conforming to a time series pattern), the display screen is determined to have a defect that cannot be compensated for by ambient light adjustment, such as a detached reflective layer or a backlight module malfunction. The display screen is then marked as defective and entered into the manufacturing execution system, generating data including the defect type, occurrence time, and associated data (images and light intensity records). The defect types include abnormal reflection and uneven backlighting. In this embodiment, after two failed verifications, sensor problems can be ruled out, and the display screen's own defects can be confirmed, such as a batch of backlight film material having substandard light transmittance, achieving accurate identification of display screen manufacturing defects. Simultaneously, it also reduces the detection time for abnormal reflection or uneven backlighting, enabling rapid detection.

[0079] Please refer to some embodiments of this application as well. Figure 4 Step S400: Adjusting the preset image detection model using the ambient light coefficient, and confirming the adjusted image detection model includes the following steps:

[0080] S410: Adjust the preset image detection model using the ambient light coefficient to obtain a preliminary image detection model. The preset image detection model is a pre-trained display screen fault detection model based on standard lighting conditions. The preliminary image detection model is an adjusted model incorporating the ambient light coefficient; the parameters in the preliminary image detection model are modified to adapt to the current lighting conditions. Specifically, the reflectivity detection threshold in the preset image detection model is multiplied by the ambient light coefficient to increase the threshold under high light intensity and suppress noise. The weights of the backlight detection layer in the preset image detection model are multiplied by the derivative of the ambient light coefficient to compensate for the impact of light intensity changes on backlight uniformity.

[0081] S420: The preliminary image detection model is validated using multimodal image sample data to obtain the validation results. The multimodal image sample data includes display screen image samples in visible light, infrared thermal imaging, and multispectral modes, covering different defect types (such as scratches or short circuits) and lighting conditions. In this embodiment, an independent test set is used, containing 20% ​​data not used in training, to evaluate the performance of the preliminary image detection model. The test verifies whether the preliminary image detection model can accurately identify preset defects such as scratches and short circuits in the display screen. Simultaneously, it checks whether the performance fluctuation of the preliminary image detection model is within acceptable limits under different ambient light coefficients (e.g., 0.8, 1.0, 1.2, or 1.5). It also determines whether the detection results for visible light, infrared, and multispectral light are consistent to avoid misjudgments caused by single-modal noise. This ensures that the adjusted model meets the detection requirements.

[0082] S430: If the verification result of the preliminary image detection model indicates that it is qualified, then the qualified preliminary image detection model will be used as the adjusted image detection model. Specifically, after verification, the qualified preliminary image detection model will be deployed to replace the original model or as a candidate model for dynamic switching. The performance indicators of the preliminary image detection model on the test set are used to determine whether it meets the detection requirements. This also ensures the consistency of detection standards across different batches and time periods.

[0083] S440: If the verification result of the preliminary image detection model indicates failure, the training sample size of the preliminary image detection model is increased to obtain a new preliminary image detection model for re-verification. The training sample size refers to the amount of data used to train the preliminary image detection model. Increasing the sample size requires covering different lighting conditions (such as high light and low light) and defect types. Specifically, more defect samples under different lighting conditions are collected, especially cases that cause the current model to fail, such as microcracks missed under high light intensity. The expanded sample size is used to retrain the model, focusing on optimizing the detection capability of failed cases, such as increasing the weight of microcrack samples. The retrained model is evaluated using a new test set to ensure performance meets standards. By covering more edge cases, such as extreme lighting and rare defects, the accuracy of the preliminary image detection model in unknown scenarios is improved. This significantly improves the accuracy of display production fault detection and production line adaptability.

[0084] In some embodiments of this application, the step of using the adjusted image detection model to detect multimodal image data and obtaining the detection results of the display screen under test during production includes:

[0085] The multimodal image data is fused to obtain fused data. The fusion process integrates the multimodal data through pixel overlay, feature concatenation, or methods such as voting fusion to extract more comprehensive defect features. Specifically, pixel overlay involves superimposing multimodal images at the pixel level, such as overlaying visible light and infrared images into multi-channel data, preserving the original spatial information. Feature concatenation extracts features from each modality separately, such as edge features in visible light and temperature gradient features in infrared, and then concatenates them into a comprehensive feature vector. Voting fusion combines the detection results of each modality independently after detection, through voting or weighted averaging; for example, if a scratch is detected in visible light or a temperature anomaly is detected in infrared, it is comprehensively judged as a defect. The fused data is the comprehensive data after fusion processing, containing multimodal defect features, such as visible light scratch features and infrared temperature anomaly features. The fused data covers more comprehensive defect features, improving the false negative rate, and the complementary nature of the multimodal data can filter single-modal noise, achieving noise suppression.

[0086] The adjusted image detection model is used to detect the fused data, obtaining the detection results of the display screen during production. The detection results are the analysis results of the image detection model on the fused data, including information such as defect type (e.g., scratches, short circuits), location (e.g., upper left corner of the screen), and severity (e.g., mild, severe). In this embodiment, complementary features of multimodal data are extracted through fusion processing to form a more complete defect profile. The adjusted image detection model is used to analyze the fused data and output high-confidence detection results, significantly improving the detection results of the display screen.

[0087] In some embodiments of this application, the step of fusing the multimodal image data to obtain fused data includes:

[0088] Based on the multimodal image data, the data type within the multimodal image data is identified. The data type refers to the specific modality of the multimodal image data, such as visible light images, infrared thermal imaging images, and multispectral images. In this embodiment, visible light images are identified through RGB channel distribution and spatial resolution (e.g., 1080P). Infrared thermal imaging images are identified through temperature range (e.g., 20-60℃) and radiometric characteristics. Multispectral images are identified through band coverage (e.g., ultraviolet 350nm, visible light 550nm, near-infrared 850nm). This avoids misprocessing infrared data as visible light data (e.g., incorrect application of edge detection algorithms).

[0089] Based on data type and preset fusion schemes, a data fusion scheme is determined. The preset fusion schemes are pre-defined fusion strategies corresponding to different combinations of data types; for example, visible light and infrared use feature-level fusion, while multispectral data uses data-level fusion. The specific fusion method determined by the data type and preset scheme includes data-level methods (e.g., pixel overlay), feature-level methods (e.g., feature stitching), and decision-level methods (e.g., voting fusion). Visible light and infrared use feature-level fusion, such as extracting visible light edge features and infrared temperature gradient features before stitching. Multispectral data uses data-level fusion, such as overlaying ultraviolet, visible, and near-infrared images into multi-channel data. A fusion strategy suitable for the data type is selected from a preset scheme library, such as data-level, feature-level, or decision-level fusion.

[0090] The data fusion scheme described above is used to fuse the multimodal image data, resulting in fused data. Specifically, the fused data is a comprehensive dataset after fusion processing, containing multimodal defect features such as visible light scratch features, infrared temperature anomaly features, and multispectral material composition features. The fused data enables the model to handle complex defects. Data-level fusion involves overlaying multimodal images at the pixel level, such as combining visible light RGB channels with an infrared temperature channel to form four-channel data. Feature-level fusion extracts features from each modality separately, such as HOG features for visible light and LBP features for infrared, and then concatenates them into a comprehensive feature vector. Feature-level fusion reduces data dimensionality and lowers the model's inference time. Decision-level fusion combines the results of independent detection of each modality through weighted voting; for example, if visible light detects scratches and infrared detects temperature anomalies, a combined judgment is made that it is a defect. The complementary nature of the multimodal data can filter out single-modal noise; for example, reflective noise in visible light images is corrected by infrared data.

[0091] In this embodiment, by analyzing the physical characteristics of multimodal data, each modal data is determined. A fusion strategy matching the data type is selected from a preset scheme library, and the fusion scheme is executed to generate fused data containing multi-dimensional defect features, providing high-quality input for subsequent detection models. The fused data improves the model's detection rate for complex defects. The integration of fusion processing and model detection significantly reduces detection time; simultaneously, the fusion of the high resolution of visible light and the high sensitivity of infrared light enables the detection of micro-cracks invisible to the human eye, achieving the detection of micro-defects. Multispectral data-level fusion can identify fluorescent coating peeling and detect material defects in displays.

[0092] In some embodiments of this application, after the step of using the adjusted image detection model to detect multimodal image data and obtaining the detection result of the display screen under test during production, the method further includes:

[0093] If the test result indicates that the display screen is in normal production, an identifier indicating that the display screen is in normal production is generated, and the test result is fed back to the display screen production equipment. The test result indicating normal production means that the adjusted image detection model detected no faults or defects in the display screen, such as scratches, short circuits, or abnormal reflections, indicating that the display screen meets the preset quality standards. The identifier indicating that the display screen is in normal production is a unique marker identifying the quality status of the display screen. The marker is recorded using a QR code, RFID tag, or database, and its content includes information such as the test time, result, and equipment ID. After the identifier is generated, it is stored in the manufacturing execution system. Furthermore, in this embodiment, the test result can also be pushed to production equipment, such as assembly lines and backlight module adjustment stations, through an IoT platform to maintain the current operating parameters of the equipment used to produce the display screen. By binding the identifier to the display screen, full-chain verification is supported.

[0094] If the detection result indicates a production abnormality in the display screen, an identifier for the abnormality is generated, and the detection result and the corresponding preset improvement plan are fed back to the display screen's production equipment. Specifically, the detection result indicating a production abnormality means that the image detection model has detected at least one fault or defect in the display screen, such as uneven backlighting or material detachment, which does not meet the display screen's production quality standards. The identifier for the abnormality in the display screen marks the display screen as unqualified and is associated with information such as the defect type (e.g., mild, moderate, or severe) for uneven backlighting. The preset improvement plan is a pre-set solution for common fault types, such as adjusting the backlight intensity, recalibrating the light intensity sensor, or changing the material batch. The identifier for the abnormality in the display screen can be stored in a database or knowledge base. Improvement plan matching involves querying the corresponding defect solution from the preset knowledge base; for example, the solution for uneven backlighting is to adjust the backlight module voltage. In this embodiment, the detection result and improvement plan are pushed to the production equipment, such as the backlight module control unit, through an IoT platform. The defect information in the identifier helps engineers quickly locate the problem; for example, if uneven backlighting occurs, it prompts that the equipment at that workstation needs to be checked. Improved solutions automatically trigger adjustments to equipment parameters, such as backlight voltage, reducing the need for manual intervention. Simultaneously, the accumulation of abnormal data and improvement solutions forms big data, driving process improvements; for example, analyzing 1000 cases of uneven backlighting can optimize backlight module design.

[0095] In this embodiment, the display screen is marked as qualified or unqualified based on the output of the image detection model, allowing users to quickly understand the production status of the display screen. Simultaneously, if the detection result indicates an abnormality in display screen production, a preset improvement plan is implemented to increase the self-recovery rate of equipment failures, enabling rapid resolution of the problem. This upgrades the quality control of display screen production from post-production inspection to real-time adjustment, significantly improving production efficiency and product quality.

[0096] In another preferred embodiment based on the above embodiments, see [reference] Figure 5 As shown, this application also provides a fault detection system for display screen production based on data feedback, used to apply the fault detection method for display screen production based on data feedback as described in any of the above claims, including an acquisition module 510, an inspection module 520, a comparison module 530, a confirmation module 540, and a detection module 550.

[0097] The acquisition module 510 is used to acquire multimodal image data and ambient light intensity of the display screen under test during the production process.

[0098] The verification module 520 is used to perform data verification processing on the multimodal image data and ambient light intensity to obtain the verified multimodal image data and ambient light intensity.

[0099] The comparison module 530 is used to compare the tested ambient light intensity with the preset light intensity to obtain the ambient light coefficient.

[0100] The confirmation module 540 is used to adjust the preset image detection model using the ambient light coefficient and confirm the adjusted image detection model.

[0101] The detection module 550 is used to detect multimodal image data using the adjusted image detection model to obtain the detection results of the display screen under test during production.

[0102] Understandably, in this embodiment, the acquisition module 510 acquires multimodal image data and ambient light intensity of the display screen under test during the production process. Acquiring multimodal image data improves the coverage of defect detection, and ambient light intensity avoids the limitations of measuring only external light intensity, thus ensuring the accuracy of detection. The inspection module 520 performs data verification processing on the multimodal image data and ambient light intensity to obtain the verified multimodal image data and ambient light intensity. The inspection module 520 cleans and standardizes the multimodal image data and ambient light intensity to ensure the accuracy and consistency of the data. The comparison module 530 compares the verified ambient light intensity with the preset light intensity to obtain the ambient light coefficient. This quantifies the illumination difference. Simultaneously, in the event of verification failure, a sensor self-test is triggered to avoid detection errors due to hardware failure. The confirmation module 540 uses the ambient light coefficient to adjust the preset image detection model and confirms the adjusted image detection model. Furthermore, the preset image detection model is dynamically adjusted to improve the detection accuracy of the adjusted image detection model for the display screen. The detection module 550 uses the adjusted image detection model to detect multimodal image data and obtain the detection results of the display screen during production. This realizes intelligent detection of display screen production faults and significantly improves the detection efficiency and quality of the display screen.

[0103] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fault detection method for display screen production based on data feedback, characterized in that, include: Acquire multimodal image data and ambient light intensity of the display screen under test during the production process; The multimodal image data and ambient light intensity are subjected to data verification processing to obtain the verified multimodal image data and ambient light intensity. The ambient light intensity after testing is compared with the preset light intensity to obtain the ambient light coefficient; The preset image detection model is adjusted using the ambient light coefficient, and the adjusted image detection model is confirmed. The adjusted image detection model is used to detect multimodal image data to obtain the detection results of the display screen under test during production. The steps for acquiring multimodal image data and ambient light intensity of the display screen under test during the production process include: Acquire initial multimodal image data of the display screen under test during the production process, including visible light image data, infrared thermal imaging data, and multispectral image data; and acquire external light intensity, screen reflectance coefficient, and backlight intensity of the display screen under test. Based on the external light intensity, screen reflectivity, and backlight intensity, the initial ambient light intensity is determined, wherein the formula for calculating the initial ambient light intensity is: D represents the initial ambient light intensity, Z represents the external light intensity, X represents the reflectance coefficient, and Y represents the backlight intensity. The initial data of the multimodal image and the initial intensity of the ambient light are time-aligned to obtain the multimodal image data and the ambient light intensity; The step of comparing the tested ambient light intensity with a preset light intensity to obtain the ambient light coefficient includes: The tested ambient light intensity is compared with the preset light intensity to confirm the light intensity ratio between the tested ambient light intensity and the preset light intensity; After verifying the light intensity ratio, the qualified light intensity ratio is taken as the ambient light coefficient. If the light intensity ratio verification fails, the sensor that acquires the ambient light intensity of the display screen under test is verified, and the multimodal image data and ambient light intensity of the display screen under test during the production process are reacquired, and a fault prompt message is generated.

2. The fault detection method for display screen production based on data feedback according to claim 1, characterized in that, The steps for performing data verification processing on the multimodal image data and ambient light intensity to obtain verified multimodal image data and ambient light intensity include: The multimodal image data and ambient light intensity are verified separately to obtain verified multimodal image data and verified ambient light intensity. The validated multimodal image data and validated ambient light intensity are denoised and standardized respectively to obtain the validated multimodal image data and ambient light intensity.

3. The fault detection method for display screen production based on data feedback according to claim 1, characterized in that, If the light intensity ratio verification fails, then after verifying the sensor that acquires the ambient light intensity of the display screen under test, reacquire the multimodal image data and ambient light intensity of the display screen under test during the production process, and generate a fault prompt message; If the recalibrated light intensity ratio fails again, a result indicating that the display screen under test has failed the test will be generated.

4. The fault detection method for display screen production based on data feedback according to claim 1, characterized in that, The steps for adjusting the preset image detection model using the ambient light coefficient and confirming the adjusted image detection model include: The ambient light coefficient is used to adjust the preset image detection model to obtain a preliminary image detection model; The preliminary image detection model was validated using multimodal image sample data to obtain the validation results of the preliminary image detection model. If the verification result of the preliminary image detection model indicates that it is qualified, the qualified preliminary image detection model will be used as the adjusted image detection model. If the verification result of the preliminary image detection model is unqualified, the training sample size of the preliminary image detection model is increased, and a new preliminary image detection model is obtained and then re-verified.

5. The fault detection method for display screen production based on data feedback according to claim 1, characterized in that, The steps for using the adjusted image detection model to detect multimodal image data and obtain the detection results of the display screen under test during production include: The multimodal image data is fused to obtain fused data; The adjusted image detection model is used to detect the fused data to obtain the detection results of the display screen under test during production.

6. The fault detection method for display screen production based on data feedback according to claim 5, characterized in that, The steps for fusing the multimodal image data to obtain fused data include: Based on the multimodal image data, the data type in the multimodal image data is identified; Based on the data type and the preset fusion scheme, determine the data fusion scheme; The data fusion scheme is used to fuse the multimodal image data to obtain fused data.

7. The fault detection method for display screen production based on data feedback according to claim 1, characterized in that, After the step of using the adjusted image detection model to detect multimodal image data and obtaining the detection results of the display screen under test during production, the method further includes: If the test result indicates that the display screen is in normal production, an indicator that the display screen under test is in normal production is generated, and the test result is fed back to the display screen production equipment; If the test result indicates a production abnormality in the display screen, an identifier for the production abnormality of the display screen under test is generated, and the test result and the corresponding preset improvement plan are fed back to the display screen production equipment.

8. A fault detection system for display screen manufacturing based on data feedback, used to apply the fault detection method for display screen manufacturing based on data feedback as described in any one of claims 1-7, characterized in that, include: The acquisition module is used to acquire multimodal image data and ambient light intensity of the display screen under test during the production process; The verification module is used to perform data verification processing on the multimodal image data and ambient light intensity to obtain the verified multimodal image data and ambient light intensity. The comparison module is used to compare the tested ambient light intensity with a preset light intensity to obtain the ambient light coefficient; The confirmation module is used to adjust the preset image detection model using the ambient light coefficient and confirm the adjusted image detection model. The detection module is used to detect multimodal image data using the adjusted image detection model to obtain the detection results of the display screen under test during production.

Images