Fault detection method and system for display screen production based on data feedback

Abstract

The invention relates to the technical field of fault monitoring, in particular to a fault detection method and system for display screen production based on data feedback. The method comprises the following steps: acquiring multi-modal image data and ambient light intensity of a display screen to be tested in a production process; performing data verification processing on the multi-modal image data and the ambient light intensity to obtain the multi-modal image data and the ambient light intensity after verification; comparing the detected ambient light intensity with preset light intensity to obtain an ambient light coefficient; adjusting a preset image detection model by using the ambient light coefficient, and confirming an adjusted image detection model; and detecting the multi-modal image data by using the adjusted image detection model to obtain a detection result of the display screen to be detected during production. The invention aims to solve the problem that the existing detection method is easily interfered by the environment, so that the detection precision of the display screen is low.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fault monitoring, in particular to a display screen production fault detection method and system based on data feedback. BACKGROUND

[0002] Display screens are devices for visual information output, and are widely used in computers, televisions and mobile phones and other electronic products. With the development of technology, display screens have transitioned from CRT display screens to more advanced LCD display screens and LED display screens.

[0003] At present, display screens are prone to abnormal problems such as screen, white screen, color deviation, bright spots, dark spots, uneven brightness, scratches or cracks during production. Fault detection of display screens during production is a key link to ensure product quality, reduce production cost, meet industry standards and improve user experience. However, when detecting the faults of the display screen, the prior art relies on contrast detection, which is easily disturbed by the environment and cannot detect low-contrast objects such as black defects on the display screen on a black background, resulting in a significant decline in detection effect. In addition, factors such as changes in environmental light and reflection interference can also seriously affect detection accuracy. At the same time, the optical imaging system used in the prior art cannot identify small defects, resulting in low detection accuracy.

[0004] Therefore, it is necessary to design a display screen production fault detection method and system based on data feedback to solve the problems existing in the current technology. SUMMARY

[0005] In view of this, the present application provides a display screen production fault detection method and system based on data feedback, aiming to solve the problem that the existing detection method is easily disturbed by the environment, resulting in low detection accuracy of the display screen.

[0006] In one aspect, the present application provides a display screen production fault detection method based on data feedback, comprising:

[0007] Obtaining multi-modal image data and environmental light intensity of a display screen to be tested during production;

[0008] Data verification processing is performed on the multi-modal image data and environmental light intensity to obtain verified multi-modal image data and environmental light intensity;

[0009] Comparing the verified environmental light intensity with a preset light intensity to obtain an environmental light coefficient;

[0010] Adjusting a preset image detection model using the environmental light coefficient to confirm an adjusted image detection model;

[0011] The detection model is used to detect the multi-modal image data, and a detection result of the display screen under test in production is obtained.

[0012] Further, the step of obtaining the multi-modal image data and the ambient light intensity of the display screen under test in the production process comprises:

[0013] The multi-modal image initial data of the display screen under test in the production process is obtained, including visible light image data, infrared thermal imaging data and multi-spectral image data, and the external light intensity, the screen reflection coefficient of the display screen under test and the backlight intensity are obtained.

[0014] Based on the external light intensity, the screen reflection coefficient and the backlight intensity, the initial intensity of the ambient light is confirmed.

[0015] The multi-modal image initial data and the initial intensity of the ambient light are time-aligned to obtain the multi-modal image data and the ambient light intensity.

[0016] Further, the step of performing data verification processing on the multi-modal image data and the ambient light intensity to obtain the verified multi-modal image data and the ambient light intensity comprises:

[0017] The multi-modal image data and the ambient light intensity are respectively verified to obtain the verified multi-modal image data and the verified ambient light intensity.

[0018] The verified multi-modal image data and the verified ambient light intensity are respectively denoised and standardized to obtain the verified multi-modal image data and the ambient light intensity.

[0019] Further, the step of comparing the verified ambient light intensity with the preset light intensity to obtain the ambient light coefficient comprises:

[0020] The verified ambient light intensity is compared with the preset light intensity to confirm the light intensity ratio between the verified ambient light intensity and the preset light intensity.

[0021] After the light intensity ratio is verified, the verified light intensity ratio is taken as the ambient light coefficient.

[0022] If the light intensity ratio verification is unqualified, the sensor for obtaining the ambient light intensity of the display screen under test is verified, the multi-modal image data and the ambient light intensity of the display screen under test in the production process are re-obtained, and a fault prompt information is generated.

[0023] Further, if the light intensity ratio verification is unqualified, the sensor for obtaining the ambient light intensity of the display screen under test is verified, the multi-modal image data and the ambient light intensity of the display screen under test in the production process are re-obtained, and a fault prompt information is generated.

[0024] If the re-determined light intensity ratio fails the verification again, a result of the detected display screen failing the detection is generated.

[0025] Further, the preset image detection model is adjusted using the ambient light coefficient, and the step of confirming the adjusted image detection model comprises:

[0026] The preset image detection model is adjusted using the ambient light coefficient to obtain an image detection preliminary model;

[0027] The image detection preliminary model is verified using multi-modal image sample data to obtain a verification result of the image detection preliminary model;

[0028] If the verification result of the image detection preliminary model indicates that it is qualified, the image detection preliminary model that passes the verification is taken as the adjusted image detection model;

[0029] If the verification result of the image detection preliminary model indicates that it is unqualified, the training sample amount of the image detection preliminary model is increased, and the new image detection preliminary model is re-verified after being obtained.

[0030] Further, the step of detecting the multi-modal image data using the adjusted image detection model to obtain the detection result of the display screen in production comprises:

[0031] The multi-modal image data is fused to obtain fused data;

[0032] The fused data is detected using the adjusted image detection model to obtain the detection result of the display screen in production.

[0033] Further, the step of fusing the multi-modal image data to obtain fused data comprises:

[0034] Based on the multi-modal image data, a data type in the multi-modal image data is confirmed;

[0035] Based on the data type and a preset fusion scheme, a data fusion scheme is determined;

[0036] The multi-modal image data is fused using the data fusion scheme to obtain the fused data.

[0037] Further, after the step of detecting the multi-modal image data using the adjusted image detection model to obtain the detection result of the display screen in production, the method further comprises:

[0038] If the detection result indicates that the display screen is produced normally, an identification that the display screen is produced normally is generated, and the detection result is fed back to a production device of the display screen.

[0039] If the detection result indicates that the display screen production is abnormal, an identifier of the display screen to be detected is generated, and the detection result and a preset improvement scheme corresponding to the detection result are fed back to the production equipment of the display screen.

[0040] Compared with the prior art, the beneficial effects of the present application are that: by acquiring the multi-modal image data and the ambient light intensity of the display screen to be detected in the production process, comprehensive multi-modal image data is collected to provide a basis for subsequent detection. The multi-modal image data and the ambient light intensity are subjected to data verification processing to obtain verified multi-modal image data and ambient light intensity; thereby ensuring the accuracy and consistency of the data and improving the reliability of subsequent detection. The verified ambient light intensity is compared with the preset light intensity to obtain the ambient light coefficient; the ambient light coefficient is used to quantify the influence of the current light on the detection. The preset image detection model is adjusted using the ambient light coefficient to confirm the adjusted image detection model; then the model is fed back with data using the ambient light coefficient to ensure that the model can accurately detect under different light, the multi-modal image data is detected using the adjusted image detection model with higher precision to obtain the detection result of the display screen to be detected during production. Then the defects of the display screen can be accurately identified to ensure the quality of the produced display screen. At the same time, the data is fed back through the ambient light intensity, so that the preset image detection model adapts to the detection environment of different display screens, and the accuracy and efficiency of the detection are improved.

[0041] On the other hand, the present application also provides a fault detection system for display screen production based on data feedback, which is used for applying the fault detection method for display screen production based on data feedback as described in any one of the above, comprising:

[0042] The acquisition module is configured to acquire multi-modal image data and ambient light intensity of a display screen to be detected in a production process.

[0043] The verification module is configured to perform data verification processing on the multi-modal image data and the ambient light intensity to obtain verified multi-modal image data and ambient light intensity.

[0044] The comparison module is configured to compare the verified ambient light intensity with a preset light intensity to obtain an ambient light coefficient.

[0045] The confirmation module is configured to adjust a preset image detection model using the ambient light coefficient to confirm an adjusted image detection model.

[0046] The detection module is configured to detect the multi-modal image data using the adjusted image detection model to obtain a detection result of the display screen to be detected during production.

[0047] It can be understood that the above-mentioned data feedback-based fault detection method and system for display screen production have the same beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0048] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is made with reference to the accompanying drawings.

[0049] Figure 1 A flowchart of the data feedback-based fault detection method for display screen production provided by the embodiments of the present application is shown in

[0050] Figure 2 A flowchart of S100 in the data feedback-based fault detection method for display screen production provided by the embodiments of the present application is shown in

[0051] Figure 3 A flowchart of S300 in the data feedback-based fault detection method for display screen production provided by the embodiments of the present application is shown in

[0052] Figure 4 A flowchart of S400 in the data feedback-based fault detection method for display screen production provided by the embodiments of the present application is shown in

[0053] Figure 5 A functional block diagram of the data feedback-based fault detection system for display screen production provided by the embodiments of the present application is shown in DETAILED DESCRIPTION

[0054] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings; however, they are not intended to limit the present disclosure to particular embodiments. Rather, the intention is to convey the concept of the present disclosure to a technical person skilled in the art. As such, the embodiments set forth herein are intended to be illustrative and not restrictive, and the scope of the present disclosure is not limited to the embodiments set forth herein. It will be understood by those skilled in the art that various modifications and changes can be made thereto without departing from the scope of the present disclosure. It is therefore intended that the present disclosure not be limited to the particular embodiments set forth herein but that the present disclosure will include all embodiments falling within the scope of the appended claims.

[0055] In some embodiments of the present application, referring to Figure 1 A data feedback-based fault detection method for display screen production, as shown in the accompanying drawings, comprises the following steps:

[0056] S100: Obtain multi-modal image data and ambient light intensity of the display screen to be tested in the production process. The multi-modal image data is image data collected by different types of sensors, covering the state information of the display screen under different physical fields. Specifically, data is collected by a multi-sensor array and a light intensity sensor, the multi-sensor array includes a visible light camera, an infrared thermal imager, and a multi-spectral camera, and the multi-modal image data and the ambient light intensity in the production process of the display screen are collected synchronously. The ambient light intensity is the intensity of the light around the display screen in the production environment, reflecting the potential impact of light on the detection of the display screen. By collecting multi-modal data, information can be complementary, thereby fully depicting the state of the display screen and improving the coverage of defect detection data. For example, hidden defects are detected by infrared, surface scratches are recognized by visible light, multi-spectral imaging breaks through the contrast limit and can identify low-contrast defects such as black scratches on a black background. The ambient light intensity provides baseline data for subsequent detection, avoiding image distortion caused by excessive light or darkness.

[0057] S200: Perform data verification processing on the multi-modal image data and the ambient light intensity to obtain verified multi-modal image data and ambient light intensity. The data verification processing includes denoising and standardization operations on the multi-modal image data and the ambient light intensity to eliminate noise interference, unify the data format, and ensure data quality.

[0058] S300: Compare the verified ambient light intensity with the preset light intensity to obtain an ambient light coefficient. The preset light intensity is a standard ambient light intensity value set in advance, such as the recommended light intensity value of the existing production workshop. The ambient light coefficient is an adjustment factor that quantifies the impact of current lighting on detection. By determining the ambient light coefficient, detection omissions or false detections caused by sudden changes in lighting are avoided.

[0059] S400: Adjust the preset image detection model using the ambient light coefficient to confirm the adjusted image detection model. The preset image detection model is a deep learning model trained based on standard lighting conditions, such as the YOLO model or the Faster R-CNN model. The adjusted image detection model is an optimized model based on the ambient light coefficient, which adapts to actual lighting conditions and avoids performance degradation due to changes in lighting. For example, if the ambient light is too strong or too dark, the preset image detection model is adjusted using the ambient light coefficient, such as modifying the threshold or weight in the preset image detection model, to ensure that the model can accurately detect under different lighting conditions, thereby achieving more accurate detection of the display screen.

[0060] S500: detecting the multi-modal image data by using the adjusted image detection model to obtain a detection result of the to-be-tested display screen in production. The multi-modal fusion detection combines the temperature abnormalities of the infrared thermal image, the surface scratches of the multi-spectral imaging image, and the appearance defects of the visible light image. For example, internal cracks of the display screen can be found through the infrared thermal image, so as to accurately identify the defect type, position, and severity of the display screen through the detection result, and the like. Specifically, the defect type includes cracks, stains, overheating, and the like, so as to ensure the quality of the display screen production. At the same time, by detecting each to-be-tested display screen, the rework cost caused by missed detection is reduced, and the overall production capacity is improved.

[0061] In the embodiment, by obtaining the multi-modal image data and the ambient light intensity of the to-be-tested display screen in the production process, comprehensive multi-modal image data is collected to provide a basis for subsequent detection. The multi-modal image data and the ambient light intensity are subjected to data verification processing to obtain verified multi-modal image data and ambient light intensity, so as to ensure the accuracy and consistency of the data and improve the reliability of subsequent detection. The verified ambient light intensity is compared with the preset light intensity to obtain an ambient light coefficient, which quantifies the influence of the current light on the detection. The ambient light coefficient is used to adjust the preset image detection model to confirm the adjusted image detection model. Then, the ambient light coefficient is used for data feedback of the model to ensure that the model can accurately detect under different lightings. The multi-modal image data is detected by using the adjusted image detection model to obtain the detection result of the to-be-tested display screen in production. Then, the defects of the display screen can be accurately identified, and the quality of the produced display screen is ensured. At the same time, by using the ambient light intensity as data feedback, the preset image detection model is adapted to different detection environments of the display screen, so as to improve the accuracy and efficiency of the display screen detection.

[0062] In some embodiments of the present application, please refer to Figure 2 , step S100: the step of obtaining the multi-modal image data and the ambient light intensity of the to-be-tested display screen in the production process comprises:

[0063] S110: Obtain the multi-modal image initial data of the to-be-tested display screen in the production process, which includes visible light image data, infrared thermal imaging data, and multi-spectral image data, and obtain the external light intensity, the screen reflection coefficient of the to-be-tested display screen, and the backlight intensity. The multi-modal image initial data is the original image without processing, which contains multi-dimensional information such as visible light, infrared thermal imaging, and multi-spectral. The external light intensity is obtained by a light intensity sensor. The multi-modal data acquisition is realized by synchronously shooting the display screen by multiple sensors, for example: a visible light camera records the appearance; an infrared thermal imager monitors the temperature distribution; a multi-spectral camera captures the material characteristics. Visible light is used to identify surface defects of the display screen, such as scratches or stains. Infrared thermal imaging can detect temperature abnormalities of the display screen, such as short circuits or overheating. Multi-spectral can analyze the material composition or microstructure, such as fluorescence reaction or interlayer peeling. Using the obtained multi-modal data can cover the characteristics of the dominant surface and the hidden internal defects of the display screen, and improve the comprehensiveness of the display screen detection. At the same time, the reflection coefficient is measured by a standard light source irradiating the screen, and the ratio of reflected light intensity to incident light intensity is measured; the screen reflection coefficient is the proportion of the screen material reflecting the incident light, and the proportion is 0 to 1, to determine the influence of ambient light on the screen display. The backlight intensity is the light intensity of the display screen itself, such as the brightness of the LED backlight module, and the unit is nit. The backlight intensity is directly measured by a luminometer to measure the screen luminance. By obtaining the reflection coefficient and the backlight intensity, the screen's own optical characteristic parameters are provided for ambient light calculation, avoiding the one-sidedness of relying only on external light.

[0064] S120: Based on the external light intensity, the screen reflection coefficient, and the backlight intensity, the initial intensity of the ambient light is confirmed. The external light intensity is the ambient light independently measured in the workshop where the display screen is produced, such as the brightness of the workshop ceiling light. The initial intensity of the ambient light comprehensively considers the equivalent ambient light intensity of the screen reflection and the ambient light independently measured in the workshop. By accurately quantifying the ambient light, errors caused by only using the external light intensity are avoided, such as avoiding strong reflection interference of a high-reflective screen in weak light. At the same time, by confirming the initial intensity of the ambient light, it can be applied to different screen types, such as high-brightness screens and matte screens, as well as production scenarios such as dust-free workshops and open production lines. At the same time, the calculation formula of the initial intensity of the ambient light is as follows:

[0065] D = Z x (1 - X) + Y;

[0066] Wherein, D represents the initial intensity of the ambient light, Z represents the external light intensity, X represents the reflection coefficient, and Y represents the backlight intensity.

[0067] S130: Time align the multi-modal image initial data and the ambient light initial intensity to obtain multi-modal image data and ambient light intensity. The time alignment is achieved by timestamp synchronization or interpolation algorithm to ensure that the multi-modal data and the ambient light intensity are collected at the same time point. Thus, the multi-modal image and the ambient light intensity input by the detection model correspond to the same production moment, avoiding analysis misplacement and ensuring data consistency. Meanwhile, the time-aligned data is more consistent with the actual production process of the display screen, improving the accuracy of model training and real-time detection.

[0068] In some embodiments of the present application, the step of performing data verification processing on the multi-modal image data and the ambient light intensity to obtain verified multi-modal image data and ambient light intensity includes:

[0069] The multi-modal image data and the ambient light intensity are respectively verified to obtain qualified multi-modal image data and qualified ambient light intensity. The qualified multi-modal image data and the qualified ambient light intensity are data that pass the integrity, consistency and rationality checks. The data passes the integrity check by checking whether the image file is missing and whether the light intensity value is empty. The multi-modal data passes the consistency check by ensuring that the multi-modal data is synchronized in space and time, such as whether the shooting time difference between the infrared image and the visible light image exceeds a threshold. The data passes the rationality check by checking the pixel value range, such as whether the 8-bit image is 0-255 and whether there is dead pixel or stripe noise, and whether the data is within the physically feasible range, such as whether the backlight intensity is negative. By filtering out invalid or erroneous data, the interference in subsequent processing is reduced, and data cleaning is achieved. Through continuous data verification, the sensor state can be monitored, such as a camera that has been outputting abnormal pixels for a long time, prompting a hardware failure, and thus completing fault warning. Thus, in the production environment of the display screen, the data of the sensor may be abnormal due to hardware failure, electromagnetic interference or transmission error, such as missing pixels or sudden changes in light intensity. Through data verification, the integrity and reliability of the input data are ensured.

[0070] The qualified multimodal image data and the qualified ambient light intensity are respectively denoised and standardized to obtain the tested multimodal image data and the ambient light intensity. The denoising process is to eliminate random noise by algorithm and retain the true signal. The image data denoising adopts adaptive filtering such as Wiener filtering or deep learning denoising. The signal-to-noise ratio is improved by denoising, for example, the temperature resolution of infrared thermal imaging is improved from ±2℃ to ±0.5℃ after denoising. The ambient light intensity is denoised by using sliding average or wavelet threshold. The standardization process is to map the data to a unified dimension or distribution, such as normalized to the [0, 1] interval. The image data adopts pixel value normalization, such as subtracting the mean value and dividing by the standard deviation, and the ambient light intensity adopts Min-Max normalization, such as mapping 0-1000Lux to 0-1. The standardization effect can eliminate the dimensional difference.

[0071] In some embodiments of the present application, please refer to Figure 3 , step S300: comparing the tested ambient light intensity with the preset light intensity to obtain the ambient light coefficient, comprising:

[0072] S310: comparing the tested ambient light intensity with the preset light intensity to confirm the light intensity ratio between the tested ambient light intensity and the preset light intensity. The light intensity ratio calculation formula is as follows:

[0073]

[0074] Wherein, K represents the light intensity ratio, I represents the tested ambient light intensity, and M represents the preset light intensity.

[0075] S320: after verifying the light intensity ratio, the verified light intensity ratio is taken as the ambient light coefficient. The verification is a logical and physical reasonableness verification of the light intensity ratio, such as whether the ratio is within the range of 0.8-1.5, to determine whether it conforms to the change rule of workshop lighting. The light intensity ratio directly reflects the deviation degree of the current ambient light intensity from the standard light intensity, such as K=1.2, which means that the ambient light intensity exceeds the standard by 20%, thereby quantifying the environmental difference. Further, the threshold or weight of the detection model is dynamically adjusted through the light intensity ratio.

[0076] S330: If the light intensity ratio check fails, the sensor for acquiring the ambient light intensity of the display under test is checked, and then the multi-modal image data and the ambient light intensity of the display under test in the production process are re-acquired, and a fault prompt information is generated. The sensor for acquiring the ambient light intensity of the display under test is checked by restarting the sensor or checking the state of the sensor, such as calibrating the deviation of the sensor, power supply calibration, and communication calibration. The fault prompt information includes error type, occurrence time, suggested processing measures, text or alarm signal, etc., and the error type is, for example, sensor failure and data overrun. In this embodiment, if the ratio is not between 0.8 and 1.5, it is marked as failed. Further, by the ratio change, it can be determined whether it conforms to the workshop light change rule. Specifically, if the check fails, the sensor self-checking operation is performed, the diagnosis instruction is sent, and the hardware state is verified. The sensor problem can be quickly located, and continuous data errors caused by hardware failure can be avoided. The ambient light intensity and the multi-modal image data are re-acquired, and the complete process of steps S100 and S200 is triggered. In the case of temporary sensor abnormality, the process is automatically restored by re-acquisition, and the need for manual intervention is reduced. At the same time, the alarm information such as light intensity sensor failure is pushed through the man-machine interface or the Internet of Things platform, and the code E003 is obtained. The fault prompt information includes the specific sensor number and error code, which shortens the maintenance response time. By filtering the transient interference caused by the sensor, such as abnormal ratio caused by light flickering. When the natural light of the workshop enters through the window and causes a sudden change in light intensity, the detection model can be automatically adjusted to avoid misjudgment of the detection result. At the same time, it also avoids the model from being adjusted incorrectly due to the error ratio, which leads to missed detection or false alarm.

[0077] In some embodiments of the present application, after the step of generating a fault prompt information if the light intensity ratio check fails, the sensor for acquiring the ambient light intensity of the display under test is checked, and then the multi-modal image data and the ambient light intensity of the display under test in the production process are re-acquired.

[0078] If the re-determined light intensity ratio fails the check again, a result of the display screen under test being unqualified is generated. Specifically, if the sensor self-check finds a fault, such as a calibration offset exceeding a threshold, a prompt information is generated, such as information prompting that the light intensity sensor needs to be calibrated. At the same time, the recalculated light intensity ratio is still outside the reasonable range, such as the light intensity ratio being less than 0.8, the light intensity ratio being greater than 1.5, or not conforming to the time sequence rule; it is determined that the display screen has defects that cannot be compensated by ambient light adjustment, such as a reflective layer falling off or a backlight module failure, and the display screen under test is marked as a defective product, and is recorded in a manufacturing execution system, generating a defect type, a time of occurrence, and associated data (image and light intensity record); wherein the defect type includes reflective abnormality and backlight unevenness. In this embodiment, after two failed checks, the sensor problem can be ruled out, confirming the display screen itself defects, such as a certain batch of backlight film material not meeting the standard of light transmittance, achieving accurate identification of display screen production defects. At the same time, it can also reduce the detection time of the display screen appearing reflective abnormality or backlight unevenness, achieving rapid detection.

[0079] In some embodiments of the present application, please refer to Figure 4 , step S400: adjusting a preset image detection model using the ambient light coefficient, and confirming the step of adjusting the image detection model includes:

[0080] S410: adjusting the preset image detection model using the ambient light coefficient to obtain an image detection preliminary model. The preset image detection model is a display screen fault detection model trained based on a standard light condition. The image detection preliminary model is an adjusted model incorporating the ambient light coefficient, and the parameters in the image detection preliminary model are corrected to adapt to the current light. Specifically, the reflective detection threshold in the preset image detection model is multiplied by the ambient light coefficient to increase the threshold to suppress noise under high light intensity. The weight of the backlight detection layer in the preset image detection model is multiplied by the derivative of the ambient light coefficient to compensate for the influence of light intensity changes on backlight uniformity.

[0081] S420: Verify the image detection preliminary model using multi-modal image sample data to obtain a verification result of the image detection preliminary model. The multi-modal image sample data includes display screen image samples in visible light, infrared thermal imaging, and multispectral modalities, covering different defect types (such as scratches or short circuits) and lighting conditions. In this embodiment, an independent test set containing 20% of data not involved in training is used to evaluate the performance of the image detection preliminary model. The test image detection preliminary model can accurately identify pre-set defects such as scratches and short circuits in the display screen. At the same time, under different ambient light coefficients (such as 0.8, 1.0, 1.2, or 1.5), whether the performance fluctuation of the image detection preliminary model is within the allowable range. At the same time, it is determined whether the visible light, infrared, and multispectral detection results are consistent to avoid misjudgment caused by single modal noise. To ensure that the adjusted model meets the detection requirements.

[0082] S430: If the verification result of the image detection preliminary model indicates that it is qualified, the image detection preliminary model that passes the verification is used as the adjusted image detection model. Specifically, after passing the verification, the qualified image detection preliminary model is deployed to replace the original model or serve as a candidate model for dynamic switching. The performance indicators of the image detection preliminary model on the test set are used to determine whether the detection requirements are met. At the same time, it can also ensure that the detection standards of different batches and different time periods are unified.

[0083] S440: If the verification result of the image detection preliminary model indicates that it is unqualified, the training sample amount of the image detection preliminary model is increased, and the new image detection preliminary model is re-verified after being obtained. The training sample amount is the amount of data used to train the image detection preliminary model. When the sample amount is increased, different lighting conditions (such as high light and weak light) and defect types need to be covered. Specifically, more defect samples under different lighting conditions are collected, especially cases that cause the current model to fail, such as micro-cracks that are missed under high light intensity. The expanded samples are used to retrain the model, and the detection capability of the failed cases is optimized, such as increasing the weight of micro-crack samples. The retrained model is evaluated using a new test set to ensure that the performance meets the standards. By covering more edge cases, such as extreme lighting and rare defects, the accuracy of the image detection preliminary model in unknown scenarios is improved. This significantly improves the accuracy and line adaptability of display screen production fault detection.

[0084] In some embodiments of the present application, the step of detecting the multi-modal image data using the adjusted image detection model to obtain a detection result of the display screen under test during production includes:

[0085] The multi-modal image data is fused to obtain fused data. The fusion processing is to integrate the multi-modal data in a pixel superposition, feature splicing or voting fusion manner, so as to extract more comprehensive defect features. Specifically, the pixel superposition is to superimpose the multi-modal images at the pixel level, such as superimposing the visible light image and the infrared image into multi-channel data, so as to retain the original spatial information. The feature splicing is to extract the features of each modal image, such as the edge features of the visible light and the temperature gradient features of the infrared, and then splice them into a comprehensive feature vector. The voting fusion is to independently detect each modal image, and then combine the detection results through voting or weighted average, such as determining that there is a scratch in the visible light or a temperature anomaly in the infrared, and then comprehensively determining that there is a defect. The fused data is the comprehensive data after the fusion processing, and the fused data contains multi-modal defect features, such as visible light scratch features and infrared temperature anomaly features. The data after the fusion covers more comprehensive defect features, improves the missed detection rate, the multi-modal data complement each other to filter single-modal noise, and noise suppression is achieved.

[0086] The adjusted image detection model is used to detect the fused data to obtain a detection result of the to-be-tested display screen in production. The detection result is an analysis result of the image detection model on the fused data, and includes information such as a defect type (such as a scratch, a short circuit), a position (such as a top left corner of the screen) and a severity (such as mild, severe) and the like. In this embodiment, the complementary features of the multi-modal data are extracted through the fusion processing to form a more complete defect image. The adjusted image detection model is used to analyze the fused data, and a detection result with high confidence is output. The detection result of the display screen is significantly improved.

[0087] In some embodiments of the present application, the step of fusing the multi-modal image data to obtain fused data includes:

[0088] Based on the multi-modal image data, the data type in the multi-modal image data is confirmed. The data type is a specific modality of the multi-modal image data, such as a visible light image, an infrared thermal imaging image, a multi-spectral image and the like. In this embodiment, the visible light image is identified through an RGB channel distribution and a spatial resolution (such as 1080P). The infrared thermal imaging image is identified through a temperature range (such as 20-60℃) and a radiation measurement characteristic. The multi-spectral image is identified through a waveband coverage range (such as ultraviolet 350nm, visible light 550nm, near-infrared 850nm). It is avoided that the infrared data is mistakenly processed as visible light data (such as an error application of an edge detection algorithm).

[0089] The data fusion scheme is determined based on the data types and preset fusion schemes. The preset fusion scheme is a preset fusion strategy corresponding to different data type combinations, such as feature-level fusion for visible light and infrared and data-level fusion for multispectral. The data fusion scheme includes specific fusion methods determined according to the data types and the preset scheme, such as data-level fusion, feature-level fusion, and decision-level fusion such as voting fusion. Visible light and infrared use feature-level fusion, such as splicing after extracting visible light edge features and infrared temperature gradient features. Multispectral uses data-level fusion, such as superimposing ultraviolet, visible light, and near-infrared images into multi-channel data. The fusion strategy that adapts to the data types is selected from the preset scheme library, such as data-level, feature-level, and decision-level.

[0090] The multi-modal image data is fused using the data fusion scheme to obtain fused data. Specifically, the fused data is comprehensive data after fusion processing, and contains multi-modal defect features, such as visible light scratch features, infrared temperature anomaly features, and multispectral material composition features. The fused data enables the model to detect complex defects. Data-level fusion is to superimpose multi-modal images at the pixel level, such as visible light RGB three channels + infrared temperature channel to form four-channel data. Feature-level fusion is to extract features of each modality, such as HOG features of visible light and LBP features of infrared, and then splice them into a comprehensive feature vector. Feature-level fusion reduces the data dimension and reduces the inference time of the model. Decision-level fusion is to independently detect each modality, and then combine the results through weighted voting, such as visible light detecting scratches and infrared detecting temperature anomalies, and then comprehensively determining defects. Multi-modal data is complementary, and can filter single-modal noise, such as light reflection noise in visible light images being corrected by infrared data.

[0091] In this embodiment, by analyzing the physical characteristics of multi-modal data, each modality data is determined, a fusion strategy that adapts to the data types is selected from the preset scheme library, a fusion scheme is executed, and fused data containing multi-dimensional defect features is generated, providing high-quality input for subsequent detection models. The fused data improves the detection rate of the model for complex defects. The fusion processing is integrated with the model detection, significantly reducing the detection time; at the same time, the fusion of visible light high resolution and infrared high sensitivity can detect micro-cracks invisible to the human eye, realizing micro-defect detection. Multispectral data-level fusion can identify fluorescent coating peeling, and can detect material defects of the display screen.

[0092] In some embodiments of the present application, after the step of detecting the multi-modal image data using the adjusted image detection model to obtain the detection result of the display screen under test during production, the method further includes:

[0093] If the detection result indicates that the display screen is produced normally, an identification of the to-be-tested display screen produced normally is generated, and the detection result is fed back to a production device of the display screen. The detection result indicating that the production is normal means that no fault or defect is found in the display screen, such as no scratch, short circuit, and abnormal reflection, by the adjusted image detection model, and it is concluded that the display screen meets the preset quality standard. The identification of the to-be-tested display screen produced normally is a mark for uniquely marking the quality state of the display screen, and the mark is recorded in a two-dimensional code, an RFID tag, or a database, and the content in the mark includes information such as detection time, result, and device ID. After the identification is generated, it is stored in a manufacturing execution system. In addition, in this embodiment, the detection result can also be pushed to the production device, such as an assembly line and a backlight module adjustment station, through an Internet of Things platform, so as to maintain the current running parameters of the device for producing the display screen. By binding the identification with the display screen, full-chain inspection is supported.

[0094] If the detection result indicates that the display screen is produced abnormally, an identification of the to-be-tested display screen produced abnormally is generated, and the detection result and a preset improvement scheme corresponding to the detection result are fed back to the production device of the display screen. Specifically, the detection result indicating that the production is abnormal means that at least one fault or defect is found in the display screen, such as uneven backlight or material falling, by the adjusted image detection model, and the display screen does not meet the production quality standard. The identification of the to-be-tested display screen produced abnormally represents that the display screen is unqualified, and is associated with information such as a defect type, such as uneven backlight, and a severity, such as mild, moderate, and severe. The preset improvement scheme is a solution measure preset for a common fault type, such as adjusting the backlight intensity, recalibrating the light intensity sensor, or replacing the material batch, and the identification of the to-be-tested display screen produced abnormally can be stored in a database or a knowledge base. The improvement scheme matching is to query the solution scheme corresponding to the defect from the preset knowledge base, such as the solution scheme for uneven backlight is to adjust the voltage of the backlight module. In this embodiment, the detection result and the improvement scheme are pushed to the production device, such as a backlight module control unit, through an IoT platform. The defect information in the identification helps engineers quickly locate the problem, such as uneven backlight, and prompts that the equipment at the station needs to be checked. The improvement scheme automatically triggers the adjustment of the device parameters, such as the adjustment of the backlight voltage, to reduce the need for manual intervention. At the same time, the abnormal data and the improvement scheme accumulate to form big data, so as to drive process improvement, such as optimizing the design of the backlight module by analyzing 1000 cases of uneven backlight.

[0095] In this embodiment, according to the output of the detection result of the adjusted image detection model, the display screen is marked as qualified or unqualified, so as to facilitate the user to quickly understand the state of the production of the display screen. At the same time, if the detection result indicates that the display screen is produced abnormally, the preset improvement scheme is used to improve the self-recovery rate of the device fault, so that the abnormal problem can be quickly solved. The quality control of the production of the display screen is upgraded from post-inspection to real-time adjustment, and the production efficiency and the product quality of the display screen are significantly improved.

[0096] In another preferred mode based on the above embodiments, referring to Figure 5 The application also provides a display screen production fault detection system based on data feedback, for applying the display screen production fault detection method based on data feedback as described in any of the above, comprising an acquisition module 510, an inspection module 520, a comparison module 530, a confirmation module 540, and a detection module 550.

[0097] The acquisition module 510 is configured to acquire multi-modal image data and ambient light intensity of a display screen to be tested in a production process.

[0098] The inspection module 520 is configured to perform data verification processing on the multi-modal image data and the ambient light intensity, to obtain verified multi-modal image data and ambient light intensity.

[0099] The comparison module 530 is configured to compare the verified ambient light intensity with a preset light intensity, to obtain an ambient light coefficient.

[0100] The confirmation module 540 is configured to adjust a preset image detection model using the ambient light coefficient, to confirm an adjusted image detection model.

[0101] The detection module 550 is configured to detect the multi-modal image data using the adjusted image detection model, to obtain a detection result of the display screen to be tested in production.

[0102] It can be understood that in this embodiment, the acquisition module 510 acquires multi-modal image data and ambient light intensity of a display screen to be tested in a production process. The multi-modal image data is acquired to improve the coverage rate of defect detection, and the ambient light intensity can avoid the limitations of single external light intensity measurement. Thus, the accuracy of detection is ensured. The inspection module 520 performs data verification processing on the multi-modal image data and the ambient light intensity, to obtain verified multi-modal image data and ambient light intensity. The inspection module 520 cleanses and standardizes the multi-modal image data and the ambient light intensity, thus ensuring the accuracy and consistency of the data. The comparison module 530 compares the verified ambient light intensity with a preset light intensity, to obtain an ambient light coefficient. Thus, the light illumination difference is quantified, and at the same time, when the verification fails, a sensor self-check is triggered to avoid detection errors due to hardware failure. The confirmation module 540 adjusts a preset image detection model using the ambient light coefficient, to confirm an adjusted image detection model. Thus, the preset image detection model is dynamically adjusted, the detection accuracy of the adjusted image detection model on the display screen is improved, and the detection module 550 detects the multi-modal image data using the adjusted image detection model, to obtain a detection result of the display screen to be tested in production. The intelligentization of display screen production fault detection is realized, and the detection efficiency and detection quality of the display screen are significantly improved.

[0103] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a system or a computer program product. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can be embodied in the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) having computer usable program code embodied thereon.

[0104] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It should be understood that each flow and / or block in the flowchart and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing device, generate a means for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart block or blocks.

[0105] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufacture product including instruction means, which implement the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart block or blocks.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operational steps are performed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a means for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart block or blocks.

[0107] Finally, it should be noted that the above embodiments are merely intended to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. A method for fault detection in display production based on data feedback, characterized in that: include: Acquire multimodal image data and ambient light intensity of the display screen under test during the production process; Performing data verification processing on the multimodal image data and the ambient light intensity to obtain verified multimodal image data and the ambient light intensity; Comparing the tested ambient light intensity with a preset light intensity to obtain an ambient light coefficient; Adjusting a preset image detection model using the ambient light coefficient, and confirming the adjustment of the image detection model; The multimodal image data is detected using the adjusted image detection model to obtain a detection result of the display screen to be tested during production.

2. The display screen production fault detection method based on data feedback according to claim 1, characterized in that: The step of obtaining multimodal image data and ambient light intensity of the display screen to be tested during the production process includes: Acquire initial multimodal image data of the display screen to be tested during the production process, including visible light image data, infrared thermal imaging data, and multispectral image data, as well as obtain external light intensity, screen reflectance coefficient, and backlight intensity of the display screen to be tested; Determining an initial intensity of ambient light based on the external light intensity, the screen reflectivity, and the backlight intensity; The multimodal image initial data and the ambient light initial intensity are time-aligned to obtain multimodal image data and ambient light intensity.

3. The display screen production fault detection method based on data feedback according to claim 1, characterized in that: The step of performing data verification processing on the multimodal image data and the ambient light intensity to obtain the verified multimodal image data and the ambient light intensity includes: Performing data verification on the multimodal image data and the ambient light intensity respectively to obtain qualified multimodal image data and qualified ambient light intensity; The verified qualified multimodal image data and the verified qualified ambient light intensity are respectively subjected to denoising and standardization processing to obtain the verified multimodal image data and ambient light intensity.

4. The display screen production fault detection method based on data feedback according to claim 1, characterized in that: The step of comparing the tested ambient light intensity with the preset light intensity to obtain an ambient light coefficient includes: Comparing the tested ambient light intensity with the preset light intensity to determine a light intensity ratio between the tested ambient light intensity and the preset light intensity; After verifying the light intensity ratio, the qualified light intensity ratio is used as the ambient light coefficient; If the light intensity ratio verification fails, after verifying the sensor for obtaining the ambient light intensity of the display screen to be tested, the multimodal image data and ambient light intensity of the display screen to be tested during the production process are re-obtained, and a fault prompt message is generated.

5. The display screen production fault detection method based on data feedback according to claim 4, characterized in that: If the light intensity ratio verification fails, after verifying the sensor for obtaining the ambient light intensity of the display screen to be tested, re-obtaining the multimodal image data and ambient light intensity of the display screen to be tested during the production process, and generating a fault prompt message; If the verification of the re-determined light intensity ratio fails again, a result indicating that the display screen to be tested fails the test is generated.

6. The display screen production fault detection method based on data feedback according to claim 1, characterized in that: The preset image detection model is adjusted using the ambient light coefficient, and the step of confirming the adjustment of the image detection model includes: Using the ambient light coefficient to adjust the preset image detection model to obtain a preliminary image detection model; Verifying the preliminary image detection model using multimodal image sample data to obtain a verification result of the preliminary image detection model; If the verification result of the preliminary image detection model indicates that it is qualified, the qualified preliminary image detection model is used as the adjusted image detection model; If the verification result of the preliminary image detection model indicates failure, the training sample size of the preliminary image detection model is increased, and a new preliminary image detection model is obtained and then re-verified.

7. The display screen production fault detection method based on data feedback according to claim 1, characterized in that: The step of using the adjusted image detection model to detect multimodal image data to obtain a detection result of the display screen to be tested during production includes: performing fusion processing on the multimodal image data to obtain fused data; The fused data is tested using the adjusted image detection model to obtain a test result of the display screen to be tested during production.

8. The display screen production fault detection method based on data feedback according to claim 7, characterized in that: The step of fusing the multimodal image data to obtain fused data includes: Based on the multimodal image data, determining a data type in the multimodal image data; Determine the data fusion plan based on the data type and preset fusion plan; The multimodal image data is fused using the data fusion scheme to obtain fused data.

9. The display screen production fault detection method based on data feedback according to claim 1, characterized in that: After the step of detecting the multimodal image data using the adjusted image detection model to obtain the detection result of the display screen to be tested during production, the method further includes: If the test result indicates that the display screen is produced normally, an indicator that the display screen to be tested is produced normally is generated, and the test result is fed back to the production equipment of the display screen; If the test result indicates that the display screen is produced abnormally, an identification of the production abnormality of the display screen to be tested is generated, and the test result and the preset improvement plan corresponding to the test result are fed back to the production equipment of the display screen.

10. A display screen production fault detection system based on data feedback, used to apply the display screen production fault detection method based on data feedback according to any one of claims 1 to 9, characterized in that: include: An acquisition module is used to obtain multimodal image data and ambient light intensity of the display screen to be tested during the production process; a verification module, configured to perform data verification processing on the multimodal image data and the ambient light intensity to obtain verified multimodal image data and the ambient light intensity; A comparison module, configured to compare the tested ambient light intensity with a preset light intensity to obtain an ambient light coefficient; A confirmation module, configured to adjust a preset image detection model using the ambient light coefficient and confirm the adjusted image detection model; The detection module is used to detect the multimodal image data using the adjusted image detection model to obtain the detection result of the display screen to be tested during production.

Images