Method and apparatus for detecting display defects

Abstract

The present application discloses a method and device for detecting defects in a display; this solution acquires visible light image data and infrared thermal imaging data of the display to be detected, performs multi-modal feature extraction and fusion to generate a fused feature matrix, and performs defect recognition based on the fused feature matrix; this solution acquires visible light image data and infrared thermal imaging data, combines the data of the two modalities for defect detection, and can capture the defect features of the display more comprehensively, etc.; by combining the two types of data, the accuracy of defect detection can be effectively improved; in addition, through multi-modal feature extraction and fusion, an association relationship that can more accurately represent the edge features and thermal distribution features of pixel units is generated, and defect detection is performed based on the fused feature matrix, which can reduce false alarms and missed detections in the detection of display defects, improve the reliability of defect detection, and can effectively detect various types of defects.

Description

Technical Field

[0001] The present application relates to the technical fields of display technology and artificial intelligence technology, and particularly relates to a method and device for detecting display defects. Background Art

[0002] As an important component of electronic devices, the quality of a display directly affects the visual experience of users and the performance of the device. During the production process of a display, due to the complexity of the manufacturing process and the non-uniformity of materials, various defects may occur in the display, such as dead pixels, bright dots, dark dots, light leakage, short circuits, overheating, etc. These defects not only affect the display effect of the display, but may also cause a decline in the performance of the device or even failure. Therefore, defect detection of the display is a key link to ensure product quality and performance.

[0003] Traditional defect detection methods usually rely on manual or single-modal image data. These methods have certain limitations in detecting complex defects, resulting in relatively low accuracy and reliability of display defect detection. Summary of the Invention

[0004] The main purpose of the present application is to provide a method and device for detecting display defects, which can improve the accuracy and reliability of display defect detection.

[0005] To achieve the above object, an embodiment of the present invention provides a method for detecting display defects, the method comprising:

[0006] Obtaining visible light image data of a display to be detected and corresponding infrared thermal imaging data, the visible light image data including image information on the surface of the display to be detected;

[0007] Performing cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data;

[0008] Performing multi-modal feature extraction on the preprocessed visible light image data and the infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, the edge feature map including edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map including thermal distribution features of each pixel unit;

[0009] Performing feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, and the fusion feature in the fusion feature matrix represents the correlation between the edge feature and the thermal distribution feature of the pixel unit;

[0010] Using a pre-trained defect recognition model, defect recognition is performed on the to-be-detected display based on the fusion feature matrix to obtain the defect recognition result of the to-be-detected display.

[0011] Correspondingly, an embodiment of the present application further provides a display defect detection device, including:

[0012] An acquisition module, configured to acquire visible light image data and corresponding infrared thermal imaging data of the to-be-detected display, where the visible light image data includes image information on the surface of the to-be-detected display;

[0013] A data preprocessing module, configured to perform cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data;

[0014] A feature extraction module, configured to perform multi-modal feature extraction on the preprocessed visible light image data and the infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, where the edge feature map includes edge features of each pixel unit of the to-be-detected display, and the thermal gradient distribution map includes thermal distribution features of each pixel unit;

[0015] A feature fusion module, configured to perform feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, where the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit;

[0016] A defect recognition module, configured to perform defect recognition on the to-be-detected display based on the fusion feature matrix using a pre-trained defect recognition model to obtain the defect recognition result of the to-be-detected display.

[0017] The display defect detection method provided in the embodiment of the present application obtains visible light image data and infrared thermal imaging data of the display to be inspected, performs multimodal feature extraction and fusion, generates a fused feature matrix, and performs defect identification based on the fused feature matrix. This solution obtains visible light image data and infrared thermal imaging data, combines the two modal data for defect detection, and can more comprehensively capture the defect characteristics of the display. Visible light image data can detect surface defects such as bad pixels, bright spots, dark spots, etc., while infrared thermal imaging data can detect thermal distribution anomalies such as short circuits and overheating. The combination of the two types of data can effectively improve the accuracy of defect detection. In addition, through multimodal feature extraction and fusion, a fused feature matrix containing rich information is generated, which can more accurately characterize the correlation between the edge features of pixel units and the thermal distribution features. Defect detection based on the fused feature matrix can reduce false positives and false negatives in display defect detection, improve the reliability of defect detection, and effectively detect various types of defects. It is suitable for rapid detection needs in large-scale production environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is a schematic diagram of a display detection system according to an embodiment of the present application;

[0020] Figure 2 A flowchart of a display defect detection method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the data preprocessing process provided in the embodiment of the present application;

[0022] Figure 4 A schematic diagram of the feature processing flow provided in the embodiment of the present application;

[0023] Figure 5 A schematic structural diagram of a display defect detection device provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0026] The embodiments of the present application provide a display defect detection method and control device, which are described in detail below. Display defect detection refers to the process of detecting and identifying various defects that may occur during the production, use, or maintenance of a display through specific technical means and methods. These defects may include but are not limited to dead pixels, bright spots, dark spots, light leakage, short circuits, overheating, etc., which can affect the display effect and performance of the display. The purpose of display defect detection is to ensure the quality and performance of the display, improve the user experience, and reduce returns and repair costs caused by defects.

[0027] Bad pixel: A pixel on the monitor cannot display color normally, usually appearing as a black or white dot.

[0028] Bright spot: A pixel on the display continuously displays high brightness, usually appearing as a bright spot.

[0029] Dark spot: A pixel on the display that displays a constant low brightness, usually appearing as a dark spot.

[0030] Light leakage: The phenomenon of light leakage on the display against a dark background, usually appearing as halos on the edges or in local areas.

[0031] Short circuit: The internal circuit of the monitor is short-circuited, causing some pixels to not display normally.

[0032] Overheating: Performance degradation or damage to the monitor due to overheating during use.

[0033] The display in the embodiment of the present invention includes but is not limited to a liquid crystal display (LCD), an organic light emitting diode display (OLED), a plasma display panel (PDP), and the like.

[0034] like Figure 1 As shown, a display detection system is provided, which may include a display to be detected, a camera device, an infrared thermal imaging device and a computer device, wherein the camera device, the infrared thermal imaging device and the computer device are connected via a wired or wireless network.

[0035] The camera device is used to collect visible light image data of the display to be detected. The camera device can be a high-resolution camera or an industrial-grade camera, which sends the collected visible light image data to the computer device.

[0036] The infrared thermal imaging device is used to collect infrared thermal imaging data of the display to be detected. The infrared imaging device can be an infrared thermal imager or other device, and the collected infrared thermal imaging data is sent to the computer device.

[0037] The computer device is configured to perform cross-modal preprocessing on the visible light image data and infrared thermal imaging data, respectively, to obtain preprocessed visible light image data and infrared thermal imaging data; perform multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map and a thermal gradient distribution map of the visible light image data, wherein the edge feature map includes edge features of each pixel unit of the display to be inspected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit; perform feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fused feature matrix, wherein the fused features in the fused feature matrix represent the correlation between the edge features and thermal distribution features of the pixel units; and perform defect recognition on the display to be inspected based on the fused feature matrix using a pre-trained defect recognition model to obtain a defect recognition result for the display to be inspected. In one embodiment, the computer device may also output a display defect recognition result so that inspectors can obtain the recognition result. The defect recognition result may include detailed information such as the location, type, and severity of the display defect, helping manufacturers and users better understand and address defect issues.

[0038] The computer device in the embodiment of the present invention may be a smart phone, a camera, a desktop computer, a tablet computer, a server, or the like.

[0039] refer to Figure 2 , Figure 2 This is a flow chart of a display defect detection method provided by an embodiment of the present application. The method may be executed by a computer device, which may be a single computer device or a cluster of multiple computer devices. The computer device may be a terminal device or a server. The display defect detection method provided by an embodiment of the present application specifically includes:

[0040] S201 : Obtain visible light image data and corresponding infrared thermal imaging data of a display to be inspected, wherein the visible light image data includes image information of a surface of the display to be inspected.

[0041] The visible light image data refers to image data of the display surface acquired by a visible light camera, including visible light information of the display pixel units. For example, a high-definition image of the display surface is acquired by using an industrial-grade camera with a resolution of 4K to capture the display pixel units.

[0042] Infrared thermal imaging data refers to the temperature distribution data of the display surface obtained by an infrared thermal imager, which contains the temperature information of the display pixel units. For example, an infrared thermal imager with a thermal sensitivity of 0.01 °C is used to scan the display to obtain thermal imaging data containing the temperature distribution of pixel units.

[0043] In one embodiment, an industrial-grade camera and an infrared thermal imager can be used to capture and scan the display respectively to obtain visible light image data and infrared thermal imaging data. By obtaining high-resolution visible light image data and high-sensitivity infrared thermal imaging data, high-quality raw data is provided for subsequent multi-modal feature extraction, improving the accuracy of defect detection.

[0044] S202. Perform cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain the preprocessed visible light image data and infrared thermal imaging data.

[0045] Among them, cross-modal preprocessing refers to preprocessing data of different modalities to make them reach a consistent format and quality before feature extraction. For example, perform geometric distortion correction, brightness normalization, and denoising processing on the visible light image data, and perform temperature calibration, spatial registration, and pseudo-color mapping on the infrared thermal imaging data.

[0046] In one embodiment, performing cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively may include:

[0047] Perform image preprocessing on the visible light image data, where the image preprocessing includes at least geometric distortion correction and brightness normalization;

[0048] Perform thermal map preprocessing on the infrared thermal imaging data to obtain enhanced infrared thermal imaging data, where the thermal map preprocessing includes at least temperature calibration, spatial registration, and pseudo-color mapping;

[0049] Perform image smoothing processing on the visible light image data after image preprocessing and the infrared thermal imaging data after thermal map preprocessing respectively to remove random noise.

[0050] Several preprocessings will be introduced in detail below:

[0051] Geometric distortion correction refers to performing geometric transformation on an image to eliminate image distortion caused by camera lens distortion. This kind of distortion usually appears as the bending or twisting of the image edge, affecting the accuracy of the image and the effect of subsequent processing. In the embodiment of the present application, through geometric distortion correction, the geometric accuracy of the image can be significantly improved, making subsequent feature extraction and analysis more reliable.

[0052] For example, an image captured using a wide-angle camera may exhibit barrel distortion, where the central part of the image is magnified and the edges are shrunk. Through geometric distortion correction, such a distorted image can be restored to an image close to its true shape.

[0053] In one embodiment, the cv2.undistort() function in the OpenCV library can be used for correction. This function requires the internal parameters and distortion coefficients of the camera as inputs, and corrects the image using these parameters to eliminate distortion.

[0054] Brightness normalization refers to adjusting the brightness values of an image to a preset range to eliminate the influence of lighting conditions on the image. This can make the images have a similar brightness distribution under different lighting conditions, facilitating subsequent processing. For example, normalizing the brightness values of the image to the range [0, 1]. If the range of the brightness values of the image is large, it can be scaled to the preset range through linear transformation.

[0055] In one embodiment, the cv2.normalize() function in the OpenCV library can be used for brightness normalization. This function can scale the pixel values of the image to a specified range, such as [0, 1] or [0, 255]. Through brightness normalization, the influence of lighting conditions on the image can be reduced, improving the consistency of the image and the accuracy of subsequent processing.

[0056] Denoising processing refers to removing the noise in an image to improve the image quality. Noise usually manifests as random pixel value changes in the image, which may be caused by thermal noise of the camera sensor, environmental light interference, etc. For example, Gaussian filtering or median filtering is used to remove the noise in the image. Gaussian filtering smooths the image through convolution operations, and median filtering removes the noise by replacing the pixel value with the median in the neighborhood.

[0057] In one embodiment, the cv2.GaussianBlur() function in the OpenCV library can be used for Gaussian filtering, or the cv2.medianBlur() function can be used for median filtering. These functions can effectively remove the noise in the image while retaining the main features of the image. Through denoising processing, the quality of the image can be significantly improved, reducing the interference of noise on subsequent feature extraction and analysis.

[0058] Among them, temperature calibration refers to adjusting the temperature values in the infrared thermal imaging data to a preset range to eliminate the errors of the temperature sensor and the influence of environmental factors. This can make the temperature data more accurate and consistent. For example, normalizing the temperature values to the range [0, 1]. If the range of the temperature data is large, it can be scaled to the preset range through linear transformation.

[0059] In one embodiment, the np.normalize() function in the NumPy library can be used for temperature calibration. This function can scale the temperature values to a specified range, such as [0, 1]. Through temperature calibration, the errors of the temperature sensor and the influence of environmental factors can be reduced, and the accuracy and consistency of temperature data can be improved.

[0060] Spatial registration refers to spatially aligning data of different modalities so that they correspond at the pixel level. This enables accurate correspondence and fusion of data of different modalities in subsequent processing. For example, an affine transformation is used to align infrared thermal imaging data with visible light image data. By calculating the geometric transformation parameters of the two, the infrared thermal imaging data can be adjusted to the same coordinate system as the visible light image data.

[0061] In one embodiment, the cv2.warpAffine() function in the OpenCV library can be used for affine transformation. This function requires a transformation matrix as input, and through this matrix, the image is spatially transformed to achieve alignment. Through spatial registration, the spatial consistency of data of different modalities can be ensured, providing a reliable basis for subsequent feature fusion and analysis.

[0062] Among them, pseudo-color mapping refers to mapping single-channel infrared thermal imaging data into a pseudo-color image to enhance the visualization effect. This can make the temperature distribution more intuitive and easy to analyze. For example, the cv2.applyColorMap() function in the OpenCV library is used to map infrared thermal imaging data into a pseudo-color image. Common pseudo-color mappings include Jet, Hot, Cool, etc. The cv2.applyColorMap() function in the OpenCV library is used for pseudo-color mapping. This function can map a single-channel image into a pseudo-color image, enhancing the visualization effect. Through pseudo-color mapping, the visualization effect of infrared thermal imaging data can be significantly improved, making the temperature distribution more intuitive and easy to analyze.

[0063] Through the above cross-modal preprocessing methods, the quality and consistency of visible light image data and infrared thermal imaging data can be significantly improved. Geometric distortion correction eliminates the influence of camera lens distortion, brightness normalization and temperature calibration reduce the influence of illumination and temperature sensor errors, denoising processing improves the image quality, spatial registration ensures the spatial consistency of data of different modalities, and pseudo-color mapping enhances the visualization effect of infrared thermal imaging data. These preprocessing steps provide a high-quality data basis for subsequent multi-modal feature extraction and fusion, improving the accuracy and reliability of defect detection.

[0064] S203. Perform multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, wherein the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit.

[0065] Multimodal feature extraction refers to extracting features from data of different modalities to characterize the different characteristics of display pixel units. For example, edge features and texture features can be extracted from visible light image data, and thermal distribution features can be extracted from infrared thermal imaging data.

[0066] An edge feature map of visible light image data is a feature map extracted from visible light images using image processing techniques. It is primarily used to highlight edge information within an image. Edge information refers to regions within an image where grayscale values change significantly. These regions typically correspond to object outlines, texture boundaries, or other important features. Visible light edge feature maps are extracted from the original visible light image using specific algorithms (such as the Canny operator and Gabor filter). They effectively enhance edge details while suppressing noise and other irrelevant information.

[0067] The edge feature map consists of edge features of each pixel on the display. Edge features are regions in an image where pixel values change significantly, typically corresponding to the boundaries of objects or regions. These changes can be sudden changes in grayscale values or colors. Edge features are important in image processing and computer vision because they provide information about the outlines of objects or regions in an image, facilitating image understanding and analysis.

[0068] Visible light edge feature maps can highlight edge information in images, which is particularly important for defect detection. For example, in display defect detection, edge feature maps can highlight the edges of pixel units, helping to identify defects such as dead pixels and light leakage.

[0069] In one embodiment, edge features may be extracted by using techniques such as a Canny operator and a Gabor filter. The visible light edge feature map can enhance texture details in an image and improve image clarity and contrast.

[0070] The Canny operator is an edge detection algorithm that calculates the gradient magnitude and direction of an image, combined with dual-threshold detection and edge connectivity, to generate an edge feature map. The Gabor filter is a filter used for texture analysis that can extract texture features and edge information from images. By applying the Gabor filter to visible light images, an edge feature map containing texture details can be generated.

[0071] Among them, the thermal gradient distribution map is an image used to represent the temperature change gradient distribution in a temperature field. By calculating the temperature gradient of each pixel point in the temperature field, it visualizes the severity of temperature changes, thereby highlighting the small changes and abnormal areas in the temperature distribution. The thermal gradient distribution map can help identify and locate temperature abnormal areas, such as defects like short circuits and overheating. Through the thermal gradient distribution map, temperature abnormal areas can be detected more accurately, improving the accuracy of defect detection. The thermal gradient distribution map can include: high, medium, and low gradient areas. Specifically:

[0072] The high gradient area represents the area where the temperature changes violently, usually corresponding to temperature abnormal or defect areas, such as short circuits and overheating.

[0073] The low gradient area represents the area where the temperature changes gently, usually corresponding to the area with uniform temperature distribution.

[0074] The medium gradient area represents the area where the temperature changes moderately, usually corresponding to the transitional area of temperature distribution.

[0075] Among them, the thermal gradient distribution map can be composed of the thermal distribution characteristics of pixel units (pixel points). The thermal distribution characteristics refer to the temperature distribution characteristics of an object or scene in a thermal imaging image, reflecting the temperature differences and change situations in different areas. These characteristics can be obtained through thermal imaging technology and visually displayed in the thermal gradient distribution map. In the monitor detection scenario, the thermal gradient distribution map visualizes the temperature gradient and shows the severity of temperature changes. The high gradient area usually corresponds to temperature abnormal or defect areas, such as short circuits and overheating. The temperature abnormal area refers to the area where the temperature distribution is significantly different from the normal area. These areas usually appear as high gradient areas in the thermal gradient distribution map, and the specific location and type of defects can be determined through further analysis.

[0076] In one embodiment, referring to Figure 3 , multi-modal feature extraction is performed on the preprocessed visible light image data and infrared thermal imaging data, which may include:

[0077] S2031. Extract edge features from the preprocessed visible light image data to obtain an edge feature map.

[0078] In one embodiment, to extract accurate edge features, by extracting edge features and texture features and fusing them, the objects and regions in the image can be more comprehensively described, improving the accuracy of subsequent image analysis. The specific method is as follows:

[0079] Extract edge features from the preprocessed visible light image data to obtain edge features;

[0080] Extract texture features from the preprocessed visible light image data to obtain texture features;

[0081] Fuse the edge features with the texture features to generate an edge feature map representing edge and texture information.

[0082] In one embodiment, the Canny operator can be used to extract edge features. Specifically, a Gaussian filter is applied to smooth the image to reduce noise. Then, the gradient magnitude and direction of the image are calculated. Next, non-maximum suppression is used to refine the edges. Finally, a double-threshold method is used to determine the final edges. In a display image, the Canny operator can detect the edges between pixel units, which are formed due to the transition between pixels of different colors or brightnesses.

[0083] In one embodiment, texture feature extraction refers to extracting features from an image that reflect the arrangement and distribution rules of pixel points, which are usually used to describe the texture information of the image.

[0084] In one embodiment, Gabor filters can be applied to extract texture features. Specifically,

[0085] Create a set of Gabor filters with different directions and frequencies. Each Gabor filter kernel is defined by parameters such as its direction (θ), frequency (λ), bandwidth (γ), etc. In OpenCV, the cv2.getGaborKernel() function can be used to generate Gabor filter kernels.

[0086] Apply the generated Gabor filter kernels to the image for convolution operations. This can be achieved through the cv2.filter2D() function. Each Gabor filter kernel will highlight the texture features in the image with specific directions and frequencies.

[0087] In one embodiment, the gray-level co-occurrence matrix (GLCM) can also be used to extract texture features. The GLCM calculates the relationship between pixel values in the image and their neighboring pixel values to generate a matrix, thereby extracting statistical quantities reflecting texture features, such as contrast, correlation, energy, and entropy. For example, for a display image, the GLCM can extract the gray-level relationship between pixel points to reflect the texture features of the display.

[0088] In one embodiment, the results of multiple Gabor filters can also be fused. The maximum operation or other fusion methods can be used to merge the texture features with different directions and frequencies into a feature map.

[0089] In one embodiment, the edge map extracted by the Canny operator and the texture feature map extracted by the Gabor filter can be fused. A weighted fusion method can be used to superimpose the two feature maps to generate an edge feature map containing rich edge and texture information.

[0090] This approach effectively extracts edge features from visible light image data and generates a high-quality edge feature map, which can provide important feature information for subsequent defect identification in applications such as display defect detection.

[0091] S2032. Use the temperature field second-order derivative calculation method to extract the thermal gradient distribution characteristics of the preprocessed infrared thermal imaging data and generate a thermal gradient distribution map.

[0092] In one embodiment, in order to improve the accuracy of the thermal gradient distribution map, the following generation method can be used:

[0093] The second-order temperature derivative of each pixel in the preprocessed infrared thermal imaging data is calculated to generate a thermal gradient matrix. The tiny holes in the gradient map are filled using morphological closing operations. The significant thermal anomaly areas are extracted through double threshold segmentation to generate a thermal gradient distribution map.

[0094] For example, the second-order derivative calculation method of the temperature field can be used to analyze preprocessed infrared thermal imaging data to generate a thermal gradient matrix. This method can effectively detect subtle changes and abnormal areas in the temperature field. For example, when inspecting the thermal distribution of electronic equipment, the second-order derivative calculation method can capture abnormal temperature gradients caused by local short circuits or overheating, thereby better locating defects.

[0095] To remove noise and small temperature fluctuations, morphological closing operations are used to fill holes, and opening operations are used to remove small noisy areas. By setting two thresholds, the pixels in the thermal gradient distribution map are divided into three regions: low gradient, medium gradient, and high gradient. A thermal gradient distribution map is generated by calculating the second-order derivative of the infrared thermal imaging data. This map visually illustrates the severity of temperature changes, with high-gradient areas corresponding to locations with significant temperature changes. In thermal imaging analysis of electronic devices, thermal gradient distribution maps can clearly show overheating areas in chips or other components, providing important evidence for fault diagnosis and repair.

[0096] Applying the second-order derivative calculation method of the temperature field to preprocessed infrared thermal imaging data can generate accurate thermal gradient distribution maps. Compared with traditional methods, this method can more sensitively detect subtle temperature anomalies. By using the second-order derivative, the temperature gradient contrast in the image can be enhanced, making defective areas more prominent. This has significant advantages in display inspection. For example, when detecting overheating of display thin-film transistors (TFTs), the thermal gradient distribution map can accurately show the shape and extent of the overheated area, thereby improving inspection efficiency.

[0097] S204. Perform feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, where the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit.

[0098] Among them, the fusion feature matrix is a matrix composed of multiple fusion features, and the fusion features represent the correlation between the edge features and the thermal distribution features of each pixel unit or point in the display.

[0099] Through the effective combination of spatial alignment and cross-modal attention fusion network, the present invention can significantly improve the accuracy and robustness of display defect detection. This method can not only detect surface defects based on visible light images, such as scratches, stains, etc., but also capture potential defects based on thermal distribution, such as abnormal temperatures of pixel units, so as to achieve comprehensive detection of display defects.

[0100] In one embodiment, in order to obtain an accurate fusion feature matrix, the edge feature map and the thermal gradient distribution map can be spatially aligned; based on the cross-modal attention fusion network, feature fusion processing is performed on the aligned edge feature map and thermal gradient distribution map to obtain the fusion feature matrix. For example, feature maps of different modalities are spatially aligned to make them correspond at the pixel level. For example, the cv2.warpAffine() function in the OpenCV library is used for affine transformation to achieve spatial docking.

[0101] In one embodiment, feature points, such as corner points and edge points, can also be extracted from the edge feature map and the thermal gradient distribution map respectively. Algorithms such as the Harris corner detection algorithm or the Canny edge detection algorithm can be used to extract feature points. Then, using a feature point matching algorithm, such as the brute-force matcher or the FLANN (Fast Library for Approximate Nearest Neighbors) matcher, corresponding feature point pairs in the two images are found. Next, according to the matched feature point pairs, using a homography matrix estimation algorithm, such as the RANSAC (Random Sample Consensus) algorithm, the homography matrix between the two images is calculated. Finally, the thermal gradient distribution map is geometrically transformed using the homography matrix to align it with the edge feature map in terms of spatial position.

[0102] The present invention can achieve feature fusion by using a cross-modal attention fusion network. By calculating the correlation between edge features and thermal gradient features at different positions and channels, an attention weight map is generated, thereby forming a fused feature matrix. For example, for a certain pixel position, if both the edge feature and the thermal gradient feature show an anomaly at this position, the attention mechanism will assign higher weights to these two features, so that the fused feature pays more attention to this area. The attention weights can be normalized by the softmax function to ensure that their sum is 1. Finally, the fused feature matrix will comprehensively consider edge information and thermal distribution information, and can more accurately reflect the anomalies of the display pixel units.

[0103] Through the effective combination of spatial alignment and the cross-modal attention fusion network, the present invention can significantly improve the accuracy and robustness of display defect detection. This method can not only detect surface defects based on visible light images, such as scratches, stains, etc., but also capture potential defects based on thermal distribution, such as temperature anomalies of pixel units, thereby achieving comprehensive detection of display defects.

[0104] In one embodiment, referring to Figure 4 , the specific manner of achieving feature fusion based on the cross-modal attention fusion network includes:

[0105] S2041. Perform weighted processing on the aligned edge feature map and thermal gradient distribution map based on the cross-modal attention fusion network.

[0106] The cross-modal attention fusion network automatically learns the correlation between different modal features through the attention mechanism and assigns them different weights. The attention mechanism can generate an attention weight map by calculating the similarity or correlation between different modal features. For example, using the Scaled Dot-Product Attention method, first calculate the dot product of the query (Query) and the key (Key), then perform scaling, and finally normalize through the softmax function to obtain the attention weights.

[0107] In one embodiment, the cross-modal attention fusion network can adopt an encoder-decoder architecture. The encoder part extracts features from the edge feature map and the thermal gradient distribution map respectively to obtain their respective different-scale feature representations. For example, use a convolutional neural network (CNN) such as ResNet, VGG, etc. to extract features. The decoder part then gradually fuses these features and dynamically adjusts the contributions of each modal feature through the cross-modal attention mechanism.

[0108] In one embodiment, under the action of the attention mechanism, a weight is assigned to the feature map of each modality. This weight can be calculated through a fully connected layer or a convolutional layer. Specifically, the aligned edge feature map and the thermal gradient distribution map are respectively input into the attention network. After a series of convolutional and pooling operations, an attention weight map is generated. Then, the attention weight map is multiplied element-wise with the original feature map to obtain the weighted edge feature map and the thermal gradient distribution map.

[0109] S2042. Add the weighted edge feature map and the thermal gradient distribution map pixel by pixel to obtain a fused feature map.

[0110] Among them, adding pixel by pixel means performing a simple addition operation on the two feature maps at the pixel level. Assuming that the weighted edge feature map and the thermal gradient distribution map have the same size and number of channels, then the pixel values at their corresponding positions can be directly added to obtain the fused feature map. The mathematical formula can be expressed as:

[0111] F fusion (i,j,k)=w e* F e (i,j,k)+w t* F t (i,j,k);

[0112] Among them, F fusion represents the fused feature map, F e and F t respectively represent the weighted edge feature map and the thermal gradient distribution map, w e and w t are the corresponding weights, and (i,j,k) represents the position and channel of the pixel.

[0113] Among them, the fused feature map contains pixel-level thermal-optical correlation features, which can simultaneously reflect the correlation information between edge features and thermal distribution features. For example, if the edge feature map shows an edge change at a certain pixel position, and the thermal gradient distribution map shows a temperature anomaly at the same position, then the value of the fused feature map at this position will be larger, thus highlighting the possible defects.

[0114] Among them, the fused feature map contains pixel-level thermal-optical correlation features, and simultaneously reflects the correlation information between edge features and thermal distribution features; the thermal-optical correlation feature refers to the feature generated by combining the edge features in the visible light image with the thermal distribution features in the infrared thermal imaging data, which can characterize the correlation relationship between the edge features and the thermal distribution features of the pixel unit. This feature can simultaneously reflect the structural information and thermal information of the display pixel unit, providing more comprehensive feature information for defect recognition.

[0115] S2043. Extract and encode the fused feature map to generate a fused feature matrix.

[0116] The present invention performs further feature extraction operations on the fused feature map to capture more advanced and abstract features. In one embodiment, operations such as convolutional layers and pooling layers can be used to extract features. For example, multiple convolutional layers are used to extract features of different scales and directions, and a pooling layer is used to reduce the resolution of the feature map and extract key feature information. Then, the extracted features are encoded to generate a fused feature matrix. Encoding methods include fully connected layers, attention pooling, etc. The fully connected layer can flatten the feature map and input it into a neural network to learn the complex relationships between features. Attention pooling assigns different weights to different features to highlight important features and suppress unimportant features.

[0117] The finally generated fused feature matrix will contain the fused feature information and can better characterize the abnormal conditions of the display pixel units. This feature matrix can be used as the input of the subsequent defect recognition model for tasks such as defect classification and localization.

[0118] In one embodiment, in order to improve the accuracy of the features and thus improve the accuracy and stability of defect detection, extracting and encoding the fused feature map to generate a fused feature matrix includes:

[0119] Step 1: Decompose the fused feature map into multiple channels, and each channel corresponds to a specific thermal-optical correlation feature;

[0120] Decompose the fused feature map into multiple channels, and each channel corresponds to a specific thermal-optical correlation feature, such as the uniformity of heat distribution, the sharpness of edges, etc., for subsequent independent processing of different features. Assume that the dimension of the fused feature map is H×W×C, where H is the height, W is the width, and C is the number of channels. Each channel c (c = 1, 2,..., C) represents a specific thermal-optical correlation feature. For example, channel 1 may correspond to the high-frequency variation of heat distribution, and channel 2 may correspond to the edge intensity, etc.

[0121] Step 2: Apply a channel attention mechanism to each channel, weight the channel according to the importance of the channel, and generate a channel-weighted feature map.

[0122] The present invention weights channels according to the importance of the channels, enabling the network to pay more attention to important feature channels and suppress unimportant feature channels. Specifically, a channel attention mechanism can be used, for example, the Squeeze-and-Excitation (SE) module. First, global average pooling is performed on the feature map of each channel to obtain C feature vectors, each vector having a dimension of 1×1×C. Then, two fully connected layers and an activation function (such as ReLU and Sigmoid) are used to generate channel weights. Finally, the channel weights are multiplied element-wise with the original feature map to obtain a channel-weighted feature map.

[0123] Step 3: Apply a spatial attention mechanism to the channel-weighted feature map, weight the features of each pixel unit in the channel-weighted feature map according to the importance of the spatial position, and generate a spatially weighted feature map.

[0124] The present invention weights the features of each pixel unit in the channel-weighted feature map according to the importance of the spatial position, highlighting important spatial regions. Specifically, a spatial attention mechanism is used, such as the spatial attention module in a convolutional neural network (CNN). First, the channel-weighted feature map is subjected to feature extraction through a convolutional layer to generate a spatial attention map. The dimension of this attention map is the same as that of the channel-weighted feature map, and each pixel value represents the importance of the corresponding position. Then, the spatial attention map is multiplied element-wise with the channel-weighted feature map to obtain a spatially weighted feature map.

[0125] Step 4: Perform feature encoding on the spatially weighted feature map to generate a fused feature matrix.

[0126] In one embodiment, feature extraction and encoding techniques in a deep learning model can be used, such as the fully connected layer in a convolutional neural network (CNN), the long short-term memory network (LSTM) in a recurrent neural network (RNN), etc. The spatially weighted feature map is flattened into a one-dimensional vector, and then feature encoding is performed through multiple fully connected layers to generate a fused feature matrix.

[0127] Through the above steps, feature extraction and encoding are performed on the fused feature map, and the generated fused feature matrix can better reflect the thermal-optical correlation features of the display pixel units, thereby improving the accuracy and stability of defect detection.

[0128] S205: Use a pre-trained defect recognition model to perform defect recognition on the to-be-detected display based on the fused feature matrix, and obtain the defect recognition result of the to-be-detected display.

[0129] Among them, the defect recognition model: refers to a model used to identify defects in a display, usually based on deep learning algorithms. For example, a dual-branch deep network and a multi-task defect recognition model are used. Among them, the defect recognition result can include the defect type, defect location information, defect severity information, etc. of the display.

[0130] For example, a multi-task defect recognition model is used to generate a thermal-optical joint localization map of the defect location, and a pixel-level segmentation mask is generated through a U-Net architecture, so as to obtain the defect location information.

[0131] For example, a multi-task defect recognition model is used to classify the defect types, and four categories such as dead pixels, light leakage, short circuit, and overheating are distinguished.

[0132] In one embodiment, the defect recognition result can also include the defect severity evaluation information. For example, a multi-task defect recognition model can be used to calculate the defect severity score, and a weighted comprehensive index of the temperature deviation value, the proportion of the dead pixel area, and the texture distortion degree can be calculated through a regression model.

[0133] In one embodiment, global and local features are extracted from the fusion feature matrix to obtain global features and local features. The global features are used to characterize the macroscopic structure of the display to be detected, and the local features characterize the anomalies of local pixel units; a pre-trained defect recognition model is used to perform defect recognition on the display to be detected based on the global features and the local features.

[0134] Among them, the global feature refers to the feature extracted from the fusion feature matrix that can characterize the macroscopic structure of the display to be detected. These features usually reflect the overall characteristics of the display, such as overall brightness, color distribution, texture pattern, etc.

[0135] The local feature refers to the feature extracted from the fusion feature matrix that can characterize the anomalies of local pixel units. These features usually reflect the characteristics of local areas of the display, such as pixel anomalies, bright spots, dark spots, spots, stripes, etc.

[0136] The present invention can effectively use a pre-trained defect recognition model to perform defect recognition on the fusion feature matrix, obtain accurate defect recognition results, and thus improve the accuracy and stability of display defect detection.

[0137] In practical applications, the following methods can be adopted to specifically train and apply the defect recognition model:

[0138] First, collect a large amount of image data of monitors, including normal monitors and defective monitors, and label this image data. The labeling information includes the type, location, size, etc. of the defects. Then, preprocess the image data, such as operations like image enhancement, normalization, and cropping, to improve the robustness and generalization ability of the model. Next, generate a fused feature matrix through a feature extraction method. This matrix contains rich defect information. Further, extract global features and local features from the fused feature matrix. Global features can be extracted through methods such as global average pooling or global max pooling, and are used to characterize the overall characteristics of the monitor, such as overall brightness, color distribution, texture pattern, etc.; local features can be extracted through methods such as convolutional layers and pooling layers, and are used to characterize the abnormal conditions of local pixel units, such as pixel anomalies, bright spots, dark spots, blotches, stripes, etc.

[0139] Select a suitable defect recognition model, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer, etc., and initialize the model, setting the hyperparameters of the model, such as the learning rate, batch size, number of iterations, etc. Fuse the extracted global features and local features to generate the final feature representation. The method of feature fusion can be simple concatenation, or more complex weighted summation, attention mechanism, etc. During the model training process, input the fused feature representation into the model for forward propagation to obtain the output of the model. Calculate the loss between the model output and the actual labeling information. Commonly used loss functions include cross-entropy loss, mean squared error loss, etc. According to the loss calculation result, perform backpropagation to update the parameters of the model. Use an optimizer (such as Adam, SGD, etc.) to optimize the model parameters to accelerate the convergence of the model.

[0140] Load the pre-trained defect recognition model and input the fused feature matrix of the monitor to be detected into the model. The model makes predictions based on the input fused feature matrix and outputs the defect recognition results, which include information such as the type, location, size, etc. of the defects. Analyze the defect recognition results output by the model, extract information such as the type, location, size, etc. of the defects, and perform visual display, such as labeling the location and type of the defects on the monitor image. Finally, save the defect recognition results to a file for subsequent analysis and processing. By comparing the model prediction results with the actual labeling information, performance metrics such as the accuracy, recall rate, and F1 score of the model can be calculated to verify the defect recognition ability of the model.

[0141] As described above, the display defect detection method provided by the embodiments of the present invention obtains visible light image data and infrared thermal imaging data of the display to be detected, performs multi-modal feature extraction and fusion to generate a fusion feature matrix, and performs defect identification based on the fusion feature matrix. This solution combines visible light image data and infrared thermal imaging data to detect defects, enabling more comprehensive capture of the defect characteristics of the display. Visible light image data can detect surface defects such as dead pixels, bright pixels, and dark pixels, while infrared thermal imaging data can detect abnormal thermal distributions such as short circuits and overheating. By combining the two types of data, the accuracy of defect detection can be effectively improved. In addition, through multi-modal feature extraction and fusion, a fusion feature matrix containing rich information is generated, which can more accurately represent the correlation between the edge features and thermal distribution features of pixel units. Defect detection based on the fusion feature matrix can reduce false alarms and missed detections in display defect detection, improve the reliability of defect detection, effectively detect various types of defects, and meet the rapid detection requirements in large-scale production environments.

[0142] Correspondingly, to better implement the above method, an embodiment of the present application also provides a display defect detection device. As Figure 5 shown, the display defect detection device includes an acquisition module 301, a data preprocessing module 302, a feature extraction module 303, a feature fusion module 304, and a defect identification module 305, as follows:

[0143] The acquisition module 301 is configured to acquire visible light image data and corresponding infrared thermal imaging data of the display to be detected, and the visible light image data includes image information on the surface of the display to be detected;

[0144] The data preprocessing module 302 is configured to perform cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data;

[0145] The feature extraction module 303 is configured to perform multi-modal feature extraction on the preprocessed visible light image data and the infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data. The edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit;

[0146] The feature fusion module 304 is configured to perform feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, and the fusion features in the fusion feature matrix represent the correlation between the edge features and thermal distribution features of the pixel units;

[0147] The defect recognition module 305 is configured to perform defect recognition on the display to be detected based on the fused feature matrix by using a pre-trained defect recognition model, so as to obtain a defect recognition result of the display to be detected.

[0148] In one embodiment, the feature fusion module 304 is configured to:

[0149] Perform spatial alignment on the edge feature map and the thermal gradient distribution map;

[0150] Perform feature fusion processing on the aligned edge feature map and the thermal gradient distribution map based on a cross-modal attention fusion network to obtain the fused feature matrix.

[0151] In one embodiment, the feature fusion module 304 is configured to:

[0152] Perform weighting on the aligned edge feature map and the thermal gradient distribution map based on a cross-modal attention fusion network to obtain a weighted edge feature map and a weighted thermal gradient distribution map;

[0153] Add the weighted edge feature map and the weighted thermal gradient distribution map pixel by pixel to obtain a fused feature map, where the fused feature map contains pixel-level thermal-optical correlation features and simultaneously reflects the correlation information between the edge feature and the thermal distribution feature;

[0154] Perform feature extraction and encoding on the fused feature map to generate a fused feature matrix.

[0155] In one embodiment, the feature fusion module 304 is configured to:

[0156] Decompose the fused feature map into multiple channels, and each channel corresponds to a specific thermal-optical correlation feature;

[0157] Apply a channel attention mechanism to each channel, weight the channels according to the importance of the channels, and generate a channel-weighted feature map;

[0158] Apply a spatial attention mechanism to the channel-weighted feature map, weight the features of each pixel unit in the channel-weighted feature map according to the importance of the spatial position, and generate a spatially weighted feature map;

[0159] Perform feature encoding on the spatially weighted feature map to generate a fused feature matrix.

[0160] In one embodiment, the data preprocessing module 302 is configured to:

[0161] Perform image preprocessing on the visible light image data, where the image preprocessing at least includes geometric distortion correction and brightness normalization;

[0162] Performing thermal map preprocessing on the infrared thermal imaging data to obtain enhanced infrared thermal imaging data, wherein the thermal map preprocessing includes at least temperature calibration, spatial registration, and pseudo-color mapping;

[0163] The visible light image data after image preprocessing and the infrared thermal imaging data after thermal map preprocessing are respectively subjected to image smoothing processing to remove random noise.

[0164] In one embodiment, the feature extraction module 303 is configured to:

[0165] Perform edge feature extraction on the preprocessed visible light image data to obtain an edge feature map;

[0166] The second-order derivative calculation method of the temperature field is used to extract the thermal gradient distribution characteristics of the preprocessed infrared thermal imaging data and generate a thermal gradient distribution map.

[0167] In one embodiment, the feature extraction module 303 is configured to:

[0168] Performing edge feature extraction on the preprocessed visible light image data to obtain edge features;

[0169] Extracting texture features from the preprocessed visible light image data to obtain texture features;

[0170] The edge features are fused with the texture features to generate an edge feature map representing edge and texture information.

[0171] In one embodiment, the feature extraction module 303 is configured to:

[0172] The second-order temperature derivative of each pixel in the preprocessed infrared thermal imaging data is calculated to generate a thermal gradient matrix. The tiny holes in the gradient map are filled using morphological closing operations. The significant thermal anomaly areas are extracted through double threshold segmentation to generate a thermal gradient distribution map.

[0173] In one embodiment, the defect identification module 305 is configured to:

[0174] Performing global and local feature extraction on the fused feature matrix to obtain global features and local features, wherein the global features are used to characterize the macro structure of the display to be inspected, and the local features characterize the abnormality of the local pixel unit;

[0175] A pre-trained defect recognition model is used to perform defect recognition on the display to be inspected based on the global features and the local features.

[0176] The implementation of the above modules can be specifically referred to in the previous method embodiments, which will not be described in detail here.

[0177] It should be noted that in specific implementation, the above-mentioned modules can be combined arbitrarily and integrated into one or several modules, or can be implemented as independent entities. In addition, the above-mentioned modules can be implemented in the form of hardware or in the form of software function modules. When the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disc, etc.

[0178] As can be seen from the above, the display detection device provided by the embodiment of the present application can capture the defect features of the display more comprehensively by acquiring visible light image data and infrared thermal imaging data and combining the data of the two modalities. The visible light image data can detect surface defects such as dead pixels, bright pixels, and dark pixels, while the infrared thermal imaging data can detect abnormal thermal distributions such as short circuits and overheating; the combination of the two types of data can effectively improve the accuracy of defect detection; in addition, through multi-modal feature extraction and fusion, a fusion feature matrix containing rich information is generated, which can more accurately characterize the correlation between the edge features and thermal distribution features of pixel units. Defect detection based on the fusion feature matrix can reduce false alarms and missed detections in display defect detection, improve the reliability of defect detection, can effectively detect various types of defects, and is suitable for the rapid detection requirements in a large-scale production environment.

[0179] As Figure 6 shown, the embodiment of the present application also provides a computer device 40, which is characterized in that it includes a processor 401 and a memory 402, wherein the memory 402 stores a computer program, and when the computer program is executed by the processor 401, the processor 401 is caused to execute the steps of any one of the above-mentioned methods.

[0180] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0181] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in each embodiment of the present application, each functional unit may be integrated in a processing unit, may exist physically alone for each unit, or two or more units may be integrated in one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0182] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0183] One aspect of the embodiments of the present application provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0184] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally further includes steps or modules not listed, or optionally further includes other step units inherent to these processes, methods, devices, products, or equipment.

[0185] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0186] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0187] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for detecting display defects, characterized in that, The method includes: Obtaining visible light image data and corresponding infrared thermal imaging data of the display to be detected, where the visible light image data contains image information on the surface of the display to be detected; Performing cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data; Performing multi-modal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, where the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit; Performing spatial alignment on the edge feature map and the thermal gradient distribution map; Performing weighted processing on the aligned edge feature map and thermal gradient distribution map based on a cross-modal attention fusion network to obtain a weighted edge feature map and a weighted thermal gradient distribution map; Performing pixel-by-pixel addition on the weighted edge feature map and the weighted thermal gradient distribution map to obtain a fused feature map, where the fused feature map contains pixel-level thermal-optical correlation features and simultaneously reflects the correlation information between the edge features and the thermal distribution features; Decomposing the fused feature map into multiple channels, with each channel corresponding to a specific thermal-optical correlation feature; Applying a channel attention mechanism to each channel, weighting the channels according to the importance of the channels to generate a channel-weighted feature map; Applying a spatial attention mechanism to the channel-weighted feature map, weighting the features of each pixel unit in the channel-weighted feature map according to the importance of the spatial position to generate a spatially weighted feature map; Encoding the spatially weighted feature map to generate a fused feature matrix, where the fused features in the fused feature matrix represent the correlation relationship between the edge features and the thermal distribution features of the pixel unit; Using a pre-trained defect recognition model to perform defect recognition on the display to be detected based on the fused feature matrix to obtain a defect recognition result of the display to be detected.

2. The method according to claim 1, wherein Performing cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively, including: Performing image preprocessing on the visible light image data, where the image preprocessing at least includes geometric distortion correction and brightness normalization; Performing thermal map preprocessing on the infrared thermal imaging data to obtain enhanced infrared thermal imaging data, where the thermal map preprocessing at least includes temperature calibration, spatial registration, and pseudo-color mapping; Performing image smoothing processing on the visible light image data after image preprocessing and the infrared thermal imaging data after thermal map preprocessing respectively to remove random noise.

3. The method according to claim 2, wherein Performing multi-modal feature extraction on the preprocessed visible light image data and infrared thermal imaging data, including: Using a preset edge detection algorithm to extract edge features from the preprocessed visible light image data to obtain an edge feature map; Using a temperature field second derivative calculation method to extract the thermal gradient distribution features of the preprocessed infrared thermal imaging data to generate a thermal gradient distribution map.

4. The method according to claim 2, characterized in that, Using a preset edge detection algorithm to extract edge features from the preprocessed visible light image data to obtain an edge feature map, including: Edge feature extraction is performed on the preprocessed visible light image data to obtain edge features; Texture feature extraction is performed on the preprocessed visible light image data to obtain texture features; The edge features and the texture features are fused to generate an edge feature map representing edge and texture information.

5. The method according to claim 3, characterized in that, A thermal gradient distribution feature of the preprocessed infrared thermal imaging data is extracted by using a second-order derivative calculation method of the temperature field to generate a thermal gradient distribution map, including: Calculating the second-order temperature derivative of each pixel unit in the preprocessed infrared thermal imaging data to generate a thermal gradient matrix; filling small holes in the gradient map by using morphological closing operation; extracting significant thermal anomaly regions by double-threshold segmentation to generate a thermal gradient distribution map.

6. The method according to claim 4, wherein Defect identification of the to-be-detected display is performed by using a pre-trained defect identification model based on the fusion feature matrix, including: Global and local feature extraction is performed on the fusion feature matrix to obtain global features and local features. The global features are used to represent the macroscopic structure of the to-be-detected display, and the local features represent the anomalies of local pixel units; Defect identification of the to-be-detected display is performed by using a pre-trained defect identification model based on the global features and the local features.

7. A display defect detection device, characterized in that, Including: An acquisition module, configured to acquire visible light image data and corresponding infrared thermal imaging data of the to-be-detected display. The visible light image data contains image information on the surface of the to-be-detected display; A data preprocessing module, configured to perform cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data; ​ ​ ​ Apply the channel attention mechanism to each channel, weight the channels according to the importance of the channels, and generate a channel-weighted feature map; apply the spatial attention mechanism to the channel-weighted feature map, weight the features of each pixel unit in the channel-weighted feature map according to the importance of the spatial position, and generate a spatially weighted feature map; perform feature encoding on the spatially weighted feature map to generate a fused feature matrix, and the fused features in the fused feature matrix represent the correlation between the edge features and the heat distribution features of the pixel units. A defect recognition module for using a pre-trained defect recognition model to perform defect recognition on the display to be detected based on the fused feature matrix, and obtaining a defect recognition result of the display to be detected.

Images