Display defect detection method and device

Abstract

The invention discloses a display defect detection method and device. According to the scheme, visible light image data and infrared thermal imaging data of a to-be-detected display are obtained, multi-modal feature extraction and fusion are performed, a fusion feature matrix is generated, and defect identification is performed based on the fusion feature matrix; according to the scheme, the visible light image data and the infrared thermal imaging data are acquired, and defect detection is performed by combining data of two modes, so that defect characteristics and the like of the display can be captured more comprehensively; the accuracy of defect detection can be effectively improved through the combination of two kinds of data; besides, through multi-modal feature extraction and fusion, the incidence relation between the edge feature and the heat distribution feature which can more accurately represent the pixel unit is generated, and defect detection is carried out based on a fusion feature matrix, so that false alarm and missing alarm of defect detection of the display can be reduced, and the reliability of defect detection is improved; and various types of defects can be effectively detected.

Description

Technical Field

[0001] The present application relates to the fields of display technology and artificial intelligence technology, and in particular to a display defect detection method and device. Background Art

[0002] As an important part of electronic equipment, the quality of the display directly affects the user's visual experience and the performance of the device. During the production process of the display, due to the complexity of the manufacturing process and the unevenness of the material, various defects may occur on the display, such as dead pixels, bright spots, dark spots, light leakage, short circuit, overheating, etc. These defects not only affect the display effect of the display, but may also cause the performance of the device to decline or even fail. Therefore, defect detection of the display is a key link to ensure product quality and performance.

[0003] Traditional defect detection methods usually rely on manual or single-modality image data. These methods have certain limitations when detecting complex defects, resulting in relatively low accuracy and reliability of display defect detection. Summary of the invention

[0004] The main purpose of the present application is to provide a display defect detection method and device, which can improve the accuracy and reliability of display defect detection.

[0005] To achieve the above object, an embodiment of the present invention provides a display defect detection method, the method comprising: Acquire visible light image data and corresponding infrared thermal imaging data of the display to be detected, wherein the visible light image data includes image information of the surface of the display to be detected; Performing cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data; Performing multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, wherein the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit; Performing feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, wherein the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit; A pre-trained defect recognition model is used to perform defect recognition on the display to be detected based on the fused feature matrix to obtain a defect recognition result of the display to be detected.

[0006] Accordingly, an embodiment of the present application further provides a display defect detection device, comprising: An acquisition module, used to acquire visible light image data and corresponding infrared thermal imaging data of the display to be detected, wherein the visible light image data includes image information of the surface of the display to be detected; A data preprocessing module, used to perform cross-modal preprocessing on the visible light image data and the infrared thermal imaging data to obtain preprocessed visible light image data and infrared thermal imaging data; a feature extraction module, configured to perform multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data, to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, wherein the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit; A feature fusion module, used for performing feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, wherein the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit; The defect recognition module is used to use a pre-trained defect recognition model to perform defect recognition on the display to be detected based on the fusion feature matrix to obtain a defect recognition result of the display to be detected.

[0007] The display defect detection method provided in the embodiment of the present application obtains visible light image data and infrared thermal imaging data of the display to be detected, performs multimodal feature extraction and fusion, generates a fusion feature matrix, and performs defect identification based on the fusion feature matrix; the scheme obtains visible light image data and infrared thermal imaging data, combines the data of the two modes for defect detection, and can more comprehensively capture the defect characteristics of the display. Visible light image data can detect surface defects, such as bad pixels, bright spots, dark spots, etc., while infrared thermal imaging data can detect thermal distribution anomalies, such as short circuits, overheating, etc.; the combination of the two types of data can effectively improve the accuracy of defect detection; in addition, through multimodal feature extraction and fusion, a fusion feature matrix containing rich information is generated, which can more accurately characterize the correlation between the edge features of the pixel unit and the thermal distribution features. Defect detection based on the fusion feature matrix can reduce false positives and false negatives in display defect detection, improve the reliability of defect detection, and effectively detect various types of defects, which is suitable for rapid detection needs in large-scale production environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 is a schematic diagram of a scene of a display detection system in an embodiment of the present application; Figure 2 A flow chart of a display defect detection method provided in an embodiment of the present application; Figure 3 A schematic diagram of the data preprocessing process provided in the embodiment of the present application; Figure 4 A schematic diagram of a feature processing flow provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a display defect detection device provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0011] The embodiments of the present application provide a display defect detection method and control device, which will be described in detail below. Display defect detection refers to the process of detecting and identifying various defects that occur during the production, use or maintenance of the display through specific technical means and methods. These defects may include but are not limited to dead pixels, bright spots, dark spots, light leakage, short circuits, overheating, etc., which will affect the display effect and performance of the display. The purpose of display defect detection is to ensure the quality and performance of the display, improve user experience, and reduce returns and repair costs caused by defects.

[0012] Bad pixel: A pixel in the monitor cannot display color normally, usually appearing as a black or white dot.

[0013] Bright spot: A pixel on the display continuously displays high brightness, usually appearing as a bright spot.

[0014] Dark spot: A pixel on the display that shows a constant low brightness, usually appearing as a dark spot.

[0015] Light leakage: The phenomenon of light leakage that occurs on a display against a dark background, usually manifesting as halos at the edges or in local areas.

[0016] Short circuit: The internal circuit of the monitor is short-circuited, causing some pixels to fail to display normally.

[0017] Overheating: Performance degradation or damage to the monitor due to overheating during use.

[0018] The display in the embodiment of the present invention includes but is not limited to a liquid crystal display (LCD), an organic light emitting diode display (OLED), a plasma display panel (PDP), etc.

[0019] like Figure 1 As shown, a display detection system is provided, which may include a display to be detected, a camera device, an infrared thermal imaging device and a computer device, wherein the camera device, the infrared thermal imaging device and the computer device are connected via a wired or wireless network.

[0020] The camera device is used to collect visible light image data of the display to be detected. The camera device can be a high-resolution camera or an industrial-grade camera, which sends the collected visible light image data to the computer device.

[0021] The infrared thermal imaging device is used to collect infrared thermal imaging data of the display to be detected. The infrared imaging device can be an infrared thermal imager or other device, and the collected infrared thermal imaging data is sent to the computer device.

[0022] A computer device is used to perform cross-modal preprocessing on the visible light image data and infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data; perform multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map and a thermal gradient distribution map of the visible light image data, wherein the edge feature map includes the edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes the thermal distribution features of each pixel unit; perform feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, wherein the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit; and use a pre-trained defect recognition model to perform defect recognition on the display to be detected based on the fusion feature matrix to obtain a defect recognition result of the display to be detected. In one embodiment, the computer device can also output a display defect recognition result so that the inspection personnel can obtain the recognition result. The defect recognition result can include detailed information such as the defect location, type and severity of the display to help manufacturers and users better understand and deal with defect problems.

[0023] The computer device in the embodiment of the present invention may be a smart phone, a camera, a desktop computer, a tablet computer, a server, or the like.

[0024] refer to Figure 2 , Figure 2 The flowchart of a display defect detection method provided by an embodiment of the present application is shown in FIG. 1 . The execution subject of the method may be a computer device, which may be a computer device or a cluster of multiple computer devices, which may be a terminal device or a server, etc. The display defect detection method provided by an embodiment of the present application specifically includes: S201 . Obtain visible light image data and corresponding infrared thermal imaging data of a display to be inspected, wherein the visible light image data includes image information of a surface of the display to be inspected.

[0025] The visible light image data refers to the image data of the display surface obtained by a visible light camera, including the visible light information of the display pixel unit. For example, an industrial-grade camera with a resolution of 4K is used to shoot the display surface to obtain a high-definition image including the display pixel unit.

[0026] Infrared thermal imaging data refers to the temperature distribution data of the display surface obtained by an infrared thermal imager, including the temperature information of the display pixel units. For example, an infrared thermal imager with a thermal sensitivity of 0.01°C is used to scan the display to obtain thermal imaging data containing the temperature distribution of the pixel units.

[0027] In one embodiment, an industrial camera and an infrared thermal imager can be used to shoot and scan the display, respectively, to obtain visible light image data and infrared thermal imaging data. By obtaining high-resolution visible light image data and high-sensitivity infrared thermal imaging data, high-quality raw data is provided for subsequent multimodal feature extraction, thereby improving the accuracy of defect detection.

[0028] S202 , performing cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data.

[0029] Among them, cross-modal preprocessing refers to preprocessing data of different modalities to achieve consistent format and quality before feature extraction. For example, geometric distortion correction, brightness standardization and denoising are performed on visible light image data, and temperature calibration, spatial registration and pseudo-color mapping are performed on infrared thermal imaging data.

[0030] In one embodiment, cross-modal preprocessing of the visible light image data and the infrared thermal imaging data may include: Performing image preprocessing on the visible light image data, wherein the image preprocessing at least includes geometric distortion correction and brightness standardization; Performing thermal map preprocessing on the infrared thermal imaging data to obtain enhanced infrared thermal imaging data, wherein the thermal map preprocessing at least includes temperature calibration, spatial registration and pseudo color mapping; The visible light image data after image preprocessing and the infrared thermal imaging data after thermal image preprocessing are respectively subjected to image smoothing processing to remove random noise.

[0031] Several preprocessing steps are described in detail below: Geometric distortion correction refers to the geometric transformation of an image to eliminate image distortion caused by camera lens distortion. This distortion usually manifests itself as bending or twisting of the image edge, affecting the accuracy of the image and the effect of subsequent processing. In the embodiment of the present application, geometric distortion correction can significantly improve the geometric accuracy of the image, making subsequent feature extraction and analysis more reliable.

[0032] For example, an image taken with a wide-angle camera may have barrel distortion, where the center of the image is enlarged and the edges are shrunk. Geometric distortion correction can restore this distorted image to an image close to its true shape.

[0033] In one embodiment, the cv2.undistort() function in the OpenCV library can be used for correction. The function requires the intrinsic parameters and distortion coefficients of the camera as inputs, and corrects the image through these parameters to eliminate distortion.

[0034] Brightness normalization refers to adjusting the brightness value of an image to a preset range to eliminate the influence of lighting conditions on the image. This can make the image have similar brightness distribution under different lighting conditions, which is convenient for subsequent processing. For example, the brightness value of an image is normalized to the interval [0, 1]. If the brightness value range of an image is large, it can be scaled to a preset range through linear transformation.

[0035] In one embodiment, the cv2.normalize() function in the OpenCV library can be used to perform brightness normalization. The function can scale the pixel values ​​of an image to a specified range, such as [0, 1] or [0, 255]. By brightness normalization, the influence of lighting conditions on the image can be reduced, and the consistency of the image and the accuracy of subsequent processing can be improved.

[0036] Denoising is the process of removing noise from an image to improve image quality. Noise usually appears as random pixel value changes in an image, which may be caused by thermal noise of the camera sensor, ambient light interference, etc. For example, Gaussian filtering or median filtering is used to remove noise from an image. Gaussian filtering smoothes the image through convolution operations, and median filtering removes noise by replacing pixel values ​​with the median value in the neighborhood.

[0037] In one embodiment, the cv2.GaussianBlur() function in the OpenCV library can be used to perform Gaussian filtering, or the cv2.medianBlur() function can be used to perform median filtering. These functions can effectively remove noise from an image while retaining the main features of the image. Through denoising, the quality of the image can be significantly improved, and the interference of noise on subsequent feature extraction and analysis can be reduced.

[0038] Temperature calibration refers to adjusting the temperature values ​​in infrared thermal imaging data to a preset range to eliminate the influence of temperature sensor errors and environmental factors. This can make the temperature data more accurate and consistent. For example, the temperature value is normalized to the interval [0, 1]. If the range of temperature data is large, it can be scaled to the preset range through linear transformation.

[0039] In one embodiment, the temperature calibration can be performed using the np.normalize() function in the NumPy library. The function can scale the temperature value to a specified range, such as [0, 1]. Through temperature calibration, the error of the temperature sensor and the influence of environmental factors can be reduced, and the accuracy and consistency of the temperature data can be improved.

[0040] Spatial registration refers to spatially aligning data from different modalities so that they correspond to each other at the pixel level. This allows data from different modalities to be accurately aligned and fused in subsequent processing. For example, infrared thermal imaging data is aligned with visible light image data using affine transformation. By calculating the geometric transformation parameters of the two, the infrared thermal imaging data can be adjusted to the same coordinate system as the visible light image data.

[0041] In one embodiment, the cv2.warpAffine() function in the OpenCV library can be used to perform affine transformation. The function requires a transformation matrix as input, through which the image is spatially transformed to achieve alignment. Through spatial registration, the consistency of data of different modalities can be ensured in space, providing a reliable basis for subsequent feature fusion and analysis.

[0042] Among them, pseudo color mapping refers to mapping single-channel infrared thermal imaging data into pseudo color images to enhance the visualization effect. This can make the temperature distribution more intuitive and easy to analyze. For example, use the cv2.applyColorMap() function in the OpenCV library to map infrared thermal imaging data into pseudo color images. Common pseudo color mappings include Jet, Hot, Cool, etc. Use the cv2.applyColorMap() function in the OpenCV library for pseudo color mapping. This function can map a single-channel image into a pseudo color image to enhance the visualization effect. Through pseudo color mapping, the visualization effect of infrared thermal imaging data can be significantly improved, making the temperature distribution more intuitive and easy to analyze.

[0043] The above cross-modal preprocessing methods can significantly improve the quality and consistency of visible light image data and infrared thermal imaging data. Geometric distortion correction eliminates the influence of camera lens distortion, brightness standardization and temperature calibration reduce the influence of illumination and temperature sensor errors, denoising improves image quality, spatial registration ensures the spatial consistency of different modal data, and pseudo-color mapping enhances the visualization of infrared thermal imaging data. These preprocessing steps provide a high-quality data foundation for subsequent multimodal feature extraction and fusion, and improve the accuracy and reliability of defect detection.

[0044] S203, performing multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, wherein the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit.

[0045] Among them, multimodal feature extraction refers to extracting features from data of different modalities to characterize the different characteristics of display pixel units. For example, edge features and texture features are extracted from visible light image data, and thermal distribution features are extracted from infrared thermal imaging data.

[0046] The edge feature map of visible light image data is a feature map extracted from visible light images through image processing technology, which is mainly used to highlight the edge information in the image. Edge information refers to the areas in the image where the grayscale value changes significantly. These areas usually correspond to the outlines, texture boundaries or other important features of the object. The visible light edge feature map is extracted from the original visible light image through specific algorithms (such as Canny operator, Gabor filter, etc.), which can effectively enhance the edge details in the image while suppressing noise and other irrelevant information.

[0047] The edge feature map is composed of the edge features of each pixel of the display. Edge features refer to areas in the image where the pixel values ​​change significantly, usually corresponding to the boundaries of objects or regions. These changes can be sudden changes in grayscale values ​​or sudden changes in color. Edge features are of great significance in image processing and computer vision because they provide outline information of objects or regions in the image, which helps to understand and analyze the image.

[0048] Visible light edge feature maps can highlight edge information in images, which is particularly important for defect detection. For example, in display defect detection, edge feature maps can highlight the edges of pixel units and help identify defects such as dead pixels and light leakage.

[0049] In one embodiment, edge features may be extracted by using techniques such as a Canny operator and a Gabor filter, and the visible light edge feature map may enhance texture details in an image and improve image clarity and contrast.

[0050] Among them, the Canny operator is an edge detection algorithm that generates an edge feature map by calculating the gradient amplitude and direction of the image, combining dual threshold detection and edge connection. The Gabor filter is a filter used for texture analysis that can extract texture features and edge information in an image. By applying the Gabor filter to a visible light image, an edge feature map containing texture details can be generated.

[0051] Among them, the thermal gradient distribution map is an image used to represent the gradient distribution of temperature changes in a temperature field. It visualizes the severity of temperature changes by calculating the temperature gradient of each pixel in the temperature field, thereby highlighting small changes and abnormal areas in the temperature distribution. The thermal gradient distribution map can help identify and locate temperature abnormal areas, such as short circuits, overheating and other defects. Through the thermal gradient distribution map, temperature abnormal areas can be detected more accurately, improving the accuracy of defect detection. The thermal gradient distribution map can include: high, medium and low gradient areas, specifically: High gradient areas indicate areas where the temperature changes dramatically, usually corresponding to temperature anomalies or defect areas, such as short circuits, overheating, etc.

[0052] Low gradient regions represent areas where temperature changes are gradual and usually correspond to areas with uniform temperature distribution.

[0053] The medium gradient region indicates the area with moderate temperature change, which usually corresponds to the transition region of the temperature distribution.

[0054] Among them, the thermal gradient distribution map can be composed of the thermal distribution characteristics of pixel units (pixel points), which refers to the characteristics of the temperature distribution of objects or scenes in thermal imaging images, reflecting the temperature differences and changes in different areas. These features can be obtained through thermal imaging technology and visualized in the thermal gradient distribution map. In the display detection scenario, the thermal gradient distribution map shows the severity of temperature changes by visualizing the temperature gradient. High-gradient areas usually correspond to temperature anomalies or defective areas, such as short circuits, overheating, etc. Abnormal temperature areas refer to areas where the temperature distribution is significantly different from normal areas. These areas are usually shown as high-gradient areas in the thermal gradient distribution map, and the specific location and type of defects can be determined through further analysis.

[0055] In one embodiment, reference Figure 3 , multimodal feature extraction of preprocessed visible light image data and infrared thermal imaging data may include: S2031. Extract edge features from the preprocessed visible light image data to obtain an edge feature map.

[0056] In one embodiment, in order to extract accurate edge features, edge features and texture features may be extracted and fused to more comprehensively describe objects and regions in the image, thereby improving the accuracy of subsequent image analysis. The specific method is as follows: Extract edge features from the preprocessed visible light image data to obtain edge features; Extracting texture features from the preprocessed visible light image data to obtain texture features; The edge feature is fused with the texture feature to generate an edge feature map representing edge and texture information.

[0057] In one embodiment, the Canny operator can be used to extract edge features. Specifically, a Gaussian filter is applied to smooth the image to reduce noise. Then the gradient magnitude and direction of the image are calculated, and then the edge is refined by non-maximum suppression, and finally the double threshold method is used to determine the final edge. In a display image, the Canny operator can detect the edges between pixel units, which are formed by the transition between pixels of different colors or brightness.

[0058] In one embodiment, texture feature extraction refers to extracting features that reflect the arrangement and distribution rules of pixel points from an image. These features are usually used to describe the texture information of the image.

[0059] In one embodiment, a Gabor filter may be applied to extract texture features. Specifically, Create a set of Gabor filters with different orientations and frequencies. Each Gabor filter kernel is defined by its orientation (θ), frequency (λ), bandwidth (γ) and other parameters. In OpenCV, you can use the cv2.getGaborKernel() function to generate a Gabor filter kernel.

[0060] Apply the generated Gabor filter kernel to the image and perform a convolution operation. This can be achieved using the cv2.filter2D() function. Each Gabor filter kernel will highlight texture features of a specific direction and frequency in the image.

[0061] In one embodiment, a gray level co-occurrence matrix (GLCM) may also be used to extract texture features. GLCM generates a matrix by calculating the relationship between pixel values ​​in an image and pixel values ​​in its neighborhood, thereby extracting statistics reflecting texture features, such as contrast, correlation, energy, and entropy. For example, for a display image, GLCM can extract the grayscale relationship between pixels to reflect the texture features of the display.

[0062] In one embodiment, the results of multiple Gabor filters may be fused. A maximum value operation or other fusion methods may be used to merge texture features of different directions and frequencies into one feature map.

[0063] In one embodiment, the edge map extracted by the Canny operator and the texture feature map extracted by the Gabor filter can be fused. A weighted fusion method can be used to superimpose the two feature maps to generate an edge feature map containing rich edge and texture information.

[0064] In this way, the edge features of visible light image data can be effectively extracted and a high-quality edge feature map can be generated, which can provide important feature information for subsequent defect identification in applications such as display defect detection.

[0065] S2032. Use the temperature field second-order derivative calculation method to extract the thermal gradient distribution characteristics of the preprocessed infrared thermal imaging data and generate a thermal gradient distribution map.

[0066] In one embodiment, in order to improve the accuracy of the thermal gradient distribution map, the following generation method can be used: The second-order temperature derivative of each pixel in the preprocessed infrared thermal imaging data is calculated to generate a thermal gradient matrix. The tiny holes in the gradient map are filled using morphological closing operations. The significant thermal anomaly areas are extracted through double threshold segmentation to generate a thermal gradient distribution map.

[0067] For example, the preprocessed infrared thermal imaging data is analyzed by the second-order derivative calculation method of the temperature field to generate a thermal gradient matrix. This method can effectively detect small changes and abnormal areas in the temperature field. For example, when detecting the thermal distribution of electronic equipment, the second-order derivative calculation method can capture the abnormal temperature gradient caused by local short circuit or overheating, so as to better locate the defect.

[0068] Then, to remove noise and small temperature fluctuations, morphological closing operations can be used to fill holes, and opening operations can be used to remove small noise areas. By setting two thresholds, the pixels in the thermal gradient distribution map are divided into three regions: low gradient area, medium gradient area, and high gradient area. A thermal gradient distribution map can be generated by calculating the second-order derivative of infrared thermal imaging data. The distribution map presents the severity of temperature changes in a visual way, where high gradient areas correspond to locations where temperature changes are significant. In thermal imaging analysis of electronic equipment, the thermal gradient distribution map can clearly show the overheating areas of chips or other components, providing an important basis for fault diagnosis and repair.

[0069] Applying the second-order derivative calculation method of the temperature field to the preprocessed infrared thermal imaging data can generate an accurate thermal gradient distribution map. Compared with traditional methods, this method can more sensitively detect small anomalies in temperature changes, and by using the second-order derivative, the temperature gradient contrast in the image can be enhanced, making the defective area more prominent. This has significant advantages in display inspection. For example, when detecting overheating of display thin film transistors (TFTs), the thermal gradient distribution map can accurately show the shape and range of the overheating area, thereby improving inspection efficiency.

[0070] S204, performing feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, wherein the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit.

[0071] The fusion feature matrix is ​​a matrix composed of multiple fusion features, and the fusion features represent the correlation between the edge features and the thermal distribution features of each pixel unit or point in the display.

[0072] The present invention can significantly improve the accuracy and robustness of display defect detection through the effective combination of spatial alignment and cross-modal attention fusion network. This method can not only detect surface defects such as scratches and stains based on visible light images, but also capture potential defects based on thermal distribution, such as temperature anomalies of pixel units, thereby achieving comprehensive detection of display defects.

[0073] In one embodiment, in order to obtain an accurate fusion feature matrix, the edge feature map and the thermal gradient distribution map may be spatially aligned; the aligned edge feature map and the thermal gradient distribution map are subjected to feature fusion processing based on a cross-modal attention fusion network to obtain the fusion feature matrix. For example, feature maps of different modalities are spatially aligned so that they correspond to each other at the pixel level. For example, the cv2.warpAffine() function in the OpenCV library is used to perform affine transformation to achieve spatial docking.

[0074] In one embodiment, feature points, such as corner points, edge points, etc., can also be extracted from the edge feature map and the thermal gradient distribution map, respectively. Harris corner detection algorithm or Canny edge detection algorithm can be used to extract feature points. Then, a feature point matching algorithm, such as Brute-Force Matcher or FLANN (Fast Library for Approximate Nearest Neighbors) matcher, is used to find the corresponding feature point pairs in the two images. Next, based on the matched feature point pairs, a homography matrix estimation algorithm, such as RANSAC (Random Sample Consensus) algorithm, is used to calculate the homography matrix between the two images. Finally, the thermal gradient distribution map is geometrically transformed using the homography matrix to align it with the edge feature map in spatial position.

[0075] The present invention can use a cross-modal attention fusion network to achieve feature fusion, and generate an attention weight map by calculating the correlation between edge features and thermal gradient features at different positions and channels, thereby forming a fused feature matrix. For example, for a certain pixel position, if both the edge feature and the thermal gradient feature show that there is an abnormality at that position, then the attention mechanism will assign higher weights to these two features, so that the fused features pay more attention to that area. The attention weights can be normalized by a softmax function to ensure that their sum is 1. Finally, the fused feature matrix will comprehensively consider the edge information and thermal distribution information, and can more accurately reflect the abnormal conditions of the display pixel units.

[0076] The present invention can significantly improve the accuracy and robustness of display defect detection through the effective combination of spatial alignment and cross-modal attention fusion network. This method can not only detect surface defects such as scratches and stains based on visible light images, but also capture potential defects based on thermal distribution, such as temperature anomalies of pixel units, thereby achieving comprehensive detection of display defects.

[0077] In one embodiment, reference Figure 4,The specific methods of realizing feature fusion based on the cross-modal attention fusion network include: S2041. Perform weighted processing on the aligned edge feature map and the thermal gradient distribution map based on a cross-modal attention fusion network.

[0078] The cross-modal attention fusion network automatically learns the association between different modal features through the attention mechanism and assigns them different weights. The attention mechanism can generate an attention weight map by calculating the similarity or correlation between different modal features. For example, using the Scaled Dot-Product Attention method, the dot product of the query and the key is first calculated, then scaled, and finally normalized by the softmax function to obtain the attention weight.

[0079] In one embodiment, the cross-modal attention fusion network can adopt an encoder-decoder architecture. The encoder part extracts features from the edge feature map and the thermal gradient distribution map respectively to obtain their respective feature representations at different scales. For example, a convolutional neural network (CNN) such as ResNet, VGG, etc. is used to extract features. The decoder part gradually fuses these features and dynamically adjusts the contribution of each modality feature through a cross-modal attention mechanism.

[0080] In one embodiment, under the action of the attention mechanism, a weight is assigned to the feature map of each modality. This weight can be calculated by a fully connected layer or a convolutional layer. Specifically, the aligned edge feature map and thermal gradient distribution map are respectively input into the attention network, and after a series of convolution and pooling operations, an attention weight map is generated. Then, the attention weight map is element-wise multiplied with the original feature map to obtain a weighted edge feature map and thermal gradient distribution map.

[0081] S2042, adding the weighted edge feature map and the thermal gradient distribution map pixel by pixel to obtain a fused feature map.

[0082] Pixel-by-pixel addition refers to a simple addition operation of two feature maps at the pixel level. Assuming that the weighted edge feature map and the thermal gradient distribution map have the same size and number of channels, the pixel values ​​at their corresponding positions can be directly added to obtain a fused feature map. The mathematical formula can be expressed as: F fusion (i,j,k)=w e* F e (i,j,k)+w t* F t (i, j, k); Among them, F fusion represents the fusion feature map, F e and Ft Respectively represent the weighted edge feature map and thermal gradient distribution map, w e and w t is the corresponding weight, (i,j,k) represents the position and channel of the pixel.

[0083] The fused feature map contains pixel-level thermal-optical correlation features, which can simultaneously reflect the correlation information of edge features and thermal distribution features. For example, if the edge feature map shows edge changes at a certain pixel location, and the thermal gradient distribution map shows temperature anomalies at the same location, the fused feature map will have a larger value at this location, thus highlighting possible defects.

[0084] The fusion feature map includes pixel-level thermal-optical correlation features, which simultaneously reflect the correlation information of edge features and thermal distribution features; the thermal-optical correlation features refer to the features that are generated by combining edge features in visible light images with thermal distribution features in infrared thermal imaging data to characterize the correlation relationship between edge features and thermal distribution features of pixel units. This feature can simultaneously reflect the structural information and thermal information of the display pixel unit, providing more comprehensive feature information for defect identification.

[0085] S2043. Perform feature extraction and encoding on the fused feature map to generate a fused feature matrix.

[0086] The present invention performs further feature extraction operations on the fused feature map to capture higher-level and more abstract features. In one embodiment, operations such as convolutional layers and pooling layers can be used to extract features. For example, multiple convolutional layers are used to extract features of different scales and directions, and pooling layers are used to reduce the resolution of the feature map and extract key feature information. Then, the extracted features are encoded to generate a fused feature matrix. The encoding method includes a fully connected layer, attention pooling, etc. The fully connected layer can flatten the feature map and input it into the neural network to learn the complex relationship between the features. Attention pooling highlights important features and suppresses unimportant features by assigning different weights to different features.

[0087] The final generated fusion feature matrix will contain the fused feature information, which can better characterize the abnormal conditions of the display pixel units. This feature matrix can be used as the input of the subsequent defect recognition model for tasks such as defect classification and positioning.

[0088] In one embodiment, in order to improve the accuracy of features and thus improve the accuracy and stability of defect detection, feature extraction and encoding are performed on the fused feature map to generate a fused feature matrix, including: Step 1: Decompose the fused feature map into multiple channels, each channel corresponds to a specific thermal-optical correlation feature; The fused feature map is decomposed into multiple channels, each channel corresponds to a specific thermal-optical correlation feature, such as the uniformity of thermal distribution, the sharpness of edges, etc., so that different features can be processed independently later. Assume that the dimension of the fused feature map is H×W×C, where H is the height, W is the width, and C is the number of channels. Each channel c (c=1,2,...,C) represents a specific thermal-optical correlation feature, for example, channel 1 may correspond to high-frequency changes in thermal distribution, channel 2 may correspond to edge intensity, etc.

[0089] Step 2: Apply the channel attention mechanism to each channel, weight the channels according to their importance, and generate a channel weighted feature map.

[0090] The present invention weights the channels according to their importance, so that the network pays more attention to important feature channels and suppresses unimportant feature channels. Specifically, a channel attention mechanism can be used, for example, a Squeeze-and-Excitation (SE) module. First, the feature map of each channel is globally averaged pooled to obtain C feature vectors, each of which has a dimension of 1×1×C. Then, channel weights are generated through two fully connected layers and an activation function (such as ReLU and Sigmoid). Finally, the channel weights are element-wise multiplied with the original feature map to obtain a channel-weighted feature map.

[0091] Step 3: Apply the spatial attention mechanism to the channel weighted feature map, weight the features of each pixel unit in the channel weighted feature map according to the importance of the spatial position, and generate a spatial weighted feature map.

[0092] The present invention weights the features of each pixel unit in the channel weighted feature map according to the importance of the spatial position, highlighting the important spatial area. Specifically, a spatial attention mechanism is used, such as the spatial attention module in the convolutional neural network (CNN). First, the channel weighted feature map is subjected to feature extraction through a convolutional layer to generate a spatial attention map. The dimension of the attention map is the same as that of the channel weighted feature map, and each pixel value represents the importance of the corresponding position. Then, the spatial attention map is element-by-element multiplied with the channel weighted feature map to obtain a spatial weighted feature map.

[0093] Step 4: Perform feature encoding on the spatial weighted feature map to generate a fusion feature matrix.

[0094] In one embodiment, feature extraction and encoding techniques in deep learning models may be used, such as fully connected layers in convolutional neural networks (CNNs), long short-term memory networks (LSTMs) in recurrent neural networks (RNNs), etc. The spatially weighted feature map is flattened into a one-dimensional vector, and then feature encoding is performed through multiple fully connected layers to generate a fused feature matrix.

[0095] Through the above steps, the fused feature map is feature extracted and encoded, and the generated fused feature matrix can better reflect the thermal-optical correlation characteristics of the display pixel units, thereby improving the accuracy and stability of defect detection.

[0096] S205 , using a pre-trained defect recognition model to perform defect recognition on the display to be detected based on the fused feature matrix, to obtain a defect recognition result of the display to be detected.

[0097] Among them, the defect recognition model refers to a model used to identify display defects, which is usually based on a deep learning algorithm. For example, a dual-branch deep network and a multi-task defect recognition model are used. Among them, the defect recognition results may include the defect type, defect location information, defect severity information, etc. of the display.

[0098] For example, a multi-task defect recognition model is used to generate a thermal-optical joint localization map of the defect location, and a pixel-level segmentation mask is generated through the U-Net architecture to obtain the defect location information.

[0099] For example, a multi-task defect recognition model is used to classify defect types and distinguish between four categories: bad pixels, light leakage, short circuits, and overheating.

[0100] In one embodiment, the defect recognition result may also provide defect severity assessment information. For example, a multi-task defect recognition model may be used to calculate the defect severity score, and a regression model may be used to calculate a weighted comprehensive index of temperature deviation, bad pixel area ratio, and texture distortion.

[0101] In one embodiment, global and local features are extracted from the fused feature matrix to obtain global features and local features, wherein the global features are used to characterize the macro structure of the display to be inspected, and the local features characterize the abnormality of the local pixel unit; a pre-trained defect recognition model is used to perform defect recognition on the display to be inspected based on the global features and the local features.

[0102] Among them, global features refer to the features extracted from the fused feature matrix that can characterize the macroscopic structure of the display to be detected. These features usually reflect the overall characteristics of the display, such as overall brightness, color distribution, texture pattern, etc.

[0103] Local features refer to features extracted from the fused feature matrix that can characterize the abnormality of local pixel units. These features usually reflect the characteristics of local areas of the display, such as pixel abnormalities, bright spots, dark spots, spots, stripes, etc.

[0104] The present invention can effectively utilize the pre-trained defect recognition model to perform defect recognition on the fused feature matrix, and obtain accurate defect recognition results, thereby improving the accuracy and stability of display defect detection.

[0105] In practical applications, the defect recognition model can be trained and applied in the following ways: First, a large amount of image data of displays is collected, including normal displays and displays with defects, and these image data are annotated. The annotated information includes the type, location, size, etc. of the defects. Then, the image data is preprocessed, such as image enhancement, normalization, cropping, etc., to improve the robustness and generalization ability of the model. Then, a fused feature matrix is ​​generated by feature extraction methods, which contains rich defect information. Further, global features and local features are extracted from the fused feature matrix. Global features can be extracted by methods such as global average pooling or global maximum pooling to characterize the overall characteristics of the display, such as overall brightness, color distribution, texture pattern, etc.; local features can be extracted by methods such as convolutional layers and pooling layers to characterize the abnormal conditions of local pixel units, such as pixel anomalies, bright spots, dark spots, spots, stripes, etc.

[0106] Select a suitable defect recognition model, such as convolutional neural network (CNN), recurrent neural network (RNN) or Transformer, initialize the model, and set the model's hyperparameters, such as learning rate, batch size, number of iterations, etc. Fuse the extracted global features and local features to generate the final feature representation. The feature fusion method can be a simple splicing or a more complex weighted summation, attention mechanism, etc. During the model training process, the fused feature representation is input into the model, and forward propagation is performed to obtain the model output. Calculate the loss between the model output and the actual annotation information. Commonly used loss functions include cross entropy loss, mean square error loss, etc. Perform back propagation based on the loss calculation results to update the model parameters. Use optimizers (such as Adam, SGD, etc.) to optimize the model parameters to accelerate the convergence of the model.

[0107] Load the pre-trained defect recognition model and input the fusion feature matrix of the display to be tested into the model. The model makes predictions based on the input fusion feature matrix and outputs defect recognition results, which include information such as the type, location, and size of the defect. Parse the defect recognition results output by the model, extract information such as the type, location, and size of the defect, and display them visually, such as marking the location and type of the defect on the display image. Finally, save the defect recognition results to a file for subsequent analysis and processing. By comparing the model prediction results with the actual annotation information, performance indicators such as the model's accuracy, recall, and F1 score can be calculated to verify the model's defect recognition ability.

[0108] As can be seen from the above, the display defect detection method provided by the embodiment of the present invention obtains visible light image data and infrared thermal imaging data of the display to be detected, performs multimodal feature extraction and fusion, generates a fusion feature matrix, and performs defect identification based on the fusion feature matrix; the scheme obtains visible light image data and infrared thermal imaging data, combines the data of the two modes for defect detection, and can more comprehensively capture the defect characteristics of the display. Visible light image data can detect surface defects, such as bad pixels, bright spots, dark spots, etc., while infrared thermal imaging data can detect thermal distribution anomalies, such as short circuits, overheating, etc.; the combination of the two types of data can effectively improve the accuracy of defect detection; in addition, through multimodal feature extraction and fusion, a fusion feature matrix containing rich information is generated, which can more accurately characterize the correlation between the edge features of pixel units and the thermal distribution features. Defect detection based on the fusion feature matrix can reduce false positives and false negatives in display defect detection, improve the reliability of defect detection, and effectively detect various types of defects, which is suitable for rapid detection needs in large-scale production environments.

[0109] Accordingly, in order to better implement the above method, the embodiment of the present application also provides a display defect detection device. Figure 5 As shown, the display defect detection device includes an acquisition module 301, a data preprocessing module 302, a feature extraction module 303, a feature fusion module 304 and a defect recognition module 305, which are specifically as follows: An acquisition module 301 is used to acquire visible light image data and corresponding infrared thermal imaging data of a display to be detected, wherein the visible light image data includes image information of a surface of the display to be detected; A data preprocessing module 302 is used to perform cross-modal preprocessing on the visible light image data and the infrared thermal imaging data to obtain preprocessed visible light image data and infrared thermal imaging data; A feature extraction module 303 is used to perform multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, wherein the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit; A feature fusion module 304 is used to perform feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, wherein the fusion features in the fusion feature matrix represent the correlation between the edge feature and the thermal distribution feature of the pixel unit; The defect recognition module 305 is used to use a pre-trained defect recognition model to perform defect recognition on the display to be detected based on the fusion feature matrix to obtain a defect recognition result of the display to be detected.

[0110] In one embodiment, the feature fusion module 304 is used to: spatially aligning the edge feature map and the thermal gradient distribution map; Based on the cross-modal attention fusion network, feature fusion processing is performed on the aligned edge feature map and the thermal gradient distribution map to obtain the fused feature matrix.

[0111] In one embodiment, the feature fusion module 304 is used to: Performing weighted processing on the aligned edge feature map and the thermal gradient distribution map based on a cross-modal attention fusion network to obtain a weighted edge feature map and a thermal gradient distribution map; The weighted edge feature map and the thermal gradient distribution map are added pixel by pixel to obtain a fused feature map, wherein the fused feature map contains pixel-level thermal-optical correlation features and reflects correlation information between edge features and thermal distribution features; Perform feature extraction and encoding on the fused feature map to generate a fused feature matrix.

[0112] In one embodiment, the feature fusion module 304 is used to: Decompose the fused feature map into multiple channels, each channel corresponds to a specific thermal-optical correlation feature; Apply a channel attention mechanism to each channel, weight the channel according to the importance of the channel, and generate a channel weighted feature map; Applying a spatial attention mechanism to the channel weighted feature map, weighting the features of each pixel unit in the channel weighted feature map according to the importance of the spatial position, and generating a spatial weighted feature map; The spatial weighted feature map is feature encoded to generate a fusion feature matrix.

[0113] In one embodiment, the data preprocessing module 302 is used to: Performing image preprocessing on the visible light image data, wherein the image preprocessing at least includes geometric distortion correction and brightness standardization; Performing thermal map preprocessing on the infrared thermal imaging data to obtain enhanced infrared thermal imaging data, wherein the thermal map preprocessing at least includes temperature calibration, spatial registration and pseudo color mapping; The visible light image data after image preprocessing and the infrared thermal imaging data after thermal image preprocessing are respectively subjected to image smoothing processing to remove random noise.

[0114] In one embodiment, the feature extraction module 303 is used to: Extract edge features from the preprocessed visible light image data to obtain an edge feature map; The second-order derivative calculation method of the temperature field is used to extract the thermal gradient distribution characteristics of the preprocessed infrared thermal imaging data and generate a thermal gradient distribution map.

[0115] In one embodiment, the feature extraction module 303 is used to: Extract edge features from the preprocessed visible light image data to obtain edge features; Extracting texture features from the preprocessed visible light image data to obtain texture features; The edge feature is fused with the texture feature to generate an edge feature map representing edge and texture information.

[0116] In one embodiment, the feature extraction module 303 is used to: The second-order temperature derivative of each pixel in the preprocessed infrared thermal imaging data is calculated to generate a thermal gradient matrix. The tiny holes in the gradient map are filled using morphological closing operations. The significant thermal anomaly areas are extracted through double threshold segmentation to generate a thermal gradient distribution map.

[0117] In one embodiment, the defect identification module 305 is used to: Performing global and local feature extraction on the fused feature matrix to obtain global features and local features, wherein the global features are used to characterize the macro structure of the display to be detected, and the local features characterize the abnormality of the local pixel unit; A pre-trained defect recognition model is used to perform defect recognition on the display to be inspected based on the global features and the local features.

[0118] For the implementation of the above modules, please refer to the previous method embodiments for details, which will not be described in detail here.

[0119] It should be noted that, in specific implementation, the above modules can be arbitrarily combined, integrated into one or more modules, or implemented as independent entities. In addition, the above modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0120] As can be seen from the above, the display detection device provided in the embodiment of the present application can obtain visible light image data and infrared thermal imaging data, and combine the data of the two modes to perform defect detection, so as to capture the defect characteristics of the display more comprehensively. Visible light image data can detect surface defects, such as bad pixels, bright spots, dark spots, etc., while infrared thermal imaging data can detect thermal distribution anomalies, such as short circuits, overheating, etc.; the combination of the two types of data can effectively improve the accuracy of defect detection; in addition, through multi-modal feature extraction and fusion, a fusion feature matrix containing rich information is generated, which can more accurately characterize the correlation between the edge features of the pixel unit and the thermal distribution features. Defect detection based on the fusion feature matrix can reduce false positives and false negatives in display defect detection, improve the reliability of defect detection, and effectively detect various types of defects, which is suitable for rapid detection needs in large-scale production environments.

[0121] like Figure 6 As shown, an embodiment of the present application further provides a computer device 40, characterized in that it includes a processor 401 and a memory 402, wherein the memory 402 stores a computer program, and when the computer program is executed by the processor 401, the processor 401 executes the steps of any of the methods described above.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0125] On one hand, an embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in one aspect of the embodiment of the present application.

[0126] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0127] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0128] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0129] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A display defect detection method, characterized in that: The method comprises: Acquire visible light image data and corresponding infrared thermal imaging data of the display to be detected, wherein the visible light image data includes image information of the surface of the display to be detected; Performing cross-modal preprocessing on the visible light image data and the infrared thermal imaging data respectively to obtain preprocessed visible light image data and infrared thermal imaging data; Performing multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, wherein the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit; Performing feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, wherein the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit; A pre-trained defect recognition model is used to perform defect recognition on the display to be detected based on the fused feature matrix to obtain a defect recognition result of the display to be detected.

2. The method according to claim 1, characterized in that Performing feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, including: spatially aligning the edge feature map and the thermal gradient distribution map; Based on the cross-modal attention fusion network, feature fusion processing is performed on the aligned edge feature map and the thermal gradient distribution map to obtain the fused feature matrix.

3. The method according to claim 2, characterized in that Performing feature fusion processing on the aligned edge feature map and the thermal gradient distribution map based on a cross-modal attention fusion network, including: Performing weighted processing on the aligned edge feature map and the thermal gradient distribution map based on a cross-modal attention fusion network to obtain a weighted edge feature map and a thermal gradient distribution map; The weighted edge feature map and the thermal gradient distribution map are added pixel by pixel to obtain a fused feature map, wherein the fused feature map contains pixel-level thermal-optical correlation features and reflects correlation information between edge features and thermal distribution features; Perform feature extraction and encoding on the fused feature map to generate a fused feature matrix.

4. The method according to claim 3, characterized in that Perform feature extraction and encoding on the fused feature map to generate a fused feature matrix, including: Decompose the fused feature map into multiple channels, each channel corresponds to a specific thermal-optical correlation feature; Apply a channel attention mechanism to each channel, weight the channel according to the importance of the channel, and generate a channel weighted feature map; Applying a spatial attention mechanism to the channel weighted feature map, weighting the features of each pixel unit in the channel weighted feature map according to the importance of the spatial position, and generating a spatial weighted feature map; The spatial weighted feature map is feature encoded to generate a fusion feature matrix.

5. The method according to any one of claims 1 to 4, characterized in that: The visible light image data and the infrared thermal imaging data are respectively subjected to cross-modal preprocessing, including: Performing image preprocessing on the visible light image data, wherein the image preprocessing at least includes geometric distortion correction and brightness standardization; Performing thermal map preprocessing on the infrared thermal imaging data to obtain enhanced infrared thermal imaging data, wherein the thermal map preprocessing at least includes temperature calibration, spatial registration and pseudo color mapping; The visible light image data after image preprocessing and the infrared thermal imaging data after thermal image preprocessing are respectively subjected to image smoothing processing to remove random noise.

6. The method according to claim 5, characterized in that Perform multimodal feature extraction on preprocessed visible light image data and infrared thermal imaging data, including: Using a preset edge detection algorithm to extract edge features from the preprocessed visible light image data to obtain an edge feature map; The second-order derivative calculation method of the temperature field is used to extract the thermal gradient distribution characteristics of the preprocessed infrared thermal imaging data and generate a thermal gradient distribution map.

7. The method according to claim 6, characterized in that The edge feature extraction of the preprocessed visible light image data is performed using a preset edge detection algorithm to obtain an edge feature map, including: Extract edge features from the preprocessed visible light image data to obtain edge features; Extracting texture features from the preprocessed visible light image data to obtain texture features; The edge feature is fused with the texture feature to generate an edge feature map representing edge and texture information.

8. The method according to claim 6, characterized in that The second-order derivative calculation method of the temperature field is used to extract the thermal gradient distribution characteristics of the preprocessed infrared thermal imaging data and generate a thermal gradient distribution map, including: The second-order temperature derivative of each pixel in the preprocessed infrared thermal imaging data is calculated to generate a thermal gradient matrix. The tiny holes in the gradient map are filled using morphological closing operations. The significant thermal anomaly areas are extracted through double threshold segmentation to generate a thermal gradient distribution map.

9. The method according to claim 7, characterized in that Using a pre-trained defect recognition model to perform defect recognition on the display to be inspected based on the fusion feature matrix includes: Performing global and local feature extraction on the fused feature matrix to obtain global features and local features, wherein the global features are used to characterize the macro structure of the display to be detected, and the local features characterize the abnormality of the local pixel unit; A pre-trained defect recognition model is used to perform defect recognition on the display to be inspected based on the global features and the local features.

10. A display defect detection device, characterized in that: include: An acquisition module, used to acquire visible light image data and corresponding infrared thermal imaging data of the display to be detected, wherein the visible light image data includes image information of the surface of the display to be detected; A data preprocessing module, used to perform cross-modal preprocessing on the visible light image data and the infrared thermal imaging data, respectively, to obtain preprocessed visible light image data and infrared thermal imaging data; a feature extraction module, configured to perform multimodal feature extraction on the preprocessed visible light image data and infrared thermal imaging data, to obtain an edge feature map of the visible light image data and a thermal gradient distribution map of the infrared thermal imaging data, wherein the edge feature map includes edge features of each pixel unit of the display to be detected, and the thermal gradient distribution map includes thermal distribution features of each pixel unit; A feature fusion module, used for performing feature fusion processing on the edge feature map and the thermal gradient distribution map to obtain a fusion feature matrix, wherein the fusion features in the fusion feature matrix represent the correlation between the edge features and the thermal distribution features of the pixel unit; The defect recognition module is used to use a pre-trained defect recognition model to perform defect recognition on the display to be detected based on the fusion feature matrix to obtain a defect recognition result of the display to be detected.

Images