Method and system for improving Mura defect classification precision, electronic equipment and storage medium

Abstract

The invention discloses a method and a system for improving Mura defect classification precision, electronic equipment and a storage medium, which are used for realizing accurate classification of Mura defects of low-gray-scale images. The method for improving the Mura defect classification precision comprises the steps of obtaining an image on a display screen to be detected; performing texture filtering processing on the image to obtain a filtered image; performing graying processing and defect segmentation processing on the filtered image to obtain a to-be-detected image; inputting the to-be-detected image into the Mura classification model to obtain a classification result; wherein the Mura classification model is obtained through the following training method: obtaining first data collected by a plurality of sensors; preprocessing the first data to obtain second data; extracting feature data in the second data; fusing the feature data according to an orthogonal sequence fusion algorithm to obtain a fusion result; and training according to a fusion result to obtain a Mura classification model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of screen defect detection technology, and in particular, to a method, system, electronic device, and storage medium for improving the accuracy of Mura defect classification. Background Art

[0002] With the development and progress of science and technology, display screens have been widely used in people's lives. However, due to process problems during industrial production, there are various defects in the manufacturing process of display screens, such as point defects, line defects, Mura defects, etc. (Mura means spots, and now specifically refers to defects on the display screen, which is uneven in the display area when the display screen is displayed at a constant brightness). Among them, Mura defect is a kind of bad defect that seriously affects the picture quality of the display screen, which is mainly manifested as uneven brightness display in the screen area. According to actual production needs, it is necessary to perform Mura defect detection on the display screen, which includes the steps of Mura defect classification.

[0003] In the mura defect detection and classification system, low grayscale images are easily restricted by a variety of factors due to their low brightness, including insufficient number of photons, low signal-to-noise ratio, limited dynamic range, sensor performance limitations, and imperfect optical systems. These factors work together to cause low grayscale images to be prone to noise pollution, loss of detail information, insufficient contrast, and color distortion, which seriously reduces the imaging quality of low grayscale images, further affects the classification model's accurate identification of defects, and thus reduces the overall classification accuracy of the system. Summary of the invention

[0004] The present application discloses a method, system, electronic device and storage medium for improving the accuracy of Mura defect classification, which are used to achieve accurate classification of Mura defects in low grayscale images.

[0005] In a first aspect, the present application discloses a method for classifying mura defects, comprising: Acquire an image of the display screen to be inspected, where the image is a Mura defect image; Performing texture filtering on the image to obtain a filtered image; Performing grayscale processing and defect segmentation processing on the filtered image to obtain an image to be detected; Inputting the image to be detected into a mura classification model to obtain a classification result, wherein the classification result includes the type, location, and severity of the mura in the image to be detected; The mura classification model is obtained by the following training method: Acquire first data collected by a plurality of sensors, wherein the plurality of sensors include at least one of the following: a visual camera, an infrared camera, and a synchronization controller; Preprocessing the first data to obtain second data; extracting feature data from the second data; The feature data are fused according to an orthogonal sequence fusion algorithm to obtain a fusion result; The Mura classification model is obtained by training according to the fusion result.

[0006] Optionally, the first data collected by multiple sensors is obtained, and the multiple sensors include at least one of the following: a visual camera, an infrared camera, and a synchronous controller including: Acquire a product surface image captured by the visual camera, wherein the product surface image includes color information and texture information of the product; Acquire an infrared image of a product captured by the infrared camera, wherein the infrared image of the product includes temperature distribution information of the product; Acquire the time information and spatial position information of the visual camera and the infrared camera respectively collected by the synchronization controller; The product surface image, the product infrared image, the time information and the spatial position information are organized into the first data.

[0007] Optionally, preprocessing the first data to obtain second data includes: Removing noise from the product surface image and the product infrared image according to a Gaussian filtering algorithm and a median filtering algorithm to obtain a de-noised image; Performing contrast enhancement and histogram equalization processing on the denoised image to obtain a processed image; The processed image, the time information and the spatial position information are normalized to obtain the second data.

[0008] Optionally, the processed image includes a surface processed image and an infrared processed image, and the extracting feature data from the second data includes: Directly extracting texture features of the local binary pattern and gray-level co-occurrence matrix of the surface-processed image, as well as color features; Inputting the surface processed image into a first feature extraction layer of a convolutional neural network and outputting a depth feature; The infrared processed image is input into the second feature extraction layer of the convolutional neural network, and the temperature distribution feature is output.

[0009] Optionally, fusing the feature data according to an orthogonal sequence fusion algorithm to obtain a fusion result includes: Merge the texture feature, the color feature, the depth feature and the temperature distribution feature according to a preset order to obtain a merged result; assigning weights to the merging result, the time information, and the spatial position information respectively according to a preset weighting algorithm; The weighted merging result is orthogonally fused with the time information and the spatial position information to obtain a fusion result.

[0010] Optionally, the training according to the fusion result to obtain the Mura classification model includes: inputting the fusion result into a classification layer of a convolutional neural network for model training to obtain the Mura classification model.

[0011] Optionally, after obtaining the classification result, the method further includes: visualizing the classification result and displaying it on a user interface.

[0012] The second aspect of the present application discloses a system for improving the accuracy of Mura defect classification, including: An imaging unit, used to obtain an image on the display screen to be inspected, wherein the image is a Mura defect image; A filtering unit, used for performing texture filtering on the image to obtain a filtered image; A processing unit, used for performing grayscale processing and defect segmentation processing on the filtered image to obtain an image to be detected; An input unit, used to input the image to be detected into a mura classification model to obtain a classification result, wherein the classification result includes the type, location and severity of mura in the image to be detected; The mura classification model is obtained by the following training method: Acquire first data collected by a plurality of sensors, wherein the plurality of sensors include at least one of the following: a visual camera, an infrared camera, and a synchronization controller; Preprocessing the first data to obtain second data; extracting feature data from the second data; The feature data are fused according to an orthogonal sequence fusion algorithm to obtain a fusion result; A mura classification model is obtained by training according to the fusion result.

[0013] A third aspect of the present application provides an electronic device, including: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method for improving the accuracy of Mura defect classification of the first aspect.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the method for improving the accuracy of Mura defect classification in the first aspect and any optional method in the first aspect is performed.

[0015] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: The present application provides a method for improving the accuracy of Mura defect classification, which mainly includes obtaining an image on a display screen to be detected, the image is a Mura defect image; performing texture filtering on the image to obtain a filtered image; performing grayscale processing and defect segmentation processing on the filtered image to obtain an image to be detected; inputting the image to be detected into a Mura classification model to obtain a classification result, and the classification result includes the type, location and severity of Mura in the image to be detected; wherein the Mura classification model is obtained by the following training method: firstly, obtaining the first data collected by multiple sensors, the multiple sensors include at least one of the following: a visual camera, an infrared camera and a synchronous controller; secondly, preprocessing the first data to obtain the second data; then extracting the feature data in the second data; then fusing the feature data according to an orthogonal sequence fusion algorithm to obtain a fusion result; finally, training the Mura classification model according to the fusion result. On the one hand, before performing image Mura defect classification, the method performs texture filtering, grayscale processing and defect segmentation processing on the image. Through the above three steps of preprocessing, the negative impact of classifying Mura defects can be reduced and the accuracy of Mura defect classification can be improved. On the other hand, first data is acquired through multiple sensors, where the first data represents multiple factors affecting the imaging of low grayscale images, and the first data is processed and fused, thereby acquiring the logical relationship of the imaging of low grayscale images; and a Mura classification model is further trained using the fused data, where the Mura classification model can assist the system to accurately and quickly identify Mura defects appearing in the image to be detected according to the logic of the imaging of low grayscale images, and correctly classify multiple Mura defects, thereby improving the working efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 A schematic diagram of an embodiment of a method for improving the accuracy of Mura defect classification in this application; Figure 2A schematic diagram of an embodiment of a method for training a mura classification model for the present application; Figure 3 A schematic diagram of an embodiment of a system for improving the accuracy of Mura defect classification in this application; Figure 4 This is a schematic diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0019] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0020] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0021] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0022] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0024] The Mura defect detection and classification system usually includes high grayscale images and low grayscale images. High grayscale images such as L255, L128, L64, etc. have high image quality and less noise, and the classification model can usually achieve good classification accuracy; low grayscale images such as L32, L16, L8, etc., due to the low brightness of low grayscale images, are affected by factors such as insufficient number of photons, low signal-to-noise ratio, dynamic range limitation, sensor performance limitations, and optical system limitations, making low brightness images prone to noise, detail loss, insufficient contrast, and color distortion, resulting in a decrease in low grayscale imaging quality, thereby reducing the classification accuracy of the classification model. Most of the existing methods are aimed at low grayscale image denoising, image enhancement, etc. to solve the problem of low classification accuracy, but such methods cannot completely eliminate interference such as noise and image distortion and usually rely on a single data source, with a relatively single feature dimension and poor robustness, resulting in a decrease in the overall classification accuracy of the system.

[0025] Based on this, the present application discloses a method, system, electronic device and storage medium for improving the accuracy of Mura defect classification, which are used to achieve accurate classification of Mura defects in low grayscale images.

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The method of the present application can be applied to a server, a device, a terminal or other devices with logic processing capabilities, and the present application does not limit this. For the convenience of description, the following description is made by taking the execution subject as an example of a terminal.

[0028] See also Figure 1 The present application provides an embodiment of a method for improving the accuracy of Mura defect classification, including: 101, obtaining an image on the display screen to be inspected, where the image is a Mura defect image; In this embodiment, the display screen to be tested is usually an LED display screen or an LCD display panel. The Mura defect of the LED screen is usually related to factors such as uneven brightness distribution of the LED light source, quality differences of LED chips, design problems of the driving circuit, and poor heat dissipation. The Mura defect of the LCD display panel is more related to the characteristics of the arrangement, orientation and fluidity of the liquid crystal molecules. In the display defect detection process, the first step is to obtain the Mura defect image of the display to be detected. The specific acquisition methods may include, but are not limited to, the following: 1. Use a high-resolution camera or video camera to directly shoot the display to be detected, but ensure that the light in the shooting environment is uniform to avoid excessive or weak light causing image quality degradation. When shooting, ensure that the camera or video camera lens is at a certain distance and angle from the display to obtain a complete display image; 2. If the display to be detected is a device that can display images, such as a computer monitor or a mobile phone screen, the image on the display can be obtained through screenshot software or the system's built-in screenshot function. When taking a screenshot, ensure that the image on the display is clear and complete, and is not interfered with by other software; 3. In some high-end detection scenarios, professional image acquisition equipment can be used to obtain images on the display. These image acquisition devices usually have higher resolution, more accurate color reproduction capabilities, and stronger anti-interference capabilities, and can obtain higher quality images.

[0029] 102, performing texture filtering on the image to obtain a filtered image; In this embodiment, the purpose of step 102 is to reduce the texture noise in the image so that the Mura defects on the image are easier to identify. By smoothing the texture in the image, some unnecessary details can be eliminated while retaining important edge information, thereby improving the accuracy and efficiency of defect detection. Specifically, Gaussian filters and mean filters can be used. Gaussian filter is a commonly used linear filter whose weight coefficient is determined by a two-dimensional Gaussian function. Gaussian filter can smooth the image well while retaining the edge information of the image. The parameters of the Gaussian filter include the size and standard deviation of the filter, and the degree of smoothing can be controlled by adjusting these parameters. The mean filter is a simple linear filter that smoothes the image by calculating the average value of the pixels in the neighborhood around each pixel in the image. Although the mean filter can remove noise, it may also cause blurring of the image edge. Therefore, when selecting the mean filter, it is necessary to weigh the balance between smoothing effect and edge retention.

[0030] In addition to Gaussian filters and mean filters, there are other texture filters to choose from, such as median filters, bilateral filters, etc., which can be selected according to specific application scenarios and needs. It should be noted that when selecting the filter size, a trade-off should be made based on the resolution of the display to be inspected and the size of the defect. At the same time, after applying the filter, the filtering results need to be evaluated, for example: by observing whether the texture noise is smoothed in the filtered image while retaining important edge information to judge the quality of the filtering effect. Some quantitative indicators can also be used to evaluate the filtering effect, such as signal-to-noise ratio, peak signal-to-noise ratio, etc., which can objectively reflect the quality changes of the image before and after filtering. 103, graying and defect segmenting the filtered image to obtain an image to be inspected; In this embodiment, grayscale processing and defect segmentation processing can simplify the subsequent processing flow, and effectively separate the defect area from the background, providing a high-quality image to be detected for subsequent detection and classification. Grayscale processing is the process of converting a color image into a grayscale image. A color image usually contains three color channels: red, green, and blue, while a grayscale image has only one brightness channel. Specifically, commonly used grayscale algorithms include weighted average method, maximum method, and minimum method. After the image is grayscaled, each pixel has only one grayscale value, which represents the brightness of the pixel. The grayscale image retains important features in the original image, such as edges, textures, etc., while removing color information, simplifying the subsequent processing flow. Defect segmentation is the process of separating the defective area in an image from the background. The defective area usually has a significant contrast difference with the background. Common segmentation algorithms include threshold segmentation, edge detection, and region growing. Specifically, threshold segmentation sets a threshold based on the grayscale value of the image, and classifies pixels with grayscale values ​​greater than the threshold as foreground (defective area), and pixels with grayscale values ​​less than the threshold as background. The edge detection algorithm can identify edge information in the image and thus segment the defective area. The region growing algorithm starts from the initial seed point and gradually expands the area according to the similarity criterion until the stop condition is met. 104, inputting the image to be detected into a mura classification model to obtain a classification result, where the classification result includes the type, location, and severity of the mura in the image to be detected; In this embodiment, according to the requirements of display screen defect detection, a suitable Mura classification model is selected. The Mura classification model in this embodiment is obtained by the following training method: obtaining first data collected by multiple sensors, the multiple sensors include at least one of the following: a visual camera, an infrared camera and a synchronous controller; preprocessing the first data to obtain second data; extracting feature data in the second data; fusing the feature data according to an orthogonal sequence fusion algorithm to obtain a fusion result; and training to obtain a Mura classification model based on the fusion result. Different types of sensors include visual cameras, infrared cameras and synchronous controllers. The first data includes image data of the product (visual features such as the shape, color, and texture of the product), temperature distribution data, time and location data of the product taken by the visual camera and the infrared camera, etc. The time and location data are used to ensure that the data collection between different sensors is synchronized, and the temporal and spatial consistency of the data can be ensured. Then the first data is preprocessed to obtain the second data. Specifically, the preprocessing includes removing noise, outliers and duplicate data in the data to improve the quality and accuracy of the data; and appropriately transforming the data to make it more suitable for subsequent analysis and processing. For example, the image data can be scaled, rotated or cropped. Next, all data should be converted into a unified format or unit to facilitate subsequent comparison and analysis. It should be noted that feature extraction is the process of extracting key information or attributes from data that are useful for subsequent analysis. Specialized algorithms or deep networks can be used to automatically extract features from signals or images, which can process large amounts of data and extract statistically significant features. Afterwards, the feature data is fused according to the orthogonal sequence fusion algorithm, which is a method for multi-sensor data fusion, that is, using the characteristics of orthogonal sequences to balance the correlation between different sensor data, thereby improving the accuracy and robustness of the fusion results. Finally, the fused feature data is used as a training data set, and a suitable machine learning model (such as support vector machine SVM) or deep learning model (such as convolutional neural network CNN) is selected to build a mura classification model. During the training process, the model learns the characteristics and rules of the data and adjusts its parameters to minimize classification errors.

[0031] After the Mura classification model is established, the image to be tested is input into the Mura classification model, and the Mura classification model classifies the Mura defects according to the data contained in the image to be tested, and outputs the classification results. It should also be noted that after obtaining the classification results, the classification results can also be visualized and displayed on the user interface, for example, the classification results can be converted into charts and legends, so that the user can perform subsequent work according to the classification results displayed on the user interface.

[0032] In this embodiment, the method for improving the accuracy of Mura defect classification is mainly to obtain an image on the display screen to be detected, the image is a Mura defect image; perform texture filtering on the image to obtain a filtered image; perform grayscale processing and defect segmentation processing on the filtered image to obtain an image to be detected; input the image to be detected into the Mura classification model to obtain a classification result, and the classification result includes the type, location and severity of Mura in the image to be detected; wherein the Mura classification model is obtained by the following training method: first, obtain the first data collected by multiple sensors, and the multiple sensors include at least one of the following: a visual camera, an infrared camera and a synchronous controller; secondly, preprocess the first data to obtain the second data; then extract the feature data in the second data; then fuse the feature data according to the orthogonal sequence fusion algorithm to obtain the fusion result; finally, train the Mura classification model according to the fusion result. On the one hand, before performing the image Mura defect classification, the method performs texture filtering, grayscale processing and defect segmentation processing on the image. Through the above three steps of preprocessing, the negative impact when classifying Mura defects can be reduced and the accuracy of Mura defect classification can be improved. On the other hand, first data is acquired through multiple sensors, and the first data represents multiple factors affecting the imaging of low grayscale images. The first data is then processed and fused, thereby acquiring the logical relationship of the imaging of low grayscale images. The fused data is further used to train a Mura classification model, and the Mura classification model can assist the system to accurately and quickly identify Mura defects appearing in the image to be detected according to the logic of low grayscale image imaging, and correctly classify multiple Mura defects, thereby improving the working efficiency of the system.

[0033] See also Figure 2 The Mura classification model in step 104 is obtained through training. The present application provides an embodiment of a training method for obtaining a Mura classification model, including: 201, obtaining a product surface image captured by a visual camera, where the product surface image includes color information and texture information of the product; In this embodiment, in order to capture the appearance characteristics of the product, especially the color and texture information, the surface image of the product can be collected by a visual camera. The color information and texture information in the surface image are crucial for subsequent analysis and identification of defects. A suitable visual camera should also be selected to ensure that its parameters such as resolution, focal length and exposure can meet the detection requirements. The camera should be aimed at the surface of the product, and the focal length and exposure should be adjusted to ensure that the image is clear and the color is accurate. An automated control system can be used to control the position and focal length of the camera to improve efficiency and accuracy.

[0034] 202, obtaining an infrared image of a product captured by an infrared camera, where the infrared image of the product includes temperature distribution information of the product; In this embodiment, the characteristics of the infrared camera are used to capture the temperature distribution of the product to detect potential thermal defects. Select an infrared camera with an appropriate temperature measurement range and resolution, aim the infrared camera at the product, and adjust the temperature measurement range and resolution to ensure that the temperature distribution of the product can be captured. In addition, infrared images can be collected at different time points and conditions to obtain more comprehensive temperature information.

[0035] 203, obtaining time information and spatial position information of the visual camera and the infrared camera respectively collected by the synchronization controller; In this embodiment, in order to ensure that the images captured by the visual camera and the infrared camera are synchronized in time and space so that the images can be accurately aligned and analyzed later, a synchronization controller can be used to record the timestamps of the images captured by the two cameras. At the same time, the spatial position information of the cameras, such as the position and angle of the cameras, is recorded. In addition, high-precision time synchronization equipment and spatial positioning technology can also be used to ensure the accuracy of the data.

[0036] 204, arranging the product surface image, the product infrared image, the time information and the spatial position information into first data; In this embodiment, the product surface image, product infrared image, time information and spatial location information obtained in step 201 to step 203 are integrated together to form a complete data set, which facilitates subsequent analysis and processing. That is, the product surface image, product infrared image, time information and spatial location information are sorted according to a certain format, and a storage method such as a database or a file system can be used to save these data, while ensuring the integrity and consistency of the data, so that the data can be accurately accessed and used later.

[0037] 205, removing noise from the product surface image and the product infrared image according to a Gaussian filtering algorithm and a median filtering algorithm to obtain a de-noised image; In this embodiment, similarly, according to the characteristics and requirements of the image, appropriate filtering parameters and algorithms are selected, and the Gaussian filtering algorithm and the median filtering algorithm can be used to reduce the noise of the image. An automated control system can also be used to detect and adjust the filtering process in real time to ensure the stability of the image quality. The product surface image and infrared image after noise reduction are obtained, and the quality of these images is significantly improved so that features can be extracted more accurately in the future.

[0038] 206, performing contrast enhancement and histogram equalization processing on the noise reduction image to obtain a processed image, where the processed image includes a surface processed image and an infrared processed image; In this embodiment, the image after noise reduction is subjected to contrast enhancement processing to improve the contrast of different regions in the image, and a histogram equalization algorithm is used to further improve the contrast distribution of the image. The image details after contrast enhancement and histogram equalization processing are clearer, providing a better basis for subsequent feature extraction.

[0039] 207, normalize the processed image, time information and spatial position information to obtain second data; In this embodiment, the processed image, time information and spatial position information are normalized, and linear normalization, nonlinear normalization or other normalization methods may be used according to actual conditions, to further ensure that the normalized data is within a suitable range and has a consistent scale. 208, directly extracting texture features of local binary patterns and gray-level co-occurrence matrices of surface-processed images, as well as color features; In this embodiment, algorithms such as local binary patterns and gray-level co-occurrence matrices are used to extract texture features. Color features, such as color histograms and color matrices, are also extracted. An automated control system can also be used to detect and adjust the feature extraction process in real time to ensure the stability and accuracy of the features. Texture features and color features provide important basis for subsequent classification and recognition.

[0040] 209, input the surface processed image into the first feature extraction layer of the convolutional neural network, and output the depth feature; In this embodiment, the surface processed image is input into the first feature extraction layer of the convolutional neural network, and the standard convolution of the 64-channel residual block and the 128-channel residual block in the general convolutional neural network is replaced by a deformable convolution. In the deformable convolution, the offset on the input feature map corresponding to each output position is first learned through an additional 3x3 convolution, and these offsets reflect the most relevant feature area position in the current context; then, the original input feature map is sampled using the predicted offset, that is, the filter is no longer translated at a fixed step size, but jumps to the new sampling point according to the offset for weighted summation to generate the final feature map. Deformable convolution can adapt to the morphological features of the target object, so that the convolution result is more in line with the target shape, and then extracts more complete target features. Finally, in the output layer, a 1x10 feature vector, i.e., a depth feature, is output according to the number of categories to be classified.

[0041] Compared with the hand-designed traditional image features (such as color, texture, grayscale, etc.), deep features are more abstract. Deep features are not calculated directly based on the pixel values ​​or simple statistics of the image, but are obtained by sliding the convolution kernel on the image and performing complex operations. This abstraction enables deep features to capture more subtle and complex feature information in the image. And because the first feature extraction layer structure uses deformable convolution and pooling operations, the extracted deep features are robust to image translation, rotation, scale changes, etc. Even if these changes occur to the target object in the image, the first feature extraction layer can still accurately extract the relevant feature information. Output deep features, which contain richer information and provide a better foundation for subsequent classification and recognition.

[0042] 210, inputting the infrared processed image into a second feature extraction layer of a convolutional neural network, and outputting temperature distribution features; In this embodiment, the infrared processed image is input into the second feature extraction layer of the convolutional neural network. Since the infrared image imaging background is relatively simple and has little noise interference, the second feature extraction layer is designed as a 5-layer convolutional neural network, including three convolutional layers, a maximum pooling layer, and an output layer; a 1x10 feature vector, i.e., the temperature distribution feature, is output.

[0043] 211, merging the texture feature, the color feature, the depth feature, and the temperature distribution feature according to a preset order to obtain a merging result; In this embodiment, the OSF fusion method is used to fuse multiple features. Since the OSF fusion method does not fuse data of all modalities at the same time, but merges them step by step in sequence to obtain a fusion result, the texture features, color features, depth features, and temperature distribution features should be merged in a preset order. The preset order is obtained by first comparing the influence of multiple features on image imaging, and then arranging the texture features, color features, depth features, and temperature distribution features.

[0044] 212, assigning weights to the merging result, the time information, and the spatial position information respectively according to a preset weighting algorithm; In this embodiment, based on the OSF fusion method in step 211, different weights are assigned to each feature according to the correlation of multiple features, which helps to balance the fusion and maximize the extraction of complementary information across features. A preset weighting algorithm is used to assign weights to the merged feature set and the time information and spatial location information. The preset weighting algorithm determines the weights according to the importance, correlation or other factors of the features to ensure that the weights are reasonably and consistently allocated.

[0045] 213, orthogonally fusing the weighted merging result with the time information and the spatial position information to obtain a fusion result; In this embodiment, after the merging in step 211 and the weighting in step 212, the merging result is fused with the time information and the spatial position information using an orthogonal fusion method. Orthogonal fusion can ensure the independence between different features, thereby improving the accuracy of classification and recognition. Among them, the OSF fusion method introduces an orthogonal loss function, which makes the embedding orthogonal at each layer by using L1 and L2 regularization terms. This ensures that the embedding is not only diverse but also complementary. That is, if there are N modal features (each modality contains at least one feature) and N-1 fusion layers Li = [M1,i, M2,i], the loss function of the OSF fusion method can be written as:

[0046] in, represents the feature vector of the first mode at layer i, represents the feature vector of the second mode at layer i, , represents the weight parameter assigned to each modality. The regularization term makes the embeddings of each layer orthogonal in the loss function. The first term forces the absolute value of the dot product of the embeddings to be close to zero, while the second term forces the square of the dot product to be close to zero. Together, these two terms ensure that the embeddings are not only diverse but also complementary.

[0047] 214, the fusion result is input into the classification layer of the convolutional neural network for model training to obtain the mura classification model.

[0048] In this embodiment, the fusion result (the fused feature vector) is input into the classification layer of the convolutional neural network for training, that is, the fused feature vector is input into the softmax classifier. The softmax classifier is a commonly used multi-classification algorithm that can output probability predictions for each category based on the input feature vector. By comparing the probability values ​​of different categories, the classification layer can determine the category to which the input data belongs and output the final classification result. After a large amount of different data is fused, classified and output, a Mura classification model is finally formed.

[0049] The above embodiments illustrate the method for improving the accuracy of Mura defect classification provided by the present application. The following describes the system, electronic device, and storage medium for improving the accuracy of Mura defect classification provided by the present application: See also Figure 3 The present application provides an embodiment of a system for improving the accuracy of mura defect classification, including: An imaging unit 301 is used to obtain an image on the display screen to be inspected, where the image is a Mura defect image; The filtering unit 302 is used to perform texture filtering on the image to obtain a filtered image; The processing unit 303 is used to perform grayscale processing and defect segmentation processing on the filtered image to obtain an image to be detected; The input unit 304 is used to input the image to be detected into the Mura classification model to obtain a classification result, where the classification result includes the type, location and severity of the Mura in the image to be detected. The Mura classification model is obtained by the following training method: acquiring first data collected by multiple sensors, where the multiple sensors include at least one of the following: a visual camera, an infrared camera and a synchronous controller; preprocessing the first data to obtain second data; extracting feature data from the second data; fusing the feature data according to an orthogonal sequence fusion algorithm to obtain a fusion result; and training to obtain the Mura classification model according to the fusion result.

[0050] Optionally, the training method in the input unit 304 is specifically used for: Acquire a product surface image captured by a visual camera, where the product surface image includes color information and texture information of the product; Acquire an infrared image of a product captured by an infrared camera, where the infrared image of the product includes temperature distribution information of the product; Obtaining the time information and spatial position information of the visual camera and the infrared camera respectively collected by the synchronization controller; The product surface image, the product infrared image, the time information and the spatial position information are organized into the first data.

[0051] Optionally, the training method in the input unit 304 is specifically used for: The noise of the product surface image and the product infrared image is removed according to the Gaussian filtering algorithm and the median filtering algorithm to obtain a de-noised image; Perform contrast enhancement and histogram equalization processing on the denoised image to obtain a processed image; The processed image, time information and spatial position information are normalized to obtain second data.

[0052] Optionally, the training method in the input unit 304 is specifically used for: Directly extract the texture features of the local binary pattern and gray-level co-occurrence matrix of the surface processed image, as well as the color features; Input the surface processed image into the first feature extraction layer of the convolutional neural network and output the deep features; The infrared processed image is input into the second feature extraction layer of the convolutional neural network, and the temperature distribution features are output.

[0053] Optionally, the training method in the input unit 304 is specifically used for: Merge texture features, color features, depth features, and temperature distribution features according to a preset order to obtain a merged result; According to a preset weighting algorithm, weights are assigned to the merging result, the time information, and the spatial position information respectively; The weighted merging result is orthogonally fused with the time information and spatial position information to obtain the fusion result.

[0054] Optionally, the training method in the input unit 304 is specifically used to: input the fusion result into the classification layer of the convolutional neural network for model training to obtain the Mura classification model.

[0055] Optionally, after the input unit 304, the method further includes: The visualization unit 305 is used to visualize the classification results and display them on the user interface.

[0056] See also Figure 4 , the present application provides an electronic device, including: Processor 401 , memory 402 , input / output unit 403 , and bus 404 .

[0057] The processor 401 is connected to the memory 402 , the input and output unit 403 , and the bus 404 .

[0058] The memory 402 stores a program, and the processor 401 calls the program to execute the following steps: Figure 1 The method for improving the classification accuracy of Mura defects in the embodiment shown in the figure or Figure 2 The training method for obtaining the Mura classification model in the illustrated embodiment.

[0059] The present application provides a computer-readable storage medium, on which a program is stored, and when the program is executed on a computer, the program performs the following steps: Figure 1 The method for improving the classification accuracy of Mura defects in the embodiment shown in the figure or Figure 2 The training method for obtaining the Mura classification model in the illustrated embodiment.

[0060] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0061] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0062] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0063] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0064] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for improving the accuracy of mura defect classification, characterized in that: include: Acquire an image of the display screen to be inspected, where the image is a Mura defect image; Performing texture filtering on the image to obtain a filtered image; Performing grayscale processing and defect segmentation processing on the filtered image to obtain an image to be detected; Inputting the image to be detected into a mura classification model to obtain a classification result, wherein the classification result includes the type, location, and severity of the mura in the image to be detected; The mura classification model is obtained by the following training method: Acquire first data collected by a plurality of sensors, wherein the plurality of sensors include at least one of the following: a visual camera, an infrared camera, and a synchronization controller; Preprocessing the first data to obtain second data; extracting feature data from the second data; The feature data are fused according to an orthogonal sequence fusion algorithm to obtain a fusion result; The Mura classification model is obtained by training according to the fusion result.

2. The method according to claim 1, characterized in that: The first data collected by the multiple sensors is obtained, and the multiple sensors include at least one of the following: a visual camera, an infrared camera, and a synchronous controller including: Acquire a product surface image captured by the visual camera, wherein the product surface image includes color information and texture information of the product; Acquire an infrared image of a product captured by the infrared camera, wherein the infrared image of the product includes temperature distribution information of the product; Acquire the time information and spatial position information of the visual camera and the infrared camera respectively collected by the synchronization controller; The product surface image, the product infrared image, the time information and the spatial position information are organized into the first data.

3. The method according to claim 2, characterized in that The preprocessing of the first data to obtain the second data includes: Removing noise from the product surface image and the product infrared image according to a Gaussian filtering algorithm and a median filtering algorithm to obtain a de-noised image; Performing contrast enhancement and histogram equalization processing on the denoised image to obtain a processed image; The processed image, the time information and the spatial position information are normalized to obtain the second data.

4. The method according to claim 3, characterized in that The processed image includes a surface processed image and an infrared processed image, and the extracting feature data from the second data includes: Directly extracting texture features of the local binary pattern and gray-level co-occurrence matrix of the surface-processed image, as well as color features; Inputting the surface processed image into a first feature extraction layer of a convolutional neural network and outputting a depth feature; The infrared processed image is input into the second feature extraction layer of the convolutional neural network, and the temperature distribution feature is output.

5. The method according to claim 4, characterized in that The fusing the feature data according to the orthogonal sequence fusion algorithm to obtain the fusion result comprises: Merge the texture feature, the color feature, the depth feature and the temperature distribution feature according to a preset order to obtain a merged result; assigning weights to the merging result, the time information, and the spatial position information respectively according to a preset weighting algorithm; The weighted merging result is orthogonally fused with the time information and the spatial position information to obtain a fusion result.

6. The method according to claim 5, characterized in that The training to obtain the Mura classification model according to the fusion result includes: inputting the fusion result into a classification layer of a convolutional neural network for model training to obtain the Mura classification model.

7. The method according to claim 1, characterized in that After obtaining the classification result, the method further includes: visualizing the classification result and displaying it on a user interface.

8. A system for improving the accuracy of mura defect classification, characterized in that: The system comprises: An imaging unit, used to obtain an image on the display screen to be inspected, wherein the image is a Mura defect image; A filtering unit, used for performing texture filtering on the image to obtain a filtered image; A processing unit, used for performing grayscale processing and defect segmentation processing on the filtered image to obtain an image to be detected; An input unit, used to input the image to be detected into a mura classification model to obtain a classification result, wherein the classification result includes the type, location and severity of mura in the image to be detected; The mura classification model is obtained by the following training method: Acquire first data collected by a plurality of sensors, wherein the plurality of sensors include at least one of the following: a visual camera, an infrared camera, and a synchronization controller; Preprocessing the first data to obtain second data; extracting feature data from the second data; The feature data are fused according to an orthogonal sequence fusion algorithm to obtain a fusion result; A mura classification model is obtained by training according to the fusion result.

9. An electronic device, characterized in that: include: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method for improving the accuracy of mura defect classification according to any one of claims 1 to 7. 10 . A computer-readable storage medium having a program stored thereon, wherein when the program is executed on a computer, the method for improving the accuracy of mura defect classification according to claim 1 is performed.

Images