Display screen abnormal point detection method and system based on image processing and classification model

Abstract

The invention provides a display screen abnormal point detection method and system based on image processing and a classification model. The method comprises the following steps: acquiring an image of a to-be-detected area of a display screen; performing image processing on the to-be-detected area image of the display screen, obtaining an abnormal area image set obtained by extraction according to a preset pixel size, and performing category marking on each abnormal area image in the abnormal area image set; and inputting the abnormal region image set and the corresponding category mark into a classification model, and detecting each abnormal region image in the abnormal region image set through the classification model to obtain a corresponding detection result. According to the invention, the detection precision and the detection efficiency can be obviously improved, and large-scale industrial production can be better supported.

Description

Technical Field

[0001] The present invention relates to the technical field of display screens, and in particular to a display screen abnormal point detection method and system based on image processing and classification models. Background Art

[0002] Display screen anomaly detection is an important task in industrial quality inspection, which aims to identify and locate subtle defects in the display screen to ensure product quality. Existing display screen anomaly detection methods generally use image processing algorithms alone or deep learning models alone.

[0003] The outlier detection solution based on image processing algorithm is mainly aimed at simple tasks and small data volume. It uses algorithms such as threshold segmentation, edge detection and template matching to analyze the pixel data of the display screen to detect outliers. Although the detection coverage of this solution is wide, it has low adaptability to the contour, size, texture, etc. of the outliers. Therefore, when facing high-resolution display screens or subtle outliers, the detection accuracy is often insufficient, especially when the outliers are small. It is difficult for this solution to distinguish subtle differences and it is easy to miss detection, which will not meet the high-precision and high-reliability industrial production requirements. In addition, in a large-scale production environment, the detection speed of this solution is slow, which is difficult to meet the high efficiency requirements of industrial production.

[0004] The detection scheme based on deep learning is mainly aimed at complex tasks and large data volumes. The model is trained by a large number of display screen abnormal samples to fit the parameters to identify abnormal points. Existing deep learning detection methods are usually based on convolutional neural networks (CNNs), which extract features from display screen images and learn from a large number of abnormal point features to detect abnormal points. There are usually two ways to process display screen images based on deep model abnormal point detection methods: one is to use the image of the entire display screen as the model input to extract and process the features of the entire image to predict the abnormal points in the display screen in order to avoid the loss of feature information. Due to the large size of the display, the image resolution is often very large, which makes the algorithm miss more when detecting tiny abnormal points, resulting in low detection accuracy. Another way to avoid the above problems is to crop and segment the display screen image into fixed small resolution images, use the small resolution images as the input of the deep learning model, and extract image features for abnormal point detection. Although this can effectively improve the detection accuracy of the algorithm for tiny abnormal points, for large resolution images, the number of image blocks after image cropping is huge, and the detection speed is too slow to meet industrial production. In addition, in order to improve accuracy, the data-driven deep learning detection method requires the collection and annotation of a large number of display screen samples with abnormal points. Due to the high cost of a single display screen and the huge human resource consumption in the data collection process, the cost of implementing the solution is also relatively high.

[0005] In summary, the existing technical solutions for detecting abnormal points on display screens cannot efficiently and accurately locate the specific positions of abnormal points, and are easily disturbed by the complex background of the display screen during the detection process. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a display screen abnormal point detection method and system based on image processing and classification models, which can significantly improve the detection accuracy and detection efficiency and better support large-scale industrial production.

[0007] In order to solve the above technical problems, the first technical solution adopted by the present invention is: The display screen abnormal point detection method based on image processing and classification model includes: Acquire the image of the area to be detected on the display screen; Performing image processing on the image of the area to be detected on the display screen, obtaining an abnormal area image set extracted according to a preset pixel size, and marking the category of each abnormal area image in the abnormal area image set; The abnormal region image set and its corresponding category labels are input into a classification model, and each abnormal region image in the abnormal region image set is detected by the classification model to obtain corresponding detection results.

[0008] Optionally, it also includes: Build a pre-trained classification model based on deep learning; The pre-trained classification model is trained using the abnormal region image sample set and its corresponding category labels, and the parameters of the pre-trained classification model are updated using a binary classification cross entropy loss function to obtain a classification model.

[0009] Optionally, the pre-trained classification model has a structure of a three-layer linear fully connected multilayer perceptron, including: The input layer goes to the first hidden layer, which outputs 64 spiritual units and uses the ReLU activation function; The first hidden layer to the second hidden layer outputs 32 neurons and uses a ReLU activation function; and The second hidden layer to the output layer outputs 1 neuron and uses the Sigmoid activation function.

[0010] Optionally, performing image processing on the image of the area to be detected on the display screen includes: Preprocessing the image of the area to be detected on the display screen to obtain a preprocessed image; Performing edge detection processing on the preprocessed image using a Sobel operator to obtain an edge-detected image; Performing Fourier transform on the image after edge detection to obtain a frequency domain image; Analyzing the statistics of the frequency domain image according to a preset frequency domain threshold to obtain abnormal points; According to the preset pixel size, an abnormal area image corresponding to each abnormal point is extracted from the image of the area to be detected on the display screen.

[0011] Optionally, the performing image processing on the image of the area to be detected on the display screen further includes: Acquire a certain number of specific frequency domain image sets corresponding to the images of the areas to be inspected on the specific display screen; Acquire a specific abnormal region image set manually extracted from the specific frequency domain image set according to a preset pixel size; The frequency domain threshold is adjusted according to the statistics of the specific abnormal region image set.

[0012] Optionally, it also includes: Using the adjusted frequency domain threshold to perform image processing on the plurality of specific display screen area images to be detected, and obtaining corresponding adjusted detection results through a classification model; If it is determined that the classification model needs to be fine-tuned based on the adjusted detection results, the classification model is fine-tuned based on the specific abnormal area image set and its corresponding manual category labels.

[0013] Optionally, acquiring the image of the area to be detected on the display screen includes: Acquire an original image containing the area to be detected on the display screen; Performing feature region detection of the largest area on the original image to confirm the area of ​​the display screen to be detected; Extracting and correcting the display screen area to be detected, and obtaining an image of the display screen area to be detected placed vertically; Obtain the abnormal point annotation information corresponding to the image of the area to be detected on the display screen.

[0014] Optionally, the performing category marking on each abnormal region image in the abnormal region image set includes: According to the abnormal point annotation information corresponding to the image of the area to be detected on the display screen, the category label corresponding to each abnormal area image in the abnormal area image set is obtained.

[0015] Another technical solution provided by the present invention is: A display screen abnormal point detection system based on image processing and classification model, including an image processing module and a classification model; The image processing module is configured to obtain an image of the display screen area to be detected; and perform image processing on the image of the display screen area to be detected, obtain an abnormal area image set extracted according to a preset pixel size, and classify each abnormal area image in the abnormal area image set; The classification model is configured to detect each abnormal region image in the abnormal region image set according to the input abnormal region image set and its corresponding category label, and obtain corresponding detection results.

[0016] Optionally, the classification model is obtained by first constructing a pre-trained classification model based on deep learning, then training the pre-trained classification model using a sample set of abnormal area images and their corresponding category labels, and updating the parameters of the pre-trained classification model using a binary cross entropy loss function.

[0017] The beneficial effect of the present invention is that the present invention first detects all abnormal area images that may have subtle abnormal points by using an image processing method and an accurate threshold, and then performs a second accurate detection and distinction of the image processing results by using a classification model trained using a sample set of abnormal area images and their corresponding category labels. As a result, the present invention can not only significantly improve the detection accuracy of subtle abnormal points in high-resolution display screens and reduce missed detection, but also improve detection efficiency at the same time, better supporting large-scale industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a flow chart of a display screen abnormal point detection method based on image processing and classification model provided in an embodiment of the present invention; Figure 2 A schematic diagram of the process of building and training a classification model in an embodiment of the present invention; Figure 3 Schematic diagram of the process of the abnormal point detection stage in an embodiment of the present invention; Figure 4 Schematic diagram of extracting the area to be detected on the display screen in an embodiment of the present invention; Figure 5 A schematic diagram of a process for optimizing and adjusting a statistical threshold in an image processing process in an embodiment of the present invention; Figure 6 A schematic diagram of a process for optimizing and adjusting parameters of a classification model in an embodiment of the present invention; Figure 7 A schematic diagram of an adaptive adjustment process in a display screen abnormal point detection method based on image processing and classification model provided in an embodiment of the present invention; Figure 8A schematic diagram of the structure of a display screen abnormal point detection system based on image processing and classification model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0020] Existing technologies for detecting outliers use only one of the image processing algorithm or deep learning model-based solutions, which is strongly related to the task (simple or complex). For tasks that can collect and annotate a large number of data samples, only data collection and high-quality annotation are required, and a good result can be fitted through a deep learning model to complete the detection task; for tasks with fewer data samples, difficult to annotate, or with lower detection difficulty, the image processing algorithm can well select the threshold and obtain the detection result. Therefore, for most tasks, only one of the solutions is required to complete the detection task.

[0021] The present invention is aimed at the task of detecting abnormal points on large-size display screens. First, the display screen needs to turn on the backlight to be clearly seen, which makes the abnormal point features relatively blurred; secondly, for large-size display screens (such as images with a size of 5000*5000), it is very difficult to mark abnormal points of approximately (5*5) size. Therefore, if only the image processing method is used, even if a precise threshold is selected, it may be interfered by the backlight, light source, etc., thereby affecting the accuracy of the detection results; and if only a deep learning model-based solution is used, deep learning model data collection and data annotation are very difficult, and the model has no way to directly receive images of such a large size.

[0022] Therefore, the present invention innovatively adopts an outlier point detection solution that combines image processing algorithm and deep learning model, so as to achieve the detection of outlier points on the display screen, and can effectively improve the detection accuracy and detection efficiency at the same time.

[0023] See also Figure 1 , Figure 1 A schematic flow chart of a display screen abnormal point detection method based on image processing and classification model provided in an embodiment of the present invention.

[0024] The embodiment of the present invention provides a display screen abnormal point detection method based on image processing and classification model, which at least includes the following detection process: S1: Acquire the image of the area to be detected on the display screen; S2: performing image processing on the image of the area to be detected on the display screen, obtaining an abnormal area image set extracted according to a preset pixel size, and marking a category of each abnormal area image in the abnormal area image set; Here, a detection is performed through the image processing to detect all areas in the image of the display screen area to be detected that may have abnormal points; then, each abnormal point is extracted from the image of the display screen area to be detected according to a preset pixel size to obtain an abnormal area image set; finally, each abnormal area image in the abnormal area image set is marked by category.

[0025] S3: Inputting the abnormal region image set and its corresponding category labels into a classification model, and detecting each abnormal region image in the abnormal region image set through the classification model to obtain corresponding detection results.

[0026] Here, the abnormal area image set obtained after one detection and its corresponding category labels are used as the input of the classification model. Compared with the existing technology that uses the entire image input or the full input after cropping, the data processing amount of the classification model will be greatly reduced, thereby significantly improving the detection efficiency. In addition, the classification model is a secondary precise detection based on the results of one detection, which can also significantly improve the accuracy of the detection and effectively avoid missed detection.

[0027] The detection process for abnormal points on display screens provided in this embodiment adopts a combination of image processing algorithms and deep learning classification models, which can not only significantly improve the detection accuracy of subtle abnormal points in high-resolution display screens and reduce missed detections; but also improve the detection efficiency at the same time and better support large-scale industrial production.

[0028] See also Figure 2 and Figure 3 , Figure 2 Schematic diagram of the process of building and training a classification model in an embodiment of the present invention; Figure 3 Schematic diagram of the process of the outlier detection stage in an embodiment of the present invention.

[0029] The embodiment of the present invention is based on Figure 1 The embodiment is further extended to provide a display screen abnormal point detection method based on image processing and classification model, which includes at least the following two stages: 1. Construction and training of classification models like Figure 2 As shown, this stage includes the following steps: Step 1-1, build a pre-trained classification model based on deep learning.

[0030] The classification model, in this embodiment, refers to a model that can classify each abnormal region image in the abnormal region image set into different categories or groups according to abnormal point features.

[0031] In this embodiment, the pre-trained classification model constructed has a structure of a multi-layer perceptron MLP with three linear layers. The structure specifically includes: The input layer to the first hidden layer is a fully connected layer, which outputs 64 mental units and uses the ReLU activation function; The first hidden layer to the second hidden layer is a fully connected layer, outputting 32 neurons and using the ReLU activation function; The second hidden layer to the output layer is a fully connected layer, which outputs 1 neuron and uses the Sigmoid activation function.

[0032] In addition, the loss function in the pre-trained classification model is preferably a binary cross entropy loss function; and the optimizer is preferably an Adam optimizer.

[0033] This step also includes initializing model parameters of the constructed pre-trained classification model, and the initialized model parameters include the number of neurons and the learning rate, etc.

[0034] It should be noted that, unlike the classification model structure of the prior art (such as resnet18 and other classification models with large parameters), the classification model constructed in this embodiment, its MLP layer number, intermediate layer dimension and other parameters are based on the input model image data is a small-size image (such as 32*32) obtained after image processing, and the model only needs to perform a simple binary classification task. This is a small-parameter model specially customized for the specific requirement.

[0035] The small parameter classification model structure adopted by the present invention can achieve lightweight deployment and significantly improve the reasoning speed. In order to avoid missed detection, the images of the abnormal point area detected by image processing in the present invention need to be as many as possible, including all possible abnormal points, so there are high requirements for the reasoning speed of the classification model. The three-layer MLP network specially created by the present invention has the characteristics of good classification effect and fast reasoning speed, so it can meet the needs well.

[0036] Step 1-2, using the abnormal area image sample set and its corresponding category labels to train the pre-trained classification model, and using the binary cross entropy loss function to update the parameters of the pre-trained classification model to obtain a classification model.

[0037] The abnormal region image sample set refers to a classification sample data set constructed by the abnormal region image set obtained by the image processing method and its corresponding category mark. Preferably, it is a classification sample data set constructed by processing the abnormal region image sample set obtained by the abnormal region in step S2 of the previous embodiment and its corresponding category mark.

[0038] Specifically, the process of acquiring the abnormal region image sample set may refer to the description of step 2-1 in the next embodiment.

[0039] Here, the constructed classification sample data set is input into the classification model constructed in step 1 to perform model training to identify outliers.

[0040] In some specific embodiments, the training process of the classification model includes: after the abnormal area image samples of a preset pixel size, such as 32*32 pixels, enter the classification model in batches, they are processed by the three fully connected layers of the classification model to obtain a probability value between 0.0 and 1.0. This probability interpretation makes the output value intuitive and easy to understand, and the output of the model can be directly interpreted as the probability that the abnormal point area image is predicted to be abnormal; after obtaining the classification probability of the current abnormal point area image, the classification loss is calculated according to the following binary cross entropy loss function formula to update the model parameters:

[0041] Among them, L 2 represents the classification and positioning loss, y i is the category label of the sample target; p i is the probability of the sample target predicted by a single fully connected layer; C is the total number of categories of all sample targets; the loss value L is obtained 2 Finally, the classification model parameters are updated according to the direction of loss value reduction to complete the training of the classification model.

[0042] 2. Outlier Detection After the classification model is trained, the classification model can be used in combination with image processing methods to detect abnormal points on the display screen. Figure 3 As shown, this stage includes the following steps: Step 2-1: Obtain an image of the area to be inspected on the display screen.

[0043] In some specific implementations, the acquisition of the image of the area to be detected on the display screen can be achieved through the following steps: 1. Collect display screen images; A high-resolution industrial camera can be optionally used to image the display screen without omission to ensure that the complete display screen image is captured; the acquired display screen image is then stored as the original image containing the display screen area to be detected for subsequent abnormal point detection.

[0044] 2. Generate an image dataset of the display screen area to be detected; The original image obtained in the previous step is processed and annotated to generate a dataset format suitable for image processing and classification models.

[0045] In an optional specific example, it specifically includes: A. Confirm the display screen area to be tested; Here, the original display screen image data (i.e. Figure 4 The largest feature area is detected (as shown on the left of the figure), and the largest white connected area in the image is detected, which is the area where the display screen is located, and it is confirmed as the display screen area to be detected.

[0046] B. Extract the display screen area to be detected; Here, the display screen area to be detected confirmed in the previous step only occupies a part of the original image and may not be placed vertically. Therefore, it is necessary to perform correction and extraction operations such as rotation and cropping to obtain an image to be detected that only contains the display screen area, that is, the display screen area image to be detected ( Figure 4 ).

[0047] C. Mark the outlier area.

[0048] A dedicated image annotation tool (such as LabeImg) can be used to manually annotate all abnormal point areas in the image of the display screen area to be detected obtained in the previous step to generate corresponding annotation information.

[0049] It should be noted that whether it is the sample data set used for classification model training or the image data set of the display screen area to be inspected during actual detection, it can be obtained through the above steps to improve the data quality of the data set and help improve the accuracy of the detection results.

[0050] In step 2-2, image processing is performed on the image of the area to be detected on the display screen to detect all areas in the image that may contain abnormal points.

[0051] In some specific implementations, this step can be implemented by the following sub-steps: 1. Preprocess the image of the area to be detected on the display screen to obtain the preprocessed image; The preprocessing includes removing noise from the image using smoothing techniques such as Gaussian filtering to prevent the noise from affecting subsequent edge detection; and converting to grayscale image display. This is because edge detection algorithms usually perform better on grayscale images. As a result, the image quality can be optimized to a certain extent and the accuracy of the image processing results can be improved.

[0052] 2. Perform edge detection on the preprocessed image using a Sobel operator to obtain an edge-detected image; Here, the edge detection process specifically includes: using the Sobel operator to perform a differential operation on the preprocessed image obtained in the previous step, and calculating the gradients of the image in the horizontal (Gx) and vertical (Gy) directions respectively. The Sobel operator is used to detect the edge of the image and emphasize the high-frequency change parts in the image. Through the gradients in the horizontal and vertical directions, the gradient amplitude of each pixel can be calculated by the following formula: ; After that, edge enhancement is performed. When performing edge enhancement, a threshold is usually set to retain edge pixels above this threshold and remove weak edges below the threshold.

[0053] 3. Perform Fourier transform on the image after edge detection to obtain a frequency domain image; Here, a two-dimensional Fourier transform is performed on the edge-detected image obtained in the previous step to convert the image from the spatial domain to the frequency domain, extract the frequency information of the image, and obtain a frequency domain image. The Fourier transform can help detect periodic features and global abnormal patterns in the image.

[0054] 4. Analyze the statistics of the frequency domain image according to a preset frequency domain threshold to obtain abnormal points; Here, by performing statistical analysis on the frequency domain image obtained in the previous step, the mean, variance, and other statistics describing the distribution characteristics of the image data are extracted; then the statistics of the image are screened through a pre-set reasonable threshold range; and the areas exceeding the threshold are marked as outliers. For example, by setting a range of the mean plus or minus n times the standard deviation, the pixels in the image that fall outside this range are marked as outliers.

[0055] 5. According to the preset pixel size, the abnormal area image corresponding to each abnormal point is extracted from the image of the area to be detected on the display screen.

[0056] Here, based on the abnormal points extracted in the previous step, the corresponding abnormal point area is found in the original image of step 2-1, and then the abnormal area image is extracted according to the preset pixel size (such as 32*32) to form an abnormal area image set for subsequent classification tasks.

[0057] In some other specific embodiments, it is also supported to flexibly adjust the extraction size of the abnormal area image according to the different sizes of the display screen. By adjusting the input image block, that is, the size and coverage of the abnormal area image, the classification model can adapt to display screens of different sizes and resolutions, thereby improving the flexibility and efficiency of detection; so as to optimize the detection for display screens of different sizes.

[0058] In addition, the category label corresponding to the extracted abnormal point area image will be determined based on the abnormal point annotation information obtained in step C of the above step 2-1.

[0059] It should be noted that the existing image processing process only includes steps 1, 2 and 4 above, that is, after edge detection is performed using the edge detection operator (Sobel), statistical analysis and detection of abnormal areas are performed directly according to the detected edge values. However, the above-mentioned image processing process of this embodiment, in particular, adds step 3 between steps 2 and 4, that is, a Fourier transform operation is added between edge detection and statistical analysis. The consideration of doing so is strongly related to the technical field of the present invention, that is, the task. Since the display screen image is taken when the display screen emits a white backlight, this will cause the abnormal area in the image to be blurred due to the influence of the light source. By converting it to the frequency domain space, the difference between the abnormal area and the normal area can be enhanced, so as to better perform statistical analysis and abnormal point detection, thereby improving the accuracy of the image processing results for the display screen image.

[0060] In step 2-3, each abnormal region image in the abnormal region image set is marked with a category.

[0061] Here, specifically based on the abnormal point annotation information corresponding to the image of the display screen area to be detected, that is, the processing result of step C under step 2-1, the category label corresponding to each abnormal area image in the abnormal area image set is obtained.

[0062] In steps 2-4, use the classification model to detect outliers.

[0063] Here, the image processing results of steps 2-3, i.e., the abnormal area image set obtained by the first detection and its corresponding category label, are input into the trained classification model for secondary accurate detection of whether it is an abnormal point, and the detection result corresponding to each abnormal area image is obtained.

[0064] The display screen abnormal point detection method based on image processing and classification model provided in this embodiment, first, based on a specific image processing algorithm, select a relatively extreme threshold value, detect all possible abnormal points as much as possible, and extract each abnormal area image according to the preset pixel size; a large amount of abnormal area image data and its category mark will be used as a training sample of the classification model to achieve the purpose of training the classification model, making it more targeted and improving the detection accuracy. Then, in the actual application process, all possible abnormal area images are detected and marked with corresponding category marks based on a specific image processing algorithm, and then input into the trained classification model to obtain the detection result after a second accurate detection. Here, since the classification model only needs to process the result of a feature detection, that is, the image that may have abnormal points and is extracted in a small size, without processing the entire display screen image or a large amount of image data after it is cropped; at the same time, classifying and marking small-size abnormal area images (such as 32*32) is simpler than directly positioning and marking, and can adapt to the task well and obtain good detection results. Therefore, the classification model can accurately identify and locate tiny abnormal points in the image, thereby significantly improving the detection efficiency and detection accuracy.

[0065] Please refer to Figure 5 , Figure 5 The figure is a schematic diagram of a process for optimizing and adjusting a statistical threshold in an image processing process in an embodiment of the present invention.

[0066] The embodiment of the present invention is further extended based on the previous embodiment, and further provides a method for optimizing the statistical threshold used in the image processing process during the construction and training stage of the classification model.

[0067] Since different types of display screens have different standards for judging brightness peaks and abnormal points under white backlight, when testing specific display screen products on an actual production line, it is often necessary to tune the thresholds in the statistical analysis in step 4 under step 2-2 in the image processing flow during the model training phase, so that the image processing algorithm can better adapt to the conditions of specific display screen products and further optimize the image processing results.

[0068] Specifically, in the model training stage in the previous embodiment, this embodiment includes the following optimization statistical threshold process in the process of performing image processing on the image of the area to be detected of the display screen through step 2-2 in the image processing process: 6. Obtain a certain number of specific frequency domain image sets corresponding to the images of the areas to be inspected on the display screen; Here, the several specific display screen area images to be inspected refer to the results of processing the first few specific display screen products on the production line or the selected few specific display screen products according to step 2-1; the specific frequency domain image set refers to the specific frequency domain image set obtained by processing the specific display screen area images to be inspected according to steps 1-3 under step 2-2.

[0069] 7. Obtaining a specific abnormal region image set manually extracted from the specific frequency domain image set according to a preset pixel size; Here, the specific abnormal region image set refers to the result of dividing each image in the specific frequency domain image set into possible abnormal point regions.

[0070] 8. Adjust the frequency domain threshold according to the statistics of the specific abnormal area image set.

[0071] The statistics here refer to the results obtained by statistical analysis of the above-mentioned specific abnormal area image set.

[0072] This embodiment adjusts the frequency domain threshold in the image processing process based on the statistics of the specific display screen product to achieve the purpose of optimizing image processing according to the characteristics of the specific display screen product; it can make the image processing algorithm better adapt to the situation of the specific display screen product, thereby optimizing the image processing result.

[0073] Please refer to Figure 6 , Figure 6 The figure is a schematic diagram of a process for optimizing and adjusting parameters of a classification model in an embodiment of the present invention.

[0074] The embodiment of the present invention is further expanded based on the previous embodiment, and also provides a method for optimizing the classification model therein.

[0075] When testing specific display screen products on an actual production line, in addition to adaptive adjustment and optimization in the image processing process, the classification model can also be adaptively adjusted and optimized to achieve the effect of optimizing the overall deployment of the display screen anomaly detection method.

[0076] In this embodiment, after optimizing and adjusting the frequency domain threshold in the image processing algorithm, the following steps are further included: 9. Use the adjusted frequency domain threshold to perform image processing on the specific display screen area to be detected images of the number of said specific abnormal area images and their corresponding category labels, and then obtain the corresponding adjusted detection results through the classification model; 10. If it is determined that the classification model needs to be fine-tuned based on the adjusted detection results, the classification model is fine-tuned based on the specific abnormal area image set and its corresponding manual category labels.

[0077] Here, whether the classification model needs to be fine-tuned is determined based on whether the classification result of the classification model is correct.

[0078] The fine-tuning is specifically carried out according to a number of specific display screen area images to be detected selected when adjusting the threshold in the image processing process, by manually marking the abnormal point areas, and using the marked data to fine-tune the classification model, so that the classification model algorithm can better adapt to the situation of specific display screen products.

[0079] Combination Figure 7 It can be seen that the embodiment of the present invention can also adaptively adjust the thresholds in the image processing process and the parameters of the classification model according to the specific display screen product, so as to make the image processing results and the results of the classification model, that is, the final detection results, more accurate.

[0080] See also Figure 8 , Figure 8 A schematic diagram of the structure of a display screen abnormal point detection system based on image processing and classification model provided in an embodiment of the present invention.

[0081] like Figure 8 As shown, based on any of the above embodiments, the embodiment of the present invention further provides a display screen abnormal point detection system 100 based on image processing and classification model, including: Image processing module 1 and classification model 2; The image processing module 1 is configured to obtain an image of the display screen area to be detected; and perform image processing on the image of the display screen area to be detected, obtain an abnormal area image set extracted according to a preset pixel size, and classify each abnormal area image in the abnormal area image set; The classification model 2 is configured to detect each abnormal region image in the abnormal region image set according to the input abnormal region image set and its corresponding category label, and obtain corresponding detection results.

[0082] In some specific embodiments, the classification model is obtained by first constructing a pre-trained classification model based on deep learning, then training the pre-trained classification model using a sample set of abnormal area images and their corresponding category labels, and updating the parameters of the pre-trained classification model using a binary cross entropy loss function.

[0083] In some other specific implementations, the image processing module can implement all the steps of the image processing process in any of the above embodiments; the classification model can implement all the steps of the classification model in any of the above embodiments. The specific steps will not be repeated here, please refer to the records of the above embodiments for details.

[0084] The display screen abnormal point detection method based on image processing and classification model and the display screen abnormal point detection system based on image processing and classification model provided by the present invention have the following advantages: 1. Improved detection accuracy of high resolution and tiny outliers. The present invention combines image processing algorithms and classification models, which can effectively deal with the detection problems of high-resolution display screens and tiny outliers by utilizing multi-scale image input and deep learning feature extraction capabilities; it can also better distinguish subtle differences through refined feature learning of classification models, significantly improve detection accuracy, reduce missed detection, and meet the high-precision requirements in industrial production.

[0085] 2. Balance between automated detection and processing speed. Without affecting the detection accuracy, the present invention achieves a significant improvement in detection speed through a reasonable image cropping strategy and efficient calculation of the classification model. Compared with traditional full-image detection, the present invention optimizes the input and processing flow of image blocks, ensuring the speed advantage while efficiently detecting tiny anomalies, and can better support large-scale industrial production.

[0086] 3. Low data dependence and efficient training. The present invention does not need to rely on a large amount of labeled abnormal sample data, and can learn effective features from less sample data through the powerful generalization ability of the classification model. Compared with traditional deep learning methods, it can achieve a higher detection accuracy improvement with limited sample data, reduce dependence on large-scale data sets, and reduce the labor cost of data collection and annotation.

[0087] From the above description, it can be known that a person skilled in the art can understand that all or part of the processes in the above technical solutions can be realized by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the above methods. After being executed by the processor, the program can also achieve the beneficial effects of the corresponding methods.

[0088] The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0089] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A display screen abnormal point detection method based on image processing and classification model, characterized in that: include: Acquire the image of the area to be detected on the display screen; Performing image processing on the image of the area to be detected on the display screen, obtaining an abnormal area image set extracted according to a preset pixel size, and marking the category of each abnormal area image in the abnormal area image set; The abnormal region image set and its corresponding category labels are input into a classification model, and each abnormal region image in the abnormal region image set is detected by the classification model to obtain corresponding detection results.

2. The display screen abnormal point detection method based on image processing and classification model as claimed in claim 1, characterized in that: Also includes: Build a pre-trained classification model based on deep learning; The pre-trained classification model is trained using the abnormal region image sample set and its corresponding category labels, and the parameters of the pre-trained classification model are updated using a binary classification cross entropy loss function to obtain a classification model.

3. The display screen abnormal point detection method based on image processing and classification model as claimed in claim 2, characterized in that: The pre-trained classification model has a structure of a three-layer linear fully connected multi-layer perceptron, including: The input layer goes to the first hidden layer, which outputs 64 spiritual units and uses the ReLU activation function; The first hidden layer to the second hidden layer outputs 32 neurons and uses a ReLU activation function; and The second hidden layer to the output layer outputs 1 neuron and uses the Sigmoid activation function.

4. The display screen abnormal point detection method based on image processing and classification model as claimed in claim 1, characterized in that: The performing image processing on the image of the area to be detected on the display screen includes: Preprocessing the image of the area to be detected on the display screen to obtain a preprocessed image; Performing edge detection processing on the preprocessed image using a Sobel operator to obtain an edge-detected image; Performing Fourier transform on the image after edge detection to obtain a frequency domain image; Analyzing the statistics of the frequency domain image according to a preset frequency domain threshold to obtain abnormal points; According to the preset pixel size, an abnormal area image corresponding to each abnormal point is extracted from the image of the area to be detected on the display screen.

5. The display screen abnormal point detection method based on image processing and classification model as claimed in claim 4, characterized in that: The image processing of the image of the area to be detected on the display screen also includes: Acquire a certain number of specific frequency domain image sets corresponding to the images of the areas to be inspected on the specific display screen; Acquire a specific abnormal region image set manually extracted from the specific frequency domain image set according to a preset pixel size; The frequency domain threshold is adjusted according to the statistics of the specific abnormal region image set.

6. The display screen abnormal point detection method based on image processing and classification model as claimed in claim 5, characterized in that: Also includes: Using the adjusted frequency domain threshold to perform image processing on the plurality of specific display screen area images to be detected, and obtaining corresponding adjusted detection results through a classification model; If it is determined that the classification model needs to be fine-tuned based on the adjusted detection results, the classification model is fine-tuned based on the specific abnormal area image set and its corresponding manual category labels.

7. The display screen abnormal point detection method based on image processing and classification model as claimed in claim 1, characterized in that: The step of acquiring the image of the area to be detected on the display screen includes: Acquire an original image containing the area to be detected on the display screen; Performing feature region detection of the largest area on the original image to confirm the area of ​​the display screen to be detected; Extracting and correcting the display screen area to be detected, and obtaining an image of the display screen area to be detected placed vertically; Obtain the abnormal point annotation information corresponding to the image of the area to be detected on the display screen.

8. The display screen abnormal point detection method based on image processing and classification model as claimed in claim 7, characterized in that: The step of marking the category of each abnormal region image in the abnormal region image set comprises: According to the abnormal point annotation information corresponding to the image of the area to be detected on the display screen, the category label corresponding to each abnormal area image in the abnormal area image set is obtained.

9. A display screen abnormal point detection system based on image processing and classification model, characterized in that: Includes image processing modules and classification models; The image processing module is configured to obtain an image of the display screen area to be detected; and perform image processing on the image of the display screen area to be detected, obtain an abnormal area image set extracted according to a preset pixel size, and classify each abnormal area image in the abnormal area image set; The classification model is configured to detect each abnormal region image in the abnormal region image set according to the input abnormal region image set and its corresponding category label, and obtain corresponding detection results.

10. The display screen abnormal point detection system based on image processing and classification model as claimed in claim 9, characterized in that: The classification model is obtained by first constructing a pre-trained classification model based on deep learning, then training the pre-trained classification model using an abnormal area image sample set and its corresponding category labels, and updating the parameters of the pre-trained classification model using a binary cross entropy loss function.

Images