A deep learning-based card slot defect image detection method, device and equipment

Abstract

The application relates to the technical field of image recognition, and provides a card slot defect image detection method, device and equipment based on deep learning, which comprises the following steps: obtaining a product original image with a card slot to make an image data set; extracting a card slot part sub-image set from the original image data set; scaling all images of the original image data set and the card slot part sub-image set to the same preset resolution size to obtain a mixed image data set; training and testing a target detection model and a two-classification model by using the mixed image data set; feeding a to-be-detected image into the trained target detection model and two-classification model in sequence; and determining the card slot position and whether the card slot is defective according to the model output result. The application has the advantages that the pixel amount of the actual input model is small, operation is fast, the whole method occupies less resources and is concentrated in a more valuable image area, and the product with the card slot can be detected more efficiently and accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a card slot defect image detection method, device and equipment based on deep learning. BACKGROUND

[0002] In the process of manufacturing display panels, the generation of surface defects of display panels is often inevitable. The impurities, breakage, stains and the like on the surface not only affect the appearance and comfort of the product, but also have adverse effects on the use performance thereof. Therefore, production enterprises attach great importance to the detection of the appearance defects of the product so as to discover in time, thereby effectively controlling the product quality, and analyzing some problems existing in the production process according to the detection result, so as to eliminate or reduce the generation of defective products.

[0003] Manual detection is the most original method for detecting the appearance defects of display panels, and this method has poor real-time performance, low efficiency, high labor intensity and is greatly affected by the experience and subjective factors of workers. In the existing card slot defect image detection scheme, in order to preserve the clarity of the defect area, the defect detection algorithm is calculated on a high-resolution image. Since the defect area is relatively small in the whole image area, most of the pixels involved in the calculation are irrelevant to the defect, and the computing power is wasted. Or, in order to pursue speed, the defect detection algorithm is calculated on a low-resolution image after down-sampling. Since the defect area itself occupies a small number of pixels, the features of the area are more blurred after down-sampling, and it is difficult to detect accurately. SUMMARY

[0004] Therefore, the technical problem to be solved by the present application is to provide a card slot defect image detection method, device and equipment based on deep learning, so as to overcome the problem that the relative size of the target object on the whole image is too small, thereby making the detection difficult and the accuracy low.

[0005] To solve the above technical problem, the present application provides a card slot defect image detection method based on deep learning, comprising: taking an original image of a product with a card slot as an original image data set by using a camera; labeling the part where the card slot is located on the original image data set, and extracting a card slot part sub-image set; scaling the images of the original image data set and the card slot part sub-image set to the same preset resolution size to obtain a mixed image data set and its label; training and testing a target detection model by using the mixed image data set; finely labeling the accurate position of the card slot on the card slot part sub-image set to obtain a card slot fine position sub-image, labeling the defect position in the card slot fine position sub-image, and making a card slot defect classification data set, which is divided into two categories of defective and non-defective, training and testing a two-classification model by using the card slot defect classification data set;

[0006] Obtaining a to-be-detected image photographed by a camera, adjusting the resolution of the to-be-detected image to a preset resolution size, inputting the to-be-detected image into a target detection model completed training to obtain a card slot region detection frame, intercepting a card slot part sub-image in the card slot region detection frame, adjusting the resolution of the card slot part sub-image to the preset resolution size again, inputting the card slot part sub-image into the target detection model completed training to obtain a card slot region accurate detection frame, and intercepting a card slot fine position sub-image in the card slot region accurate detection frame; inputting the card slot fine position sub-image into a binary classification model to obtain a classification result, and determining whether the card slot has a defect.

[0007] Preferably, the product with a card slot original image is more than 50 million pixels.

[0008] Preferably, the card slot part is labeled, the labeling information includes coordinates and a category of the card slot part, and is saved as a txt file.

[0009] Preferably, the images of the original image data set and the images of the card slot part sub-image set are all scaled to a same preset resolution size, and the preset resolution size is 224*224.

[0010] The resolution size of the image of the original image data set is set to the image resolution size of the image of the card slot part sub-image set.

[0011] Preferably, the number of defective images is expanded by using part of the images without defects in the card slot defect classification data set as a basis to perform image distortion, damage and modification, so as to balance the number of two types of images in the card slot defect classification data set, and obtain a target card slot defect classification data set.

[0012] Preferably, the target detection model is based on a YOLO target detection framework, part of samples of a mixed image data set are selected as a batch input into the target detection model for training until the network converges, and the model parameters with the minimum loss are reserved.

[0013] The target detection model is a one-stage regression model, the model size is 50M, and the model output result includes target positioning boundary, category and confidence.

[0014] Preferably, the binary classification model has a model size of 10M, and the model output result is greater than or equal to 0.5, which is determined as a card slot defect, and the model output result is less than 0.5, which is determined as a card slot without defect.

[0015] The application also provides a card slot defect image detection device based on deep learning, which comprises:

[0016] An original image data set acquisition module: an original image data set is obtained by photographing a product with a card slot by a camera;

[0017] card slot part sub-image set acquisition module: labeling the part where the card slot is located on the original image dataset, and extracting the card slot part sub-image set;

[0018] mixed image dataset acquisition module: scaling the images of the original image dataset and the images of the card slot part sub-image set to the same preset resolution size to obtain a mixed image dataset and its label;

[0019] target detection model training module: training and testing a target detection model using the mixed image dataset;

[0020] card slot fine position sub-image acquisition module: fine labeling the accurate position of the card slot on the card slot part sub-image set to obtain a card slot fine position sub-image;

[0021] card slot defect classification dataset acquisition module: labeling the defect position in the card slot fine position sub-image to make a card slot defect classification dataset, which is divided into two categories: defective and non-defective;

[0022] two-classification model training module: training and testing a two-classification model using the card slot defect classification dataset;

[0023] defect detection module: obtaining a to-be-detected image captured by a camera, adjusting the resolution of the to-be-detected image to a preset resolution size, inputting the to-be-detected image into the trained target detection model to obtain a card slot region bounding box, cutting a card slot part sub-image in the card slot region bounding box, adjusting the resolution of the card slot part sub-image to the preset resolution size again, inputting the card slot part sub-image into the trained target detection model to obtain a card slot region accurate bounding box, cutting a card slot fine position sub-image in the card slot region accurate bounding box, and inputting the card slot fine position sub-image into the trained two-classification model to obtain a classification result and determine whether the card slot has a defect.

[0024] The application also provides a card slot defect image detection device based on deep learning, comprising:

[0025] an image acquisition device configured to obtain an original image of a product with a card slot captured by a camera;

[0026] a host computer in communication connection with the image acquisition device, configured to execute a computer program to realize the steps of the card slot defect image detection method based on deep learning according to any one of the above.

[0027] The above technical solution of the application has the following advantages compared with the prior art:

[0028] The card slot defect image detection method based on deep learning provided by the application, by training to obtain a target detection model and a binary classification model, inputting a to-be-detected image into the target detection model to obtain a card slot part sub-image, and inputting the card slot part sub-image into the target detection model to obtain a card slot fine position sub-image, whether there is a defect on a very small component compared with a full image is detected, if every pixel of the full image is detected, most of the calculation is wasted in unimportant areas; and the same target detection model is used for twice detection provided by the application, the model parameters are unchanged, the whole method occupies less resources and is concentrated in more valuable image areas, the computing power is focused by calculating the thumbnail in the twice detection, so that the product with a card slot is detected more efficiently and accurately. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to make the content of the application more easily understood, the application will be further described in detail below according to specific embodiments of the application and in conjunction with the drawings, in which

[0030] Figure 1 is a flowchart of the card slot defect image detection method based on deep learning provided by the application;

[0031] Figure 2 is a training flowchart of the card slot defect image detection method based on deep learning provided by the application;

[0032] Figure 3 is an original image example diagram;

[0033] Figure 4 is a card slot fine position sub-image;

[0034] Figure 5 is a card slot defect classification image example diagram;

[0035] Figure 6 is a C1, C2, C3, C4 data set relationship diagram;

[0036] Figure 7 is a card slot defect image detection flowchart based on deep learning provided by the application. DETAILED DESCRIPTION

[0037] The application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the application and implement it, but the embodiments are not as a limitation on the application.

[0038] Referring to Figure 1 , Figure 1 is a card slot defect image detection method flowchart based on deep learning provided by the application:

[0039] Step S101: Take the original image of the product with the card slot as the original image dataset by using the camera;

[0040] The original image of the product with the card slot is more than 50 million pixels.

[0041] Step S102: Label the card slot part in the original image dataset and extract the card slot part sub-image set;

[0042] The label information includes the coordinates and category of the card slot part, and is saved as a txt file.

[0043] Step S103: Scale the images of the original image dataset and the card slot part sub-image set to the same preset resolution size to obtain a mixed image dataset and its label, including:

[0044] The image resolution size of the original image dataset is set to the image resolution size of the card slot part sub-image set.

[0045] Step S104: Train and test the target detection model using the mixed image dataset;

[0046] The target detection model is based on the YOLO target detection framework, and part of the sample of the mixed image dataset is selected as a batch input to the target detection model for training until the network converges, and the model parameter with the smallest loss is retained.

[0047] The target detection model is a one-stage regression model with a model size of 50M, and the model output result includes target positioning boundary, category and confidence.

[0048] Step S105: Fine label the accurate position of the card slot on the card slot part sub-image set to obtain a card slot fine position sub-image;

[0049] Step S106: Label the defect position in the card slot fine position sub-image to make a card slot defect classification dataset, which is divided into two categories: defective and non-defective, including:

[0050] Use part of the non-defective images in the card slot defect classification dataset as the basis for image distortion, damage and modification to expand the number of defective images, so as to balance the number of images in the two categories in the card slot defect classification dataset, and obtain a target card slot defect classification dataset.

[0051] Step S107: Train and test the two-classification model using the card slot defect classification dataset;

[0052] The binary classification model has a model size of 10M, and if the model output result is greater than or equal to 0.5, it is determined that the card slot is defective, and if the model output result is less than 0.5, it is determined that the card slot is not defective.

[0053] Step S108: Obtain the image to be detected photographed by the camera, adjust the resolution of the image to be detected to a preset resolution size, input the image to the target detection model that has been trained to obtain a card slot region detection frame, cut a card slot part sub-image in the card slot region detection frame, adjust the resolution of the card slot part sub-image to a preset resolution size again, input the card slot part sub-image into the target detection model that has been trained to obtain a card slot region accurate detection frame, and cut a card slot fine position sub-image in the card slot region accurate detection frame; input the card slot fine position sub-image into the binary classification model that has been trained to obtain a classification result, and determine whether the card slot is defective.

[0054] Referring to Figure 2 , Figure 2 is a training flowchart of the card slot defect image detection method based on deep learning, and the specific implementation manner is as follows:

[0055] Step S201: photograph an original image of a product with a card slot by using a camera;

[0056] Referring to Figure 3 , Figure 3 is an original image example, and the resolution thereof is about 6144*9400 pixels.

[0057] Step S202: obtain the original image photographed by the camera by using a computer system to obtain an original image data set C1;

[0058] Step S203: mark the card slot part in the original image data set C1, mark the coordinates and categories of the card slot part, save as a txt file, cut the card slot sub-region image in the marking frame, and add the card slot part sub-image to a card slot part sub-image set C2;

[0059] Step S204: finely mark the accurate position of the card slot on the image of the card slot part sub-image set C2 again;

[0060] Referring to Figure 4 , Figure 4 is a card slot fine position sub-image.

[0061] Step S205: scale all the images of the original image data set and the images of the card slot part sub-image set to 512*512 pixel sizes to obtain a mixed image data set C3;

[0062] Step S206: select part of the images in the mixed image data set C3 as a training set, and the rest as a test set;

[0063] Step S207: training the target detection model, comprising:

[0064] establishing a deep learning model based on a YOLO target detection framework;

[0065] inputting multiple training set image data samples as a batch into the target detection model for training;

[0066] repeatedly inputting the entire training set image for multiple rounds until the network converges;

[0067] using the test set image to verify the current model loss s at the end of each round of training, and retaining the model parameters with the smallest loss.

[0068] Step S208: inputting the scaled pictures in the original image data set C1 into the trained target detection model to obtain the relative coordinates of the card slot part, and then converting them into the coordinates on the original image, cutting the sub-image of the card slot part from the original image, inputting the sub-image of the card slot part into the same target detection model to obtain the fine position coordinates of the card slot on the sub-image, cutting the sub-image of the fine position of the card slot from the sub-image of the card slot part, and making a card slot defect classification data set C4;

[0069] Referring to Figure 5 , Figure 5 is a card slot defect classification image example, and "OK" represents a non-defective card slot image, and "NG" represents a defective card slot image.

[0070] Referring to Figure 6 , Figure 6 is a C1, C2, C3, C4 data set relationship schematic diagram, C1 is an original image data set, the image size is 6144*9400 pixels; C2 is a card slot part sub-image set, the image size is 512*512 pixels; C3 is a mixed image data set, the image size is 512*512 pixels; and C4 is a card slot defect classification data set.

[0071] Step S209: making a binary classification label for the images in the card slot defect classification data set C4, and dividing them into two categories of defective and non-defective. Since the number of non-defective images is much larger than that of defective images, partial non-defective images are used as the basis for image damage, distortion, and modification to expand the number of defective images, so that the original number of two types of images is roughly the same.

[0072] Step S210: training a binary classification model, comprising:

[0073] The card slot defect classification data set C4 extracts part of the picture as a training sample and inputs the binary classification deep learning model for training, and the remaining card slot defect classification data set picture is used to verify the model until the accuracy no longer improves, and a trained binary classification deep learning model is obtained.

[0074] Referring to Figure 7 As shown in the figure, Figure 7 The card slot defect image detection flowchart based on deep learning is provided by the present application, and the specific implementation manner is as follows:

[0075] Step S701: acquiring a to-be-detected image photographed by a camera;

[0076] Step S702: adjusting the resolution of the to-be-detected image to 512*512 pixel size, and inputting the to-be-detected image into a trained target detection model, wherein the trained target detection model outputs a card slot region detection frame coordinate;

[0077] Step S703: converting the card slot region detection frame coordinate into an original image coordinate, and cutting a sub-image of a card slot region from an original image;

[0078] Step S704: adjusting the resolution of the sub-image of the card slot region to 512*512 pixel size, and inputting the sub-image into the trained target detection model, wherein the trained target detection model outputs a card slot region accurate detection frame coordinate;

[0079] Step S705: converting the card slot region accurate detection frame coordinate into an original image coordinate, and cutting a sub-image of a card slot fine position from the original image;

[0080] Step S706: inputting the sub-image of the card slot fine position into a trained binary classification model, wherein if an output result of the trained binary classification model is greater than or equal to 0.5, it is determined that the card slot has a defect, and if the output result of the trained binary classification model is less than 0.5, it is determined that the card slot has no defect.

[0081] In the embodiment, a deep learning-based card slot defect image detection method is trained, in the first detection, the image to be detected is down-sampled to 512*512 resolution, the card slot region is roughly determined through a target detection model, and the model processes 512*512 pixels; in the second detection, the roughly determined region is further subjected to a target detection model to find the accurate small region where the card slot is located, and the model also processes 512*512 pixels; whether there is a defect on a very small component compared with the whole image is detected, if every pixel of the whole image is detected, most of the calculation is wasted in unimportant areas, and the two detections provided by the application use the same target detection model, the model parameters are unchanged, the whole method occupies less resources and is concentrated in more valuable image areas; finally, only the accurate small region where the card slot is located is sent into a binary classification model for judgment, and the product with the card slot is detected more efficiently and accurately.

[0082] The embodiment of the application also provides a deep learning-based card slot defect image detection device, which comprises:

[0083] An original image data set acquisition module: original images of products with card slots are shot by a camera as original image data sets;

[0084] A card slot part sub-image set acquisition module: the parts where the card slots are located are labeled on the original image data sets, and card slot part sub-image sets are extracted;

[0085] A mixed image data set acquisition module: images of the original image data sets and images of the card slot part sub-image sets are all scaled to the same preset resolution size to obtain a mixed image data set and its label;

[0086] A target detection model training module: a target detection model is trained and tested by using the mixed image data set;

[0087] A card slot fine position sub-image acquisition module: the accurate positions of the card slots are finely labeled again on the card slot part sub-image sets to obtain card slot fine position sub-images;

[0088] A card slot defect classification data set acquisition module: defect positions are labeled in the card slot fine position sub-images to make a card slot defect classification data set, which is divided into two categories of defective and non-defective;

[0089] A binary classification model training module: a binary classification model is trained and tested by using the card slot defect classification data set;

[0090] The defect detection module: acquires the image to be detected photographed by the camera, adjusts the resolution of the image to be detected to a preset resolution size, inputs into the target detection model completed training to obtain a detection frame of the card slot area, intercepts a card slot part sub-image in the detection frame of the card slot area, adjusts the resolution of the card slot part sub-image to the preset resolution size again, inputs the card slot part sub-image into the target detection model completed training to obtain an accurate detection frame of the card slot area, and intercepts a card slot fine position sub-image in the accurate detection frame of the card slot area; inputs the card slot fine position sub-image into the binary classification model completed training to obtain a classification result, and judges whether the card slot has defects.

[0091] The embodiment of the application further provides a card slot defect image detection device based on deep learning, comprising:

[0092] An image acquisition device is used to acquire a product original image with a card slot photographed by a camera.

[0093] A host computer is in communication connection with the image acquisition device, and is used to execute a computer program to realize the steps of the card slot defect image detection method based on deep learning.

[0094] Those skilled in the art should understand that the embodiments of the application can be provided as a method, device or computer program product. Therefore, the application can adopt a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the application can adopt a computer program product in the form of one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.

[0095] The application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a machine that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device that realizes the function specified in one block or multiple blocks.

[0096] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0098] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Based on the above description, one of ordinary skill in the art can make other various changes or modifications to the present application. Here, it is not necessary or possible to exhaust all the embodiments. The obvious changes or modifications derived therefrom are still within the scope of the present application.

Claims

1.A deep learning-based card slot defect image detection method, characterized by, The method comprises the following steps: Taking the original image of the product with the card slot as the original image dataset by using the camera; Labeling the part where the card slot is located on the original image dataset, and extracting the card slot part sub-image set; Scaling all the images of the original image dataset and the card slot part sub-image set to the same preset resolution size to obtain a mixed image dataset and its label; Training and testing a target detection model using the mixed image dataset; Refining the accurate position of the card slot on the card slot part sub-image set to obtain a card slot fine position sub-image; Labeling the defect position in the card slot fine position sub-image to make a card slot defect classification dataset, which is divided into two categories: defective and non-defective; Training and testing a binary classification model using the card slot defect classification dataset; Obtaining the image to be detected by the camera, adjusting the resolution of the image to be detected to the preset resolution size, inputting it into the trained target detection model to obtain the detection frame of the card slot area, cutting the card slot part sub-image in the detection frame of the card slot area, adjusting the resolution of the card slot part sub-image to the preset resolution size again, inputting the card slot part sub-image into the trained target detection model to obtain the accurate detection frame of the card slot area, cutting the card slot fine position sub-image in the accurate detection frame of the card slot area, and inputting the card slot fine position sub-image into the trained binary classification model to obtain the classification result and determine whether the card slot has defects. 2.The deep learning-based slot defect image detection method of claim 1, wherein, The original image of the product with the card slot is more than 50 million pixels. 3.The deep learning-based slot defect image detection method of claim 1, wherein, The labeled information of the part where the card slot is located includes the coordinates and category of the part where the card slot is located, and is saved as a txt file. 4.The deep learning-based slot defect image detection method of claim 1, wherein, Scaling all the images of the original image dataset and the card slot part sub-image set to the same preset resolution size comprises: Setting the image resolution size of the original image dataset to the image resolution size of the card slot part sub-image set. 5.The deep learning-based image detection method of a card slot defect according to claim 1, wherein, Based on part of the images without defects in the card slot defect classification dataset, image distortion, damage, and modification are used to expand the number of defective images, so as to balance the number of images in the two categories in the card slot defect classification dataset, and obtain a target card slot defect classification dataset. 6.The deep learning-based image detection method of a card slot defect according to claim 1, wherein, The target detection model is based on the YOLO target detection framework, part of the samples of the mixed image dataset are selected as a batch input into the target detection model for training until the network converges, and the model parameters with the smallest loss are retained. 7.The deep learning-based slot defect image detection method of claim 6, wherein, The target detection model is a one-stage regression model, the model size is 50M, and the model output result includes target positioning boundary, category, and confidence. 8.The deep learning-based slot defect image detection method of claim 1, wherein, The binary classification model has a model size of 10M, and the model output result is greater than or equal to 0.5, which is determined as a defective card slot, and the model output result is less than 0.5, which is determined as a non-defective card slot. 9.A deep learning-based card slot defect image detection device, characterized by, The method comprises the following steps: An original image dataset acquisition module is used to take the original image of the product with the card slot as the original image dataset by using the camera; A card slot part sub-image set acquisition module is used to label the part where the card slot is located on the original image dataset, and extract the card slot part sub-image set; The mixed image dataset acquisition module: scales all the images of the original image dataset and the images of the card slot part sub-dataset to the same preset resolution size, to obtain a mixed image dataset and its label; The target detection model training module: trains and tests a target detection model using the mixed image dataset; The card slot fine position sub-image acquisition module: finely labels the accurate position of the card slot on the card slot part sub-dataset, to obtain a card slot fine position sub-image; The card slot defect classification dataset acquisition module: labels the defect position in the card slot fine position sub-image, to obtain a card slot defect classification dataset, which is classified into two categories: defective and non-defective; The binary classification model training module: trains and tests a binary classification model using the card slot defect classification dataset; The defect detection module: acquires a to-be-detected image captured by a camera, adjusts the resolution of the to-be-detected image to a preset resolution size, inputs the to-be-detected image into the trained target detection model to obtain a card slot region bounding box, intercepts a card slot part sub-image in the card slot region bounding box, adjusts the resolution of the card slot part sub-image to the preset resolution size again, inputs the card slot part sub-image into the trained target detection model to obtain a card slot region accurate bounding box, intercepts a card slot fine position sub-image in the card slot region accurate bounding box, and inputs the card slot fine position sub-image into the trained binary classification model to obtain a classification result, to determine whether the card slot has a defect. 10.A deep learning-based card slot defect image detection device, characterized by, The image acquisition device is configured to acquire an original image of a product with a card slot captured by a camera; The host computer is in communication connection with the image acquisition device, and is configured to execute a computer program to implement the steps of the card slot defect image detection method based on deep learning according to any one of claims 1 to 8. ​

Images