Deep learning-based cell edge defect detection method and system

Abstract

The application provides a CELL edge defect detection method and system based on deep learning, and relates to the technical field of liquid crystal display screen defect detection. A plurality of to-be-detected images are imported; an original lightweight segmentation model DDRNet is changed into a revised segmentation model Rep-DDRNet with a single branch structure, feature detection is performed on the to-be-detected images, and suspected crack areas of the to-be-detected images are determined; a plurality of single-branch structure revised segmentation models DDRNet are called in parallel to perform GPU parallel operation on the plurality of to-be-detected images, and a plurality of suspected crack areas are obtained; the plurality of suspected crack areas are classified one by one through an image classification model Rep-VGG, and interference type and real crack type judgment is performed; crack feature extraction is performed on the real crack image through machine vision Blob analysis, and defect grading is performed one by one, and images greater than a set level are alarmed. The deep learning segmentation model is optimized, the F1-score accuracy of the test data set is improved by 6%, and the overall operation time is saved by more than 1 / 3 through GPU operation according to a parallel architecture.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of liquid crystal display screen defect detection, and particularly relates to a CELL edge defect detection method and system based on deep learning. BACKGROUND

[0002] Liquid crystal display has been developed for more than 20 years as a widely used electronic product. With the rapid development of the liquid crystal display industry and the increasingly high requirements for display screen quality, appearance defect detection equipment has become an essential part of the production process in the industry. Especially in the CELL process stage, mechanical cutting and grinding operations are easy to cause defects to the edge part of the product. In the entire liquid crystal production process, the CELL section needs to detect as many defective screens as possible to avoid defective products flowing into the downstream. Therefore, the reliability and efficiency of the detection equipment are highly required.

[0003] At present, the defect detection method based on machine vision has been widely applied in various industries. The traditional machine vision-based method usually uses image processing algorithms or artificially designed defect features plus classifiers to detect and identify appearance defects. However, these traditional algorithms usually have a targeted nature, require tedious and diverse image preprocessing steps and artificially designed feature extraction methods, and also require professional algorithm developers to develop. In addition, the algorithm iteration speed is slow, the universality is poor, and it is not conducive to rapid production and model change.

[0004] In the current industrial 4.0 era, higher requirements are put forward for flexible production, which needs to adapt to the production of mixed products of multiple sizes and varieties. Since the machine vision algorithm needs to extract specific features for each product, it cannot effectively adapt to rapid production and model change. In recent years, the defect detection method based on deep learning has been widely applied in industrial scenarios. Compared with traditional algorithms, the deep learning-based algorithm has many advantages, such as low requirement for image acquisition conditions, ability to learn universal features in large data and automatically update model parameters, avoidance of complex algorithm processes designed by artificial, reduction of development difficulty, and high robustness and precision. For new products, the deep learning-based detection method also has excellent detection effect. Therefore, the defect detection method based on deep learning has become a research hotspot in the current technology field, and is expected to further improve the defect detection method in industrial production. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a detection method based on deep learning, which mainly detects defects such as breakage, cracks, scratches, shells and burrs on the edge of a liquid crystal panel.

[0006] The deep learning algorithm can automatically extract features from data, avoiding the cumbersome process of a large amount of manual feature engineering required by traditional algorithms. Deep learning algorithms can adapt to various complex defect forms, and the more deep learning, the more new approximate defects can be detected, so it has strong generalization ability. This makes the deep learning algorithm have strong adaptability and scalability for defect detection tasks for multiple products and application scenarios. Deep learning algorithms have transfer learning capabilities, and when there are over-detection or under-detection, image labeling can be performed by on-site engineers, which is easy to maintain.

[0007] But because when detecting edge defects, the camera imaging data is large, the application innovates in operation efficiency and accuracy, and uses parallel processing architecture to achieve good detection effect.

[0008] The technical solution of the application to solve the above technical problems is as follows:

[0009] The CELL edge defect detection method based on deep learning comprises the following steps:

[0010] Import multiple images to be detected;

[0011] Change the residual connection structure in the original lightweight segmentation model DDRNet to Rep structure to obtain a modified segmentation model Rep-DDRNet with multiple branch structures, and train the modified segmentation model Rep-DDRNet with multiple branch structures;

[0012] The trained modified segmentation model Rep-DDRNet with multiple branch structures is re-parameterized to a modified segmentation model Rep-DDRNet with single branch structure, and the modified segmentation model Rep-DDRNet with single branch structure is obtained. The feature detection of the modified segmentation model Rep-DDRNet with single branch structure is performed on the images to be detected, and the suspected crack area of the images to be detected is determined by the detected resolution features;

[0013] A plurality of the modified segmentation model Rep-DDRNet with single branch structure is called in parallel to perform GPU parallel operation on a plurality of the images to be detected, and a plurality of suspected crack areas corresponding to the images to be detected is obtained;

[0014] The image classification model Rep-VGG is used to classify a plurality of suspected crack areas corresponding to the images to be detected one by one, and a plurality of suspected crack area types corresponding to the suspected crack areas is obtained. The interference type and the real crack type of a plurality of suspected crack area types are judged one by one, and a plurality of real crack images is obtained.

[0015] The crack feature extraction is performed on a plurality of real crack images through machine vision Blob analysis, and the crack features of the plurality of real crack images are graded one by one, and the crack defect greater than a preset crack defect grade is alarmed.

[0016] The interference types are dust, hair, stains and the like, and the real crack types are breakage, cracks, scratches, shells and burrs at edges of liquid crystal panels.

[0017] The extracted crack features are length, width, area and contrast of real crack defects.

[0018] The beneficial effects of the present application are: through deep learning to detect CELL edge defects, referring to the Rep structure in the image classification model Rep-VGG, innovating the original lightweight segmentation model DDRNet, compared with the original model, the operation time is saved by 10%, the F1-score accuracy of the test data set is improved by 6%; through parallel architecture to simultaneously process 2 branch images for GPU operation, the overall operation time is saved by more than 1 / 3; the deep learning segmentation model is faster and more accurate, so as to reduce the over-detection rate of defect detection, and the time and accuracy of liquid crystal screen edge detection are improved.

[0019] Further, the residual connection structure in the original lightweight segmentation model DDRNet is changed to the Rep structure, and the multi-branch structure of the modified segmentation model Rep-DDRNet is re-parameterized to change the modified segmentation model Rep-DDRNet into a single-branch structure.

[0020] The standard residual convolution block RB in the original lightweight segmentation model DDRNet is changed to a multi-branch structure, 1*1 convolution and residual branch structure are added, and the re-parameterized convolution block Rep is changed.

[0021] The re-parameterized convolution block Rep in the multi-branch structure of the modified segmentation model Rep-DDRNet is re-parameterized by 1*1 convolution and residual branch, and the multi-branch structure of the modified segmentation model Rep-DDRNet is changed to a single-branch structure.

[0022] The re-parameterization is to modify the standard residual convolution block RB in the modified segmentation model DDRNet in the same structural re-parameterization as the image classification model Rep-VGG.

[0023] The beneficial effect of adopting the further scheme is that the single-branch structure modified segmentation model Rep-DDRNett is a lightweight semantic segmentation model based on a convolutional neural network, and compared with a traditional deep learning model, the single-branch structure modified segmentation model Rep-DDRNet adopts a mixed resolution strategy.

[0024] Further, the feature detection on the to-be-detected image by the single-branch structure modified segmentation model Rep-DDRNet determines the suspected crack region of the to-be-detected image through the detected resolution features, and specifically comprises:

[0025] The feature map of the to-be-detected image is subjected to multiple times of down-sampling feature extraction to extract a detail feature map of the to-be-detected image;

[0026] The feature map of the to-be-detected image is subjected to multiple times of down-sampling feature extraction to extract a semantic feature map of the to-be-detected image;

[0027] The detail feature map and the semantic feature map of the to-be-detected image are subjected to feature fusion to obtain a cascaded feature map;

[0028] The cascaded feature map is subjected to up-sampling feature recovery to obtain the suspected crack region in the to-be-detected image.

[0029] The down-sampling and the up-sampling are prior art; the down-sampling feature extraction is to reduce the feature resolution while extracting the feature, so that the extracted feature can represent a large-resolution feature, while reducing the calculation amount and making the feature have more global information.

[0030] The up-sampling feature extraction is to enlarge the feature resolution while extracting the feature, realizing the mapping of the feature from a small resolution to a large resolution.

[0031] The beneficial effect of adopting the further scheme is that the detail feature extraction, semantic feature extraction, feature fusion and semantic segmentation on the image process the features from high resolution to low resolution, which can greatly reduce the calculation complexity while maintaining high resolution, thereby achieving optimal effect on multiple image segmentation data sets.

[0032] Further, the single-branch structure modified segmentation model Rep-DDRNet comprises a detail feature branch component, the detail feature branch component comprises a standard convolution block CONV, a reparameterization convolution block Rep and a single residual convolution block RBB; the multiple times of down-sampling feature extraction on the feature map of the to-be-detected image to extract the detail feature map of the to-be-detected image specifically comprises:

[0033] The standard convolution block CONV is used to perform down-sampling feature extraction on the image to be detected to obtain a 1 / 2 resolution feature map; the re-parameterized convolution block Rep is used to perform down-sampling feature extraction on the 1 / 2 resolution feature map to obtain a 1 / 4 resolution feature map; the re-parameterized convolution block Rep is used to perform down-sampling feature extraction on the 1 / 4 resolution feature map to obtain a first 1 / 8 resolution feature map; the re-parameterized convolution block Rep is used to perform feature extraction on the first 1 / 8 resolution feature map to obtain a second 1 / 8 resolution feature map; the re-parameterized convolution block Rep is used to perform feature extraction again on the second 1 / 8 resolution feature map to obtain a third 1 / 8 resolution feature map; the single residual convolution block RBB is used to perform feature extraction on the third 1 / 8 resolution feature map to obtain a detail feature map of the image to be detected.

[0034] The beneficial effect of the above further scheme is that multiple detail feature branch extractions are sequentially performed on the image from high resolution to low resolution, which realizes high precision while reducing computational complexity.

[0035] Further, the modified segmentation model Rep-DDRNet of the single-branch structure includes a semantic feature branch component, the semantic feature branch component includes a re-parameterized convolution block Rep and a single residual convolution block RBB; the feature map of the image to be detected is subjected to multiple down-sampling feature extractions to extract a semantic feature map of the image to be detected, specifically:

[0036] After obtaining the first 1 / 8 resolution feature map, the re-parameterized convolution block Rep is used to perform down-sampling feature extraction on the first 1 / 8 resolution feature map to obtain a 1 / 16 resolution feature map; the re-parameterized convolution block Rep is used to perform down-sampling feature extraction on the 1 / 16 resolution feature map to obtain a 1 / 32 resolution feature map; the single residual convolution block RBB is used to perform down-sampling feature extraction on the 1 / 32 resolution feature map to obtain a 1 / 64 resolution feature map, and further obtain a semantic feature map of the image to be detected.

[0037] The beneficial effect of the above further scheme is that multiple semantic feature branch extractions are sequentially performed on the image from high resolution to low resolution, which realizes high precision while reducing computational complexity.

[0038] Further, the modified segmentation model Rep-DDRNet of the single-branch structure includes a feature fusion component, the feature fusion component includes a feature fusion layer; the detail feature map and the semantic feature map of the image to be detected are subjected to feature fusion to obtain a cascaded feature map, specifically:

[0039] The detail feature map and the semantic feature map are pyramid-pooling fused through the feature fusion layer to obtain the cascade feature map.

[0040] The pyramid-pooling fusion is prior art, and a pyramid-pooling feature fusion method uses feature maps of different scales for pyramid-pooling and cascades the results together to realize perception of global context information and fine semantic segmentation results.

[0041] The beneficial effect of the above further scheme is that the fusion of the detail feature map and the semantic feature map obtains the cascade feature map with rich information, which facilitates the subsequent semantic segmentation layer to obtain the crack region.

[0042] Further, the modified segmentation model Rep-DDRNet of the single-branch structure includes a semantic segmentation component, and the semantic segmentation component includes a semantic segmentation layer; the cascade feature map is up-sampled for feature recovery to obtain a suspected crack region in the to-be-detected image, and specifically:

[0043] The pixel points of the cascade feature map are classified by the semantic segmentation layer, and the cascade feature map after multiple down-sampling is up-sampled for feature recovery to obtain a semantic segmentation result with the same size as the original input to-be-detected image, and the semantic segmentation result is the suspected crack region.

[0044] The beneficial effect of the above further scheme is that the image after resolution change is restored to the same semantic segmentation result as the original image, which facilitates the subsequent judgment of the crack level.

[0045] Further, the GPU parallel operation is performed on the plurality of to-be-detected images by parallel calling of a plurality of the modified segmentation model Rep-DDRNet of the single-branch structure, and specifically:

[0046] When Flask is initialized, a plurality of independent processes are created, and one independent process initializes one single modified segmentation model Rep-DDRNet of the single-branch structure, and the plurality of independent processes one-to-one define a plurality of corresponding independent information queues; when the plurality of to-be-detected images enter the message queue, the plurality of modified segmentation models Rep-DDRNet of the single-branch structure are called in parallel by the plurality of independent processes, and GPU operation is performed on the plurality of corresponding independent information queues.

[0047] The beneficial effect of the above further scheme is that the use of multiple processes and message queues improves the calculation speed, reduces the request response time of the service, and thus improves the calculation efficiency; the parallel processing quantity of the GPU parallel operation architecture mainly depends on the GPU memory occupied by the deep learning model and the GPU memory size, and more processes can be started to achieve faster speed.

[0048] Another technical solution of the present application to solve the above technical problems is as follows:

[0049] The CELL edge defect detection system based on deep learning comprises a deep learning segmentation module, a GPU parallel operation architecture, a deep learning classification module and a machine vision feature extraction module.

[0050] A plurality of images to be detected are imported.

[0051] The deep learning segmentation module is used to change the residual connection structure in the original lightweight segmentation model DDRNet into Rep structure, to obtain a modified segmentation model Rep-DDRNet with a multi-branch structure, and to train the modified segmentation model Rep-DDRNet with a multi-branch structure.

[0052] The GPU parallel operation architecture is used to perform GPU parallel operation on a plurality of images to be detected by parallel calling a plurality of modified segmentation models Rep-DDRNet with a single-branch structure, to obtain suspected crack regions corresponding to the plurality of images to be detected.

[0053] The deep learning classification module is used to classify the suspected crack regions corresponding to the plurality of images to be detected one by one through an image classification model Rep-VGG, to obtain suspected crack region types corresponding to the plurality of suspected crack regions, and to judge the interference types and the real crack types of the plurality of suspected crack region types one by one, to obtain a plurality of real crack images.

[0054] The machine vision feature extraction module is used to extract crack features of the plurality of real crack images through machine vision Blob analysis, and to grade the defects one by one according to the crack features of the plurality of real crack images, and to alarm when the crack defect grade is greater than a pre-set crack defect grade.

[0055] The beneficial effects of the present application are: detecting CELL edge defects through deep learning, referring to the Rep structure in the image classification model Rep-VGG, innovating the original lightweight segmentation model DDRNet, compared with the original model, the operation time is saved by 10%, and the F1-score accuracy of the test data set is improved by 6%; through parallel architecture, GPU operation is performed on 2 branch images at the same time, and the overall operation time is saved by more than 1 / 3; the deep learning segmentation model is faster and more accurate, so as to reduce the over-checking rate of defect detection, and improve the time and accuracy of liquid crystal screen edge detection.

[0056] Further, the deep learning segmentation module comprises a detail feature branch component, a semantic feature branch component, a feature fusion component and a semantic segmentation component, specifically:

[0057] The detail feature branch component is used for performing multiple times of down-sampling feature extraction on the feature map of the to-be-detected image, and extracting a detail feature map of the to-be-detected image;

[0058] The semantic feature branch component is used for performing multiple times of down-sampling feature extraction on the feature map of the to-be-detected image, and extracting a semantic feature map of the to-be-detected image;

[0059] The feature fusion component is used for performing feature fusion on the detail feature map and the semantic feature map of the to-be-detected image, and obtaining a cascaded feature map;

[0060] The semantic segmentation component is used for performing up-sampling feature recovery on the cascaded feature map, and obtaining a suspected crack area in the to-be-detected image.

[0061] The beneficial effects of the above further scheme are: the image is subjected to detail feature extraction, semantic feature extraction, feature fusion and semantic segmentation, and the features from high resolution to low resolution are processed, which can greatly reduce the calculation complexity while maintaining high resolution, thereby achieving optimal effect on multiple image segmentation data sets. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The flowchart of the CELL edge defect detection method based on deep learning provided by the embodiment of the present application is shown in the figure;

[0063] Figure 2 The module block diagram of the CELL edge defect detection system based on deep learning provided by the embodiment of the present application is shown in the figure;

[0064] Figure 3 The module block diagram of the modified segmentation model Rep-DDRNet provided by the embodiment of the present application is shown in the figure;

[0065] Figure 4A structural diagram of an original lightweight segmentation model DDRNet provided for an embodiment of the present application is shown in the figure;

[0066] Figure 5 A structural diagram of a modified segmentation model Rep-DDRNet provided for an embodiment of the present application is shown in the figure;

[0067] Figure 6 A structural diagram of a standard residual convolution operation block provided for an embodiment of the present application is shown in the figure;

[0068] Figure 7 A structural diagram of a modified training Rep operation block provided for an embodiment of the present application is shown in the figure;

[0069] Figure 8 A structural diagram of an inference Rep operation block after reparameterization of the training Rep operation block provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0070] The principles and features of the present application are described below in conjunction with the accompanying drawings, and the examples are only used to explain the present application and are not intended to limit the scope of the present application.

[0071] As shown in the figure, the CELL edge defect detection method based on deep learning includes the following steps: Figure 1

[0072] Import a plurality of images to be detected;

[0073] Change the residual connection structure in the original lightweight segmentation model DDRNet into a Rep structure to obtain a modified segmentation model Rep-DDRNet with a multi-branch structure, and train the modified segmentation model Rep-DDRNet with a multi-branch structure;

[0074] Change the trained modified segmentation model Rep-DDRNet with a multi-branch structure into a modified segmentation model Rep-DDRNet with a single-branch structure through structure reparameterization to obtain a modified segmentation model Rep-DDRNet with a single-branch structure; perform feature detection on the images to be detected through the modified segmentation model Rep-DDRNet with a single-branch structure, and determine the suspected crack area of the images to be detected through the detected resolution features;

[0075] Perform GPU parallel operation on a plurality of images to be detected through parallel calling of a plurality of modified segmentation models Rep-DDRNet with a single-branch structure to obtain suspected crack areas corresponding to the plurality of images to be detected;

[0076] ​The suspected crack region types corresponding to the plurality of suspected crack regions are obtained by classifying the plurality of suspected crack regions corresponding to the plurality of to-be-detected images one by one through an image classification model Rep-VGG, and the interference types and the real crack types of the plurality of suspected crack region types are judged one by one to obtain a plurality of real crack images.

[0077] The crack features of the plurality of real crack images are extracted through machine vision Blob analysis, and the defect classification is performed one by one according to the crack features of the plurality of real crack images, and an alarm is given to the crack defect level greater than a pre-set crack defect level.

[0078] The scheme detects the CELL edge defect through deep learning, refers to the Rep structure in the image classification model Rep-VGG, innovates the original lightweight segmentation model DDRNet, saves 10% of the operation time compared with the original model, and improves the F1-score accuracy of the test data set by 6%; the GPU operation is performed through the parallel architecture to process two branch images at the same time, and the overall operation time is saved by more than 1 / 3; the deep learning segmentation model is faster and more accurate, so as to reduce the over-checking rate of defect detection, and to improve the time and accuracy of liquid crystal screen edge detection.

[0079] Preferably, the residual connection structure in the original lightweight segmentation model DDRNet is replaced with the Rep structure, and the multi-branch structure of the modified segmentation model Rep-DDRNet is replaced with the single-branch structure of the modified segmentation model Rep-DDRNet, specifically:

[0080] The standard residual convolution block RB (i.e., the standard residual convolution operation block, as shown in Figure 6 The standard residual convolution block RB (i.e., the standard residual convolution operation block, as shown in Figure 7 The standard residual convolution block RB (i.e., the standard residual convolution operation block, as shown in

[0081] The residual connection structure in the original lightweight segmentation model DDRNet is replaced with the Rep structure, and the multi-branch structure of the modified segmentation model Rep-DDRNet is replaced with the single-branch structure of the modified segmentation model Rep-DDRNet, specifically: Figure 8

[0082] Preferably, the feature of the to-be-detected image is detected through the single-branch structure of the modified segmentation model Rep-DDRNet, and the suspected crack region of the to-be-detected image is determined through the detected resolution feature, specifically: ​

[0083] Multiple downsampling feature extractions are performed on the feature map of the image to be detected to extract the detailed feature map of the image to be detected;

[0084] The semantic feature map of the image to be detected is extracted by performing multiple downsampling feature extractions on the feature map of the image to be detected.

[0085] The detail feature map and the semantic feature map of the image to be detected are fused to obtain a cascaded feature map;

[0086] Upsampling feature recovery is performed on the cascaded feature map to obtain the suspected crack area in the image to be detected.

[0087] In the above embodiments, the modified segmentation model DDRNet with a single-branch structure is a lightweight semantic segmentation model based on convolutional neural networks. Compared with traditional deep learning models, the modified segmentation model DDRNet with a single-branch structure adopts a mixed resolution strategy to perform detail feature extraction, semantic feature extraction, feature fusion, and semantic segmentation on the image. At the same time, it processes features from high resolution to low resolution, which can greatly reduce computational complexity while maintaining high resolution, thereby achieving optimal results on multiple image segmentation datasets.

[0088] like Figure 5 As shown, CONV is a standard convolutional block, Rep is a reparameterized convolutional block, and RBB is a single residual convolutional layer; detail feature branches for extracting high-resolution features from the image are obtained from (1) to (6).

[0089] Preferably, the modified segmentation model Rep-DDRNet with a single-branch structure includes a detail feature branch component, which includes a standard convolutional block CONV, a reparameterized convolutional block Rep, and a single residual convolutional block RBB; the step of performing multiple downsampling feature extractions on the feature map of the image to be detected to extract the detail feature map of the image to be detected specifically involves:

[0090] The standard convolution block CONV(1) is used to perform down-sampling feature extraction on the image to be detected to obtain a 1 / 2 resolution feature map; the re-parameterized convolution block Rep(2) is used to perform down-sampling feature extraction on the 1 / 2 resolution feature map to obtain a 1 / 4 resolution feature map; the re-parameterized convolution block Rep(3) is used to perform down-sampling feature extraction on the 1 / 4 resolution feature map to obtain a first 1 / 8 resolution feature map; the re-parameterized convolution block Rep(4) is used to perform feature extraction on the first 1 / 8 resolution feature map to obtain a second 1 / 8 resolution feature map; the re-parameterized convolution block Rep(5) is used to perform feature extraction on the second 1 / 8 resolution feature map again to obtain a third 1 / 8 resolution feature map; and the single residual convolution block RBB(6) is used to perform feature extraction on the third 1 / 8 resolution feature map to obtain a detail feature map of the image to be detected.

[0091] From (7) to (9), a semantic feature branch for extracting semantic information of a picture is provided.

[0092] Preferably, the modified segmentation model Rep-DDRNet of the single-branch structure comprises a semantic feature branch component, the semantic feature branch component comprising a re-parameterized convolution block Rep and a single residual convolution block RBB; and the feature map of the image to be detected is subjected to multiple down-sampling feature extractions to extract a semantic feature map of the image to be detected, specifically as follows:

[0093] After obtaining the first 1 / 8 resolution feature map, the re-parameterized convolution block Rep(7) is used to perform down-sampling feature extraction on the first 1 / 8 resolution feature map to obtain a 1 / 16 resolution feature map; the re-parameterized convolution block Rep(8) is used to perform down-sampling feature extraction on the 1 / 16 resolution feature map to obtain a 1 / 32 resolution feature map; and the single residual convolution block RBB(9) is used to perform down-sampling feature extraction on the 1 / 32 resolution feature map to obtain a 1 / 64 resolution feature map, thereby obtaining a semantic feature map of the image to be detected.

[0094] Preferably, the modified segmentation model Rep-DDRNet of the single-branch structure comprises a feature fusion component, the feature fusion component comprising a feature fusion layer; and the detail feature map and the semantic feature map of the image to be detected are subjected to feature fusion to obtain a cascaded feature map, specifically as follows:

[0095] The feature fusion layer (10) is used to perform pyramid pooling fusion on the detail feature map and the semantic feature map to obtain the cascaded feature map.

[0096] Preferably, the modified segmentation model Rep-DDRNet of the single-branch structure comprises a semantic segmentation component, the semantic segmentation component comprising a semantic segmentation layer; the cascade feature map is up-sampled for feature recovery to obtain a suspected crack area in the image to be detected, specifically:

[0097] The pixel points of the cascade feature map are classified by the semantic segmentation layer (11), and the cascade feature map after multiple down-sampling is recovered by up-sampling to obtain a semantic segmentation result with the same size as the original input image to be detected, and the semantic segmentation result is the suspected crack area.

[0098] In the above embodiment, the image is sequentially extracted multiple times for detail feature branch extraction and multiple times for semantic feature branch extraction from high resolution to low resolution, achieving high precision while reducing computational complexity; the detail feature map and the semantic feature map are fused to obtain a cascade feature map with rich information; the image after resolution change is restored to the same semantic segmentation result as the original image, facilitating subsequent crack level judgment.

[0099] Preferably, the GPU parallel operation of the plurality of images to be detected by the plurality of modified segmentation models Rep-DDRNet of the single-branch structure is specifically:

[0100] When Flask is initialized, a plurality of independent processes are created, and one independent process initializes one single modified segmentation model Rep-DDRNet of the single-branch structure, and a plurality of independent processes define a plurality of corresponding independent information queues; when a plurality of images to be detected enter the message queue, a plurality of corresponding modified segmentation models Rep-DDRNet of the single-branch structure are called in parallel by the plurality of independent processes, and GPU operation is performed on the plurality of corresponding independent information queues.

[0101] Specifically, the traditional GPU operation architecture will cause threads to be unable to fully parallelize concurrent tasks due to the global interpreter lock (GIL) of Python, and the operation time axis is illustrated in Table 1.

[0102]

[0103] Table 1-Traditional GPU image processing table

[0104] There are 4 images to be detected in Table 1, and there are three steps in deep learning, namely GPU pre-processing, GPU operation and GPU post-processing. It is assumed that each step takes 10ms. When image 1 starts operation, other images can only wait for image 1 to complete GPU pre-processing before starting GPU pre-processing of image 2 due to GIL lock. When image 4 starts GPU pre-processing, it has actually waited for 30ms to start operation. The concurrent operation architecture is actually a pipeline architecture.

[0105] In view of this complex situation, multiple processes and message queues are used to improve the calculation speed and reduce the request response time of the service. Using multiple processes can divide large computing tasks into multiple independent processes, thereby improving the computing efficiency. The message queue is a mechanism for asynchronous communication, which can realize the coordinated work between processes. Therefore, by using the advantages of multiple processes and message queues, the architecture can run multiple independent deep learning models at the same time. At the same time, it can also handle multiple connections by itself through Flask, and realize multiple requests and responses without affecting each other.

[0106] The GPU parallel operation architecture is based on the multi-thread triggering of Flask, so that the architecture can perform concurrent processing every time it receives an unordered call from the outside. When Flask is initialized, multiple independent processes are created, and each of the multiple independent processes initializes a single branch structure of the modified segmentation model DDRNet, and then the multiple independent processes define multiple corresponding independent information queues one by one; When the information queue has data, the process will automatically call the single branch structure of the modified segmentation model DDRNet for operation. When the external operation is called 4 times at the same time, the single image data is put into the information queue 1, and the double image data is put into the information queue 2, and the pipeline operation is performed inside the process corresponding to each information queue to achieve parallel processing of 2 images. As shown in Table 2, 20ms of processing time can be saved. The number of parallel processing of the architecture mainly depends on the GPU memory occupied by the deep learning model and the size of the GPU memory. More processes can be started to achieve faster speed.

[0107] Image 2 GPU pre-processing GPU operation GPU post-processing Image 3 Wait GPU pre-processing GPU operation GPU post-processing Image 4 Wait GPU pre-processing GPU operation GPU post-processing

[0108] Table 2-parallel operation GPU image processing table

[0109] As Figure 2 shown, the CELL edge defect detection system based on deep learning includes a deep learning segmentation module, a GPU parallel operation architecture, a deep learning classification module and a machine vision feature extraction module; the GPU parallel operation architecture includes multiple deep learning segmentation modules;

[0110] Import multiple images to be detected;

[0111] The deep learning segmentation module is used to change the residual connection structure in the original lightweight segmentation model DDRNet into a Rep structure, to obtain a modified segmentation model Rep-DDRNet with a multi-branch structure, and to train the modified segmentation model Rep-DDRNet with a multi-branch structure; the trained modified segmentation model Rep-DDRNet with a multi-branch structure is re-parameterized to change into a modified segmentation model Rep-DDRNet with a single-branch structure, to obtain a modified segmentation model Rep-DDRNet with a single-branch structure; the modified segmentation model Rep-DDRNet with a single-branch structure is used to perform feature detection on the to-be-detected image, and the suspected crack area of the to-be-detected image is determined according to the detected resolution features;

[0112] The GPU parallel operation architecture is used to perform GPU parallel operation on a plurality of to-be-detected images by parallel calling of a plurality of modified segmentation models Rep-DDRNet with a single-branch structure, to obtain suspected crack areas corresponding to the plurality of to-be-detected images;

[0113] The deep learning classification module is used to classify the suspected crack areas corresponding to the plurality of to-be-detected images one by one by using an image classification model Rep-VGG, to obtain suspected crack area types corresponding to the plurality of suspected crack areas, and to judge the interference types and the real crack types of the plurality of suspected crack area types one by one, to obtain a plurality of real crack images;

[0114] The machine vision feature extraction module is used to extract crack features of the plurality of real crack images by machine vision Blob analysis, and to grade the defects one by one according to the crack features of the plurality of real crack images, and to alarm when the crack defect grade is greater than a pre-set crack defect grade.

[0115] The scheme detects CELL edge defects by deep learning, refers to the Rep structure in the image classification model Rep-VGG, and innovates the original lightweight segmentation model DDRNet. Compared with the original model, the operation time is saved by 10%, and the F1-score accuracy of the test data set is improved by 6%. The GPU operation is performed by simultaneously processing 2-branch images by using a parallel architecture, and the overall operation time is saved by more than 1 / 3. The deep learning segmentation model is faster and more accurate, so as to reduce the over-detection rate of defect detection and improve the time and accuracy of liquid crystal screen edge detection.

[0116] As shown in Figure 3 The deep learning segmentation module includes a detail feature branch component, a semantic feature branch component, a feature fusion component, and a semantic segmentation component, specifically:

[0117] The detail feature branch component is used for multiple times of down-sampling feature extraction on the feature map of the to-be-detected image, and extracts a detail feature map of the to-be-detected image.

[0118] The semantic feature branch component is used for multiple times of down-sampling feature extraction on the feature map of the to-be-detected image, and extracts a semantic feature map of the to-be-detected image.

[0119] The feature fusion component is used for feature fusion on the detail feature map and the semantic feature map of the to-be-detected image, and obtains a cascaded feature map.

[0120] The semantic segmentation component is used for up-sampling feature recovery on the cascaded feature map, and obtains a suspected crack region in the to-be-detected image.

[0121] As shown in Figure 4 DDRNet is an original lightweight segmentation model, wherein image is a to-be-detected image, (1) CONV is a standard convolution block of a 1 / 2 resolution feature map, (2) RB is a standard residual convolution block of a 1 / 4 resolution feature map, (3) RB is a standard residual convolution block of a 1 / 8 resolution feature map, (4) RB is a standard residual convolution block of a 1 / 8 resolution feature map, (5) RB is a standard residual convolution block of a 1 / 8 resolution feature map, (6) RBB is a single residual convolution block of a 1 / 8 resolution feature map, (7) RB is a standard residual convolution block of a 1 / 16 resolution feature map, (8) RB is a standard residual convolution block of a 1 / 32 resolution feature map, (9) RBB is a single residual convolution block of a 1 / 64 resolution feature map, (10) is a fusion layer, (11) is a semantic segmentation layer, and output is an output real crack image.

[0122] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device.

[0123] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based CELL edge defect detection method, characterized in that, The method comprises the following steps: Importing a plurality of images to be detected; Replacing the residual connection structure in the original lightweight segmentation model DDRNet with a Rep structure to obtain a modified segmentation model Rep-DDRNet with a multi-branch structure, and training the modified segmentation model Rep-DDRNet with a multi-branch structure; Through structural reparameterization of the trained modified segmentation model Rep-DDRNet with a multi-branch structure, the modified segmentation model Rep-DDRNet with a single-branch structure is obtained, and the suspected crack regions of the images to be detected are determined through the detected resolution features by using the modified segmentation model Rep-DDRNet with a single-branch structure; A plurality of the modified segmentation models Rep-DDRNet with a single-branch structure are called in parallel to perform GPU parallel operation on a plurality of the images to be detected, and the suspected crack regions corresponding to the plurality of images to be detected are obtained; The suspected crack regions corresponding to the plurality of images to be detected are classified one by one by using the image classification model Rep-VGG, the suspected crack region types corresponding to the plurality of suspected crack regions are obtained, and the interference types and the real crack types of the plurality of suspected crack region types are judged one by one to obtain a plurality of real crack images; Through machine vision Blob analysis, crack features of the plurality of real crack images are extracted, and the plurality of real crack images are classified one by one according to the crack features of the plurality of real crack images, and an alarm is given when the crack defect level is greater than a pre-set crack defect level; The replacement of the residual connection structure in the original lightweight segmentation model DDRNet with a Rep structure and the replacement of the trained modified segmentation model Rep-DDRNet with a multi-branch structure with a modified segmentation model Rep-DDRNet with a single-branch structure are as follows: The standard residual convolution block RB in the original lightweight segmentation model DDRNet is replaced with a multi-branch structure, 1*1 convolution and residual branch structure are added, and a reparameterized convolution block Rep is obtained; The reparameterized convolution block Rep in the modified segmentation model Rep-DDRNet with a multi-branch structure is reparameterized with 1*1 convolution and residual branch, and the multi-branch structure of the modified segmentation model Rep-DDRNet is replaced with a single-branch structure; The GPU parallel operation on a plurality of the images to be detected by calling a plurality of the modified segmentation models Rep-DDRNet with a single-branch structure in parallel is as follows: In the initialization of Flask, a plurality of independent processes are created, and one independent process initializes one single branch structure of a modified segmentation model Rep-DDRNet, and the plurality of independent processes define a plurality of corresponding independent information queues; when a plurality of said to be detected images enter said information queue, said plurality of independent processes are called in parallel to correspond to a plurality of said single branch structure of a modified segmentation model Rep-DDRNet, and GPU operation is performed on a plurality of corresponding independent information queues. 2.The deep learning-based CELL edge defect detection method of claim 1, wherein, The feature detection of the to-be-detected image is performed by the single branch structure of the modified segmentation model Rep-DDRNet, and the suspected crack area of the to-be-detected image is determined by the detected resolution feature, specifically: The feature map of the to-be-detected image is subjected to multiple times of down-sampling feature extraction, and the detail feature map of the to-be-detected image is extracted; The feature map of the to-be-detected image is subjected to multiple times of down-sampling feature extraction, and the semantic feature map of the to-be-detected image is extracted; The detail feature map and the semantic feature map of the to-be-detected image are subjected to feature fusion to obtain a cascaded feature map; The cascaded feature map is subjected to up-sampling feature recovery to obtain the suspected crack area in the to-be-detected image. 3.The deep learning-based CELL edge defect detection method of claim 2, wherein, The single branch structure of the modified segmentation model Rep-DDRNet includes a detail feature branch component, and the detail feature branch component includes a standard convolution block CONV, a reparameterization convolution block Rep, and a single residual convolution block RBB; the feature map of the to-be-detected image is subjected to multiple times of down-sampling feature extraction, and the detail feature map of the to-be-detected image is extracted, specifically: The to-be-detected image is subjected to down-sampling feature extraction by the standard convolution block CONV to obtain a 1 / 2 resolution feature map; the 1 / 2 resolution feature map is subjected to down-sampling feature extraction by the reparameterization convolution block Rep to obtain a 1 / 4 resolution feature map; the 1 / 4 resolution feature map is subjected to down-sampling feature extraction by the reparameterization convolution block Rep to obtain a first 1 / 8 resolution feature map; the first 1 / 8 resolution feature map is subjected to feature extraction by the reparameterization convolution block Rep to obtain a second 1 / 8 resolution feature map; the second 1 / 8 resolution feature map is subjected to feature extraction by the reparameterization convolution block Rep again to obtain a third 1 / 8 resolution feature map; the third 1 / 8 resolution feature map is subjected to feature extraction by the single residual convolution block RBB to obtain the detail feature map of the to-be-detected image. 4.The deep learning-based CELL edge defect detection method of claim 3, wherein, The single branch structure of the modified segmentation model Rep-DDRNet includes a semantic feature branch component, and the semantic feature branch component includes a reparameterization convolution block Rep and a single residual convolution block RBB; the feature map of the to-be-detected image is subjected to multiple times of down-sampling feature extraction, and the semantic feature map of the to-be-detected image is extracted, specifically: When the first 1 / 8 resolution feature map is obtained, the first 1 / 8 resolution feature map is subjected to down-sampling feature extraction by the re-parameter convolution block Rep to obtain a 1 / 16 resolution feature map; the 1 / 16 resolution feature map is subjected to down-sampling feature extraction by the re-parameter convolution block Rep to obtain a 1 / 32 resolution feature map; the 1 / 32 resolution feature map is subjected to down-sampling feature extraction by the single residual convolution block RBB to obtain a 1 / 64 resolution feature map, and then a semantic feature map of the to-be-detected image is obtained.

5. The deep learning based CELL edge defect detection method of claim 4, wherein, The modified segmentation model Rep-DDRNet of the single-branch structure comprises a feature fusion assembly, and the feature fusion assembly comprises a feature fusion layer; the detail feature map and the semantic feature map of the to-be-detected image are subjected to feature fusion to obtain a cascaded feature map, specifically as follows: The detail feature map and the semantic feature map are subjected to pyramid pooling fusion by the feature fusion layer to obtain the cascaded feature map.

6. The deep learning-based CELL edge defect detection method according to claim 5, characterized in that, The modified segmentation model Rep-DDRNet of the single-branch structure comprises a semantic segmentation assembly, and the semantic segmentation assembly comprises a semantic segmentation layer; the cascaded feature map is subjected to up-sampling feature recovery to obtain a suspected crack region in the to-be-detected image, specifically as follows: The pixel points of the cascaded feature map are subjected to category analysis by the semantic segmentation layer, and the cascaded feature map subjected to multiple down-sampling is subjected to feature recovery by up-sampling to obtain a semantic segmentation result with the same size as the original input to-be-detected image, and the semantic segmentation result is the suspected crack region.

7. A deep learning based CELL edge defect detection system characterized in that, Comprise: a deep learning segmentation module, a GPU parallel operation architecture, a deep learning classification module and a machine vision feature extraction module; a plurality of to-be-detected images are imported; the deep learning segmentation module is used to change the residual connection structure in the original lightweight segmentation model DDRNet into a Rep structure to obtain a modified segmentation model Rep-DDRNet of a multi-branch structure, train the modified segmentation model Rep-DDRNet of the multi-branch structure, re-parameterize the trained modified segmentation model Rep-DDRNet of the multi-branch structure into a modified segmentation model Rep-DDRNet of a single-branch structure, and obtain the modified segmentation model Rep-DDRNet of the single-branch structure; the to-be-detected image is subjected to feature detection by the modified segmentation model Rep-DDRNet of the single-branch structure, and a suspected crack region of the to-be-detected image is determined according to the detected resolution feature; the GPU parallel operation architecture is used to perform GPU parallel operation on a plurality of to-be-detected images by parallel calling of a plurality of modified segmentation models Rep-DDRNet of the single-branch structure to obtain suspected crack regions corresponding to the plurality of to-be-detected images; The deep learning classification module is configured to classify the suspected crack regions corresponding to the plurality of images to be detected one by one through an image classification model Rep-VGG, obtain suspected crack region types corresponding to the plurality of suspected crack regions, and judge the interference types and the real crack types of the plurality of suspected crack region types one by one, to obtain a plurality of real crack images; The machine vision feature extraction module is configured to extract crack features of the plurality of real crack images through machine vision Blob analysis, and grade the defects one by one according to the crack features of the plurality of real crack images, and alarm when the crack defect level is greater than a preset crack defect level; The residual connection structure in the original lightweight segmentation model DDRNet is replaced by a Rep structure, and the trained multi-branch structure of the modified segmentation model Rep-DDRNet is re-parameterized to a single-branch structure of the modified segmentation model Rep-DDRNet, specifically as follows: The standard residual convolution block RB in the original lightweight segmentation model DDRNet is replaced by a multi-branch structure, 1*1 convolution and residual branch structure are added, and a re-parameterized convolution block Rep is obtained; The re-parameterized convolution block Rep in the multi-branch structure of the modified segmentation model Rep-DDRNet is re-parameterized by 1*1 convolution and residual branch, and the multi-branch structure of the modified segmentation model Rep-DDRNet is replaced by a single-branch structure; The GPU parallel operation of the plurality of images to be detected by the plurality of single-branch structure modified segmentation models Rep-DDRNet is specifically as follows: When Flask is initialized, a plurality of independent processes are created, and each independent process initializes a single single-branch structure modified segmentation model Rep-DDRNet, and a plurality of corresponding independent information queues are defined one by one; when the plurality of images to be detected enter the information queue, the plurality of single-branch structure modified segmentation models Rep-DDRNet are called in parallel by the plurality of independent processes, and GPU operation is performed on the plurality of corresponding independent information queues.

8. The deep learning based CELL edge defect detection system of claim 7, wherein, The deep learning segmentation module includes a detail feature branch component, a semantic feature branch component, a feature fusion component, and a semantic segmentation component, specifically as follows: The detail feature branch component is configured to perform multiple times of down-sampling feature extraction on the feature map of the image to be detected, to extract a detail feature map of the image to be detected; The semantic feature branch component is configured to perform multiple times of down-sampling feature extraction on the feature map of the image to be detected, to extract a semantic feature map of the image to be detected; The feature fusion component is configured to perform feature fusion on the detail feature map and the semantic feature map of the image to be detected, to obtain a cascaded feature map; The semantic segmentation component is configured to perform up-sampling feature recovery on the cascaded feature map, to obtain a suspected crack region in the image to be detected.

Images