Metal plate defect detection method based on region segmentation

Abstract

The invention belongs to the technical field of image processing, and discloses a metal plate defect detection method and device based on region segmentation. According to the method, the foreground region of the sheet metal image is extracted and segmented through region segmentation, the defect judgment of the segmented region is completed by using the classification network model, the target frames of different segmented regions are fused, and the final target frame is output by using the mapping relationship between the segmented regions and the original image. By adopting the technical scheme of the invention, the large-resolution image can be detected, the network model is simplified, the post-processing process in the target detection NMS is omitted, and the detection speed is improved.

Description

Technical Field

[0001] The invention relates to the fields of image classification, target detection and image segmentation, and in particular to a sheet metal defect detection method based on region segmentation. Background Art

[0002] As a common material for manufacturing mechanical parts, sheet metal is widely used in aerospace, automobile manufacturing, home appliance manufacturing and other industries. Due to the complexity and long-term use environment, scratches, scars, corrosion, cracks and other defects often appear on the surface of the strip, which can easily affect the quality of subsequent products. Manual surface defect detection of steel plates has problems such as slow detection speed, high cost and insufficient stability. The automatic detection technology based on vision has the advantages of fast speed, low cost and good stability, and is highly favored in automation and intelligent manufacturing.

[0003] Traditional visual inspection methods mainly include two steps: feature extraction and classification. First, local binary patterns (LBP) or histogram of oriented gradients (HOG) are used to generate abstract features and input features, and then classifiers such as support vector machines (SVM) and K-nearest neighbors (KNN) are used to complete feature classification. However, when faced with complex surfaces with many defect types, different defect sizes, and irregular defect shapes, the above-mentioned visual inspection methods have low classification accuracy, poor robustness and real-time performance, and are difficult to meet industrial requirements.

[0004] With the popularization of high-performance computers and the development of artificial intelligence, deep learning-based detection methods have achieved great success, and the detection efficiency and accuracy in various fields, including industrial detection, have been significantly improved. However, the current mainstream target detection algorithms usually use small-resolution images as detection objects, and it is difficult to directly process large-resolution images. Although some algorithms try to segment and detect large-resolution images by sliding windows, these methods usually fail to effectively distinguish between foreground and background areas, resulting in the image captured by the sliding window may contain a large amount of background or transition areas between foreground and background. This method not only wastes precious computing resources, but also increases the difficulty of detection due to the complexity of the transition area.

[0005] Based on this, the present invention proposes a sheet metal defect detection method based on region segmentation, which completes defect positioning through foreground region segmentation, image classification and target frame fusion, simplifies the network model, eliminates the NMS post-processing process in target detection, and improves the detection speed. Summary of the invention

[0006] In view of the problem that the existing technology cannot effectively detect large-resolution sheet metal images and the target detection model algorithm is complex, the present invention proposes a sheet metal defect detection method based on region segmentation, which completes the defect positioning through foreground region segmentation, image classification and target frame fusion, reduces the complexity of the model, eliminates the NMS post-processing process in target detection, and improves the detection speed.

[0007] To achieve the above purpose, a sheet metal defect detection method based on region segmentation is provided, comprising the following steps:

[0008] Obtaining the image of the steel plate to be inspected and the sample defect set;

[0009] Extract foreground area corner points and segment the foreground area;

[0010] Generate a classification training data set based on the overlapping area between the sliding window and the defect area, and input it into the classification network for training;

[0011] Use the region segmentation and classification network model to make predictions on the input sheet metal image;

[0012] Adjacent target detection frames are fused to obtain the final defect detection result.

[0013] Optionally, extracting corner points of the foreground area and segmenting the foreground area includes:

[0014] Binarize the image to be detected to obtain a foreground area mask of the image to be detected;

[0015] A random consensus algorithm is used to fit the edge data point set, and the four edge lines are calculated in turn in the obtained sets of inner and outer points, and the positions of the four vertices in the foreground area of ​​the image are calculated according to the intersection of the lines;

[0016] Based on the slope information of a certain edge line, the foreground area is rotated around the center of the image, and the foreground area is swung to complete the segmentation of the area by window sliding interception.

[0017] Optionally, generating a classification training data set according to the overlapping area of ​​the sliding window and the defect area, and inputting the data set into a classification network for training, comprises the following steps:

[0018] Extract the corner points of the foreground area and segment the foreground area through the above steps to obtain a sheet metal image containing only the foreground area;

[0019] Use the open source tool labelImg to label the collected sheet metal images and generate corresponding labeling information;

[0020] Calculate the ratio of the overlapping area of ​​each segmented window area and the marked defect area to the area of ​​the window area, and generate a classification data set by comparing it with a preset threshold;

[0021] Some high-quality samples are screened out from the generated classification data set for preliminary training of the network. The trained network is used to further screen the data set, and finally the screened data set is used to fully train the network.

[0022] Optionally, a region segmentation and classification network model is used to make predictions on the input sheet metal image, including:

[0023] The mapping relationship between the foreground area extracted by the method of extracting foreground area corner points and segmenting the foreground area and the original image of the foreground area;

[0024] The prediction results of the segmented area are input through the classification network model;

[0025] According to the prediction result and mapping relationship of the cut area, the detection result of the corresponding area in the original image is obtained.

[0026] Optionally, the adjacent target detection frames are fused to obtain a final defect detection result, including:

[0027] For every two adjacent target detection boxes in the detection results, calculate the Euclidean distance between their center points;

[0028] Determine whether the target box needs to be fused based on the Euclidean distance and the predicted category;

[0029] For the detection frames that need to be fused, a minimum enclosing rectangle containing the two detection frame areas is generated as a new detection frame based on the boundary coordinates of the two detection frames.

[0030] In addition, in order to achieve the above-mentioned purpose, the present invention also proposes a steel plate defect detection device, the steel plate defect detection device comprising:

[0031] Image acquisition module: used to acquire sheet metal images;

[0032] Region segmentation module: used to extract and segment the foreground area;

[0033] Region detection module: used to predict defect category and location;

[0034] Result fusion module: used to fuse the detection frames and output the final defect detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 is a schematic diagram of a flow chart of a first embodiment of the present invention;

[0037] Figure 2 is a flow chart of extracting corner points of a foreground area and segmenting the foreground area according to an embodiment of the present invention;

[0038] Figure 3 is a method for calculating IOW according to an embodiment of the present invention;

[0039] Figure 4 It is a structural block diagram of a sheet metal defect detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The present invention provides a sheet metal defect detection method, referring to Figure 2 , Figure 2 It is a schematic flow chart of a first embodiment of a steel plate defect detection method of the present invention.

[0043] Step S10: Acquire the image of the steel plate to be inspected and the sample defect set.

[0044] It should be noted that the execution subject of this embodiment is a sheet metal defect detection device. The sheet metal defect detection device has functions such as data processing, data communication, and program running, and its form can be an integrated controller, a control computer, or other devices with similar functions. This embodiment does not make specific restrictions on this.

[0045] The sheet metal image acquisition device is composed of a computer, a camera and a fill light source, wherein the fill light source is used to provide a uniform lighting effect for the sheet metal sample to be inspected. The computer is responsible for controlling the camera to acquire images of the sheet metal sample, and transmits the acquired defect images to the computer through a USB interface for processing and analysis.

[0046] In order to optimize the imaging effect, the sheet metal sample is placed on a dark background for shooting in this embodiment to enhance the contrast between the sheet metal target and the background, thereby improving the detection accuracy. In addition, considering that there may be reflection interference on the surface of the sheet metal sample, the acquisition device adopts a surface light source design. The surface light source can effectively disperse light, reduce reflection interference, and ensure the uniformity and clarity of the sheet metal image.

[0047] Step S20: Extract the foreground area corner points and segment the foreground area. Figure 2 This is the processing flow of the foreground segmentation algorithm, and its specific scheme method is as follows:

[0048] Step S21: Foreground area mask: Input the original image, binarize the image, and obtain the foreground and background masks.

[0049] Step S22: Move the foreground area to the center of the image.

[0050] Binarize the image to obtain the foreground and background masks. Calculate the center point of the foreground area , where N is the number of foreground pixels and x and y are the pixel coordinates. , where H and W are the height and width of the image respectively. After the image is moved by Trans pixels, the foreground area moves to the center of the image.

[0051] Step S23: Use a random consistency algorithm to fit the edge of the foreground area and obtain vertex coordinates.

[0052] Extract the boundary points of the foreground area from the binary image, and rotate the boundary coordinates by bounday and θ. ,in The random consistency algorithm is used to fit the edge of the foreground area. The coordinates of the intersection of the fitted straight lines are the vertex coordinates of the foreground area. . Corresponding to the vertex coordinates of the foreground area in step S22 ;

[0053] Step S24: Straightening and segmenting the foreground area. Calculate the required rotation angle t for straightening the foreground area based on the slope information of a certain edge line, with the center of the image as the center. Rotate the steel plate to straighten the foreground area. On this basis, you can customize the size of the sliding window and the moving step length to complete the segmentation of the image containing only the foreground area. The position of the segmented area in the original input image can be obtained by the following calculation method.

[0054]

[0055] Among them, points are the coordinate points of each segmented area, center is the center coordinate of the image, and points_ori is the coordinate point of the segmented area mapped back to the original image.

[0056] Step S30: Generate a classification training data set according to the overlapping area between the sliding window and the defect area, and input it into the classification network for training.

[0057] It should be noted that, through the foreground area extraction method of S20, a data set containing only the foreground area can be obtained. Defect annotation is performed on each sample defect image in the foreground area data set to determine the defect area and defect type of each defect area in each sample defect image.

[0058] In the specific implementation, defects are annotated for each sample defect image in the sample defect set, and the collected steel plate image is marked using the open source labelimg tool. The annotation information is generated in the following way: each defect area is annotated with a rectangular block, the coordinates of the rectangular area are the coordinates of the defect area, and the defect type in the rectangle is used as the annotation type.

[0059] It should be noted that, in the foreground area dataset, the sliding window method is applied to each image, and IOW is calculated, where the overlap area of ​​the window area and the mark is divided by the area of ​​the window area. The calculation method is as follows: Figure 3 As shown in the figure, if IOW>C (where C is a constant between 0 and 1, let C=0.25), the segmented area and the marked defect area are set to the same defect type. This method significantly reduces the work of generating classification data sets, but there will still be a small number of samples in the generated data set whose marking types do not match the actual ones. In order to further improve the quality of the data set and reduce manual operations. First, a part of high-quality samples are manually screened from the data set for preliminary training of the classification network. Then the preliminary trained classification network is used to predict the remaining unprocessed data set. According to the model prediction results, the data samples are screened and classified to automatically correct potential labeling errors. Finally, the high-quality samples after model screening are combined with the initial screening samples to form the final classification training data set.

[0060] It should be noted that data enhancement is performed by random flipping, random rotation, random movement, random affine transformation, etc.

[0061] It is understandable that the features of the defects captured by the window area may be at the edge of the window area, so it is not suitable for center cropping data enhancement. When using spatial position transformation data enhancement, such as random movement, it is necessary to ensure that the transformed image still retains the features of the edge area, so the image needs to be scaled first before performing movement data enhancement.

[0062] It should be noted that in terms of the selection and training of the network model, EfficientNet V2 is used as the basic network model. This model has the advantages of high parameter efficiency and superior performance. Of course, other classification models can also be selected, and this embodiment does not limit this.

[0063] Step S40: using the region segmentation and classification network model to predict the input sheet metal image.

[0064] It should be noted that, through the method of step S20, the segmented foreground area image is obtained, and a mapping relationship between each segmented area and the original image is established. The segmented foreground area image is input into the trained network model, and the defect category of each segmented area is predicted to obtain the prediction result of each segmented area. Through the mapping relationship between the segmented image and the original image, the defect category result of each segmented area is associated back to the original image.

[0065] Step S50: adjacent object detection frames are merged.

[0066] It should be noted that after completing the regional level defect detection, the adjacent target detection frames in the detection results are fused. For every two adjacent target detection frames in the detection results, calculate the Euclidean distance d between their center points. Compare the calculated Euclidean distance d with the preset fusion threshold F. If d ≤ F, it is determined that the two detection frames point to the same defect area and the fusion operation is performed. If d>F, the fusion is not performed and the independence of the two detection frames is retained. If d≤ F and the predicted categories are different, the detection frame with a large confidence level is selected to be retained. For the detection frames that need to be fused, a minimum enclosing rectangle containing the two detection frame areas is generated as a new detection frame based on the boundary coordinates of the two detection frames. After completing the comparison and fusion of all adjacent detection frames, the final fused defect detection result is output.

[0067] In addition, an embodiment of the present invention further provides a sheet metal defect detection device based on region segmentation, which is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been described will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0068] Figure 4 : is a structural block diagram of a sheet metal defect detection device based on region segmentation according to an embodiment of the present application, the device comprising:

[0069] The image acquisition module 10 is used to acquire sheet metal images in real time.

[0070] The region segmentation module 20 is used to obtain and segment the image foreground region of the sheet metal image.

[0071] The region detection module 30 is used to determine the defect type of the segmented region.

[0072] The result fusion module 40 is used to fuse different detection frames to obtain the final prediction result.

[0073] In one embodiment, the image acquisition module 10 collects real-time images of the sheet metal surface by connecting a camera and a computer;

[0074] Use fill light source to illuminate sheet metal images to optimize image quality;

[0075] The collected sheet metal images are transmitted to the data processing device via a USB interface or other communication interface;

[0076] During the imaging process, the sheet metal is placed on a dark background to enhance the contrast between the target and the background, and a surface light source is used to reduce light reflection interference.

[0077] In one embodiment, the region segmentation module 20 extracts and segments the foreground region of the image;

[0078] Apply rotation and segmentation operations to the foreground area to obtain a corrected foreground image to support dataset generation and subsequent defect detection;

[0079] Establish a mapping relationship between the segmented image and the original image to provide support for subsequent detection modules.

[0080] In one embodiment, the region detection module 30 uses a trained classification network model to perform defect detection on the segmented image;

[0081] The segmented image is input into the network model, and the defect category of each partition is output;

[0082] According to the mapping relationship between the segmented image and the original image, the detection results of the segmented area are associated back to the original image.

[0083] In one embodiment, the result fusion module 40 fuses the target boxes in the detection results;

[0084] Calculate the Euclidean distance of the detection frames to determine whether adjacent detection frames need to be fused;

[0085] After performing fusion judgment on all detection frames, the final detection results are output, including the category, location and size information of sheet metal defects;

[0086] It should be understood that the above description is only a preferred embodiment of the present invention, and its purpose is to illustrate the technical solution of the present invention, rather than any limitation on the technical scope of the present invention. In specific applications, those skilled in the art can make appropriate adjustments and modifications to the technical solution of the present invention according to actual needs, but these adjustments and modifications should be deemed to fall within the protection scope of the present invention.

[0087] It should be further explained that, although the steps in the flowchart in the embodiment of the present application are displayed in sequence according to the order of the arrows, this does not mean that the steps must be strictly executed in this order. Unless the order restrictions are clearly specified in the specific description, the execution order of the steps shown in the flowchart can be flexibly adjusted. These steps can be performed in other orders, and some steps can be decomposed into multiple sub-steps or stages, which can be completed at the same time or at different times, and can be performed alternately or in turn with other steps or sub-steps.

[0088] In addition, the workflow described above is only an exemplary description and does not limit the protection scope of the present invention. Those skilled in the art may choose to implement some or all of the steps according to specific application requirements to achieve the technical effects of this embodiment.

[0089] Through the description of the above implementation methods, those skilled in the art can understand that the embodiments of the present invention can be implemented by software plus a necessary general hardware platform. Of course, in some cases, the embodiments of the present invention can also be implemented by pure hardware, but generally, implementation by software is a better implementation method. Based on this, the technical solution of the present invention can essentially be embodied as a software product, which can be stored in a storage medium such as a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and contains a number of instructions for enabling a terminal device (such as a computer, a server, a mobile terminal or other network device) to execute the methods in the embodiments described in the present invention.

[0090] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0091] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0092] It should be emphasized again that the specific embodiments described in this specification are only preferred embodiments of the present invention and are not intended to limit the scope of patent protection of the present invention. Any equivalent structural transformation, equivalent process transformation, or technical solutions directly or indirectly applied to other related technical fields based on the contents of the present specification and drawings shall be deemed to fall within the scope of patent protection of the present invention.

Claims

1. A sheet metal defect detection method based on region segmentation, characterized in that: include: Obtaining the image of the steel plate to be inspected and the sample defect set; Extract foreground area corner points and segment the foreground area; Generate a classification training data set based on the overlapping area between the sliding window and the defect area, and input it into the classification network for training; Use the region segmentation and classification network model to make predictions on the input sheet metal image; Adjacent target detection frames are fused to obtain the final defect detection result.

2. The method according to claim 1, characterized in that The step of extracting corner points of the foreground area and segmenting the foreground area comprises: Binarize the image to be detected to obtain a foreground area mask of the image to be detected; The straight lines of the four edges are calculated in turn from the obtained sets of inner and outer points, and the coordinates of the four vertices in the foreground area of ​​the image are calculated according to the intersection of the straight lines; Calculate the slope of a certain edge line based on the coordinate information and calculate the angle required to rotate the foreground area to correct it; The steel plate is rotated with the center of the image to align the foreground area and realize automatic segmentation of the foreground area.

3. The method according to claim 1, characterized in that The generating of a classification training data set according to the overlapping area of ​​the sliding window and the defect area and inputting the data set into the classification network for training includes: The foreground area dataset extracted by the method of claim 2 is used to obtain a sheet metal image containing only the foreground area; Use the open source tool labelImg to label the collected sheet metal images and generate corresponding labeling information; Generate a classification data set according to the overlapping area between the cut window area and the marked defect area; Some high-quality samples are screened out from the generated classification data set for preliminary training of the network. The trained network is used to further screen the data set, and finally the screened data set is used to fully train the network.

4. A sheet metal defect detection method based on region segmentation as claimed in claim 1, characterized in that: The method of using the region segmentation and classification network model to predict the input sheet metal image includes: Extracting the mapping relationship between the partition image and the original image; The prediction results of the segmented area are input through the classification network model; According to the prediction result and mapping relationship of the cut area, the detection result of the corresponding area in the original image is obtained.

5. A sheet metal defect detection method based on region segmentation as claimed in claim 1, characterized in that: The adjacent target detection frames are fused to obtain the final defect detection result, including: Calculate the Euclidean distance between two adjacent target detection boxes in the prediction results in sequence; Adjacent detection frames are fused according to the Euclidean distance and predicted category to obtain the detection results of the corresponding areas in the original image.

6. A sheet metal defect detection device based on region segmentation, characterized in that: The sheet metal defect detection device comprises: Image acquisition module: used to acquire sheet metal images; Region segmentation module: used to extract and segment the foreground area; Region detection module: used to predict defect category and location; Result fusion module: used to fuse the detection frames and output the final defect detection results.

Images