Complex part defect data labeling method, defect detection method and multi-view multi-illumination data acquisition device

Abstract

The invention belongs to the field of defect detection, and particularly discloses a complex part defect data labeling method, a defect detection method and a multi-view multi-illumination data acquisition device.In the data acquisition device, a rotary table provides a plurality of views of a part to be detected, and the design of a light source provides a plurality of illumination conditions for a camera; information acquisition of a complete surface image of the to-be-detected part is ensured; for defect data, in order to avoid manual marking of image pixels, fluorescence marking is carried out at defect positions in advance, so that clear defect positions can be obtained under the condition of fluorescence illumination in the acquisition process; and high-precision pixel-level mask information of defect positions in the image is obtained by using a three-level defect segmentation method. According to the invention, data acquisition of complex parts under multiple visual angles and illumination conditions can be realized, and defect data can be automatically labeled.

Description

Technical Field

[0001] The present invention belongs to the field of defect detection, and more specifically, relates to a method for labeling defect data of complex parts, a defect detection method, and a multi-view and multi-illumination data acquisition device. Background Art

[0002] In today's industrial production, ensuring product quality is of utmost importance. Against this backdrop, the application of industrial vision inspection technology has become particularly crucial. Industrial vision inspection technology uses an automated vision system to inspect products in order to identify defective, flawed, or non-compliant products, thereby ensuring that product quality meets standards. In recent years, anomaly detection technology has been widely applied to the surface defect quality inspection of industrial products. In industrial vision inspection, through the analysis of product surface images, anomaly detection can identify various defects such as scratches, dents, impurities, etc., thus excluding unqualified products at an early stage of production.

[0003] However, due to the problem of self-occlusion in complex parts, it is difficult to complete the detection from a single perspective. Moreover, under different perspectives and illumination conditions, the appearance and defect manifestations of products may vary greatly. Many small industrial defects can only be observed under specific illumination and perspectives. Therefore, multi-view and multi-illumination detection can more comprehensively evaluate the product state, reducing missed detections and false detections. In addition, the industrial production environment is complex and variable, and multi-angle and multi-illumination detection can adapt to different production conditions, ensuring the stability and reliability of the detection system. Currently, anomaly detection relies on the training of a large amount of data, and the amount of data in multi-view and multi-illumination data is huge, and the cost of manual annotation is extremely high. Summary of the Invention

[0004] Aiming at the above defects or improvement requirements of the prior art, the present invention provides a method for labeling defect data of complex parts, a defect detection method, and a multi-view and multi-illumination data acquisition device, aiming to achieve the automatic annotation of complex part defect data, thereby providing a large amount of data for defect detection training and improving the accuracy of defect detection.

[0005] To achieve the above object, according to the first aspect of the present invention, a method for labeling defect data of complex parts is proposed, including the following steps:

[0006] Collect images of the part to be measured from different angles, and the defect areas of the part to be measured are pre-fluorescently marked; for each angle, several visible light images under different visible light illumination conditions and one fluorescent image under fluorescent illumination conditions are collected;

[0007] Process the fluorescent image, segment the defect image from it, and use the defect image as the label of the visible light image at the corresponding angle, thereby forming a defect annotation data set and realizing defect data annotation.

[0008] As a further preference, the fluorescence image is processed to segment out the defect image, including:

[0009] Extract the region of interest ROI from the fluorescence image, then use the binarization method to segment the ROI to obtain the mask Mask_f; use image morphological operations to fine-tune the mask Mask_f to obtain the defect image.

[0010] As a further preference, extracting the region of interest ROI from the fluorescence image includes:

[0011] Use the pre-trained deep learning model SAM to perform rough segmentation on the fluorescence image to obtain the mask Mask_r, and further obtain the region of interest ROI.

[0012] As a further preference, using the binarization method to segment the ROI includes:

[0013] Calculate the mean value of all pixel values in the ROI, and use this mean value as the binarization threshold to segment the ROI to obtain the mask Mask_f.

[0014] As a further preference, the image morphological operations include dilation, erosion, opening operation and closing operation.

[0015] According to the second aspect of the present invention, there is provided a defect detection method based on the above-mentioned complex part defect data annotation method, which is characterized in that it includes the following steps:

[0016] Train the neural network with the defect annotation data set, and use the trained neural network as the defect detection model; realize the defect detection of complex parts based on this defect detection model.

[0017] According to the third aspect of the present invention, there is provided a multi-view and multi-illumination data acquisition device for implementing the above-mentioned complex part defect data annotation method, which is characterized in that it includes a turntable, a visible light component, a fluorescence component and a camera, wherein:

[0018] The turntable is used to adjust the angle of the part to be measured;

[0019] The visible light component is used to provide different visible light illumination conditions, and the fluorescence component is used to provide fluorescence illumination conditions;

[0020] The camera is used to obtain images of the part to be measured under visible light or fluorescence illumination.

[0021] As a further preference, the visible light component includes four strip-shaped light sources and one annular light source, the camera is arranged at the center of the annular light source, and the four strip-shaped light sources are arranged around the outer periphery of the annular light source.

[0022] As a further preference, it further includes an adjustment structure. The visible light component, the fluorescence component and the camera are fixedly connected to form a multi-light data collector; the adjustment structure is used to adjust the overall pose of the multi-light data collector.

[0023] As a further preference, the adjustment structure includes a horizontal adjuster, a vertical adjuster and a pitch adjuster. Among them, the horizontal adjuster and the vertical adjuster are respectively used to adjust the horizontal distance and the vertical distance between the multi-light data collector and the part to be measured, and the pitch adjuster is used to adjust the inclination angle of the multi-light data collector.

[0024] Generally speaking, compared with the prior art through the above technical solutions conceived by the present invention, the following technical advantages are mainly possessed:

[0025] 1. The present invention conducts data collection under multi-view and multi-light conditions. For defective data, fluorescence marking is performed in advance at the defective position. Therefore, under the fluorescence lighting conditions during the collection process, clear defective positions can be obtained; thus, it is possible to achieve pixel-level automatic annotation of all images with defective parts without relying on manual annotation.

[0026] 2. The present invention designs a three-level defect segmentation method, which can not only effectively locate and segment defects in images, but also ensure the accuracy and stability of the segmentation results, and obtain high-precision pixel-level mask information of the defect positions in the images; this multi-level processing strategy enables accurate identification of defects in complex and changing image environments, providing a solid foundation for subsequent image analysis and defect repair.

[0027] 3. The present invention designs a multi-view and multi-light data collection device. The turntable provides multiple views of the part to be measured, and the designed light source provides multiple lighting conditions for the camera, ensuring the acquisition of information on the complete surface image of the part to be measured, and enabling the acquisition of image data of the part to be measured at any angle and under various lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic structural diagram of the multi-view and multi-light data collection device according to an embodiment of the present invention;

[0029] Figure 2 It is a schematic structural diagram of the multi-light source according to an embodiment of the present invention;

[0030] Figure 3 It is a flowchart of part image data collection according to an embodiment of the present invention;

[0031] Figure 4 It is an image of the part under visible light and fluorescence according to an embodiment of the present invention;

[0032] Figure 5 It is a flowchart of multi-view and multi-light data collection according to an embodiment of the present invention;

[0033] Figure 6 Images of parts with multiple perspectives and multiple illuminations according to embodiments of the present invention;

[0034] Figure 7 Flow chart of defect annotation according to embodiments of the present invention;

[0035] Figure 8 Result diagram of defect segmentation according to embodiments of the present invention.

[0036] In all the drawings, the same reference numerals are used to represent the same elements or structures, where: 001 - control panel, 101 - turntable, 102 - part to be measured, 201 - horizontal adjuster, 202 - support rod, 203 - vertical adjuster, 204 - pitch adjuster, 205 - multi - illumination data collector, 301, 302, 303, 304 - strip light sources, 305 - annular light source, 401 - camera, 501 - fluorescent lamp. Detailed implementation manners

[0037] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] A method for annotating defect data of complex parts provided by an embodiment of the present invention includes the following steps:

[0039] S1. Fluorescently mark the defect areas of the parts in advance; collect images of the parts after fluorescent marking from different angles. For each angle, several visible - light images under different visible - light illumination conditions and one fluorescent image under a fluorescent light source are collected.

[0040] S2. Process each fluorescent image separately, and segment the defect image from it; use the defect image as the label for the visible - light image at the corresponding angle, so as to form a defect annotation data set and realize defect data annotation.

[0041] Further, in order to effectively identify and locate defects in the images, when processing the fluorescent images, a step - by - step and multi - level processing strategy is adopted, as Figure 7 shown, specifically including:

[0042] (1) First, focus on a preliminary rough segmentation of the defect area. The purpose of this step is to quickly locate the approximate position of the defect and lay a foundation for subsequent fine processing.

[0043] At this stage, a pre-trained deep learning model, SAM (Segment Anything Model), is used to perform a rough segmentation of the input fluorescence image I. The SAM model is trained with a large amount of image data and can segment various objects in the image, performing well even under complex backgrounds and diverse shapes:

[0044] Mask_r = SAM(I)

[0045] Where Mask_r is the rough segmentation mask predicted by the SAM model.

[0046] The obtained rough segmentation mask Mask_r is operated on to extract the region of interest (ROI):

[0047] ROI = Mask_r & I

[0048] The purpose of extracting the ROI is to reduce the computational load of subsequent processing and focus on the defect itself, thereby improving the accuracy of segmentation. The process of extracting the ROI is actually a pixel-by-pixel AND operation on the original image and the rough segmentation mask, resulting in a sub-image that only contains the defect region.

[0049] (2) After the ROI extraction is completed, a binary method is used to perform a fine segmentation of the ROI. Binarization is a method of converting an image into an image that only contains two colors, usually black and white, which can more clearly define the boundary of the defect. In the present invention, the average value of the pixels in the defect region is calculated and used as the binarization threshold Th to perform the fine segmentation:

[0050] Th = Average(ROI)

[0051] Where Average is the average value calculation function, which obtains the average value of all pixel values in the ROI.

[0052] Furthermore, the process of binarizing to obtain the fine segmentation mask Mask_f is as follows:

[0053] Mask_f ij = 1 if ROI ij > Th

[0054] Mask_f ij = 0 if ROI ij ≤ Th

[0055] Where i and j are the subscript indices of the image.

[0056] (3)Finally, to further improve the segmentation accuracy, image morphological operations are used to fine-tune the mask Mask_f obtained from the fine segmentation. Image morphological operations include dilation, erosion, opening, and closing operations, etc. They are a series of transformations based on set theory and are used to change the shape of objects in the image. The purpose of these operations is to remove noise, smooth edges, and fill small holes in the defective areas, so as to obtain a more accurate segmentation result. The morphological operation can be expressed as:

[0057] Mask = Morphological_Operation(Mask_f)

[0058] where Morphological_Operation is the morphological operation operator.

[0059] Specifically, for different defect types, different image morphological operations can be adopted: for highly reflective materials, such as metal materials, the fluorescence area has edge collisions due to reflected light, and erosion operation is preferably used; for rough surface materials, such as ceramic materials, the rough surface leads to uneven fluorescence distribution, and local protrusions or isolated points may appear, and opening operation is preferably used; for transparent or translucent materials, such as glass and plastic materials, light may penetrate or scatter, resulting in blurred boundaries of the fluorescence area, and closing operation is preferably used.

[0060] Through the above steps, not only can the defects in the image be effectively located and segmented, but also the accuracy and stability of the segmentation result can be ensured; the defect segmentation result is as Figure 8 shown. This multi-level processing strategy enables accurate identification of defects in a complex and changing image environment, providing a solid foundation for subsequent image analysis and defect repair.

[0061] Furthermore, defect detection of complex parts is realized based on the defect annotation dataset, including:

[0062] The neural network is trained through the defect annotation dataset, specifically with visible light images as the input and defect mask labels as the output; the trained neural network is used as the defect detection model, and defect detection of complex parts is realized based on this defect detection model.

[0063] Defect detection of complex parts is realized based on a defect annotation dataset. Preferably, a semi-supervised anomaly detection method based on a teacher-student framework is adopted. The detection network uses two feature extraction models, namely: a teacher model and a student model. The teacher model is a feature extraction model pre-trained in a public dataset in advance, while the student model is an untrained model. In addition to the aforementioned defect annotation data (abnormal data), defect-free visible light images (normal data) also need to be collected. During the training process, the normal and abnormal data are respectively input into the teacher model and the student model, and the difference between the features output by the teacher model and the student model is used as the output of the detection network. The training is carried out with the goal of minimizing the feature difference of the normal data extracted by the two models and maximizing the feature difference of the abnormal data extracted by the two models, and only the parameters of the student model are updated. After the training is completed, a defect detection model is obtained. For any input visible light image to be detected, when the feature difference between the teacher model and the student model is greater than the set threshold, it is determined to be abnormal, and the defect detection model outputs the corresponding defect mask label.

[0064] A multi-view and multi-illumination data acquisition device provided by an embodiment of the present invention is used to implement the aforementioned image data acquisition method, such as Figure 1 shown, including a turntable 101, a visible light component, a fluorescence component, and a camera 401, where:

[0065] The turntable 101 is used to place the part 102 to be measured and adjust the angle. The turntable 101 is specifically an electric turntable that can rotate and pause at any angle along the central axis. The part 102 to be measured is any complex industrial part; the visible light component is used to provide different visible light illumination conditions, and the fluorescence component is used to provide fluorescence illumination; the camera 401 is used to obtain an image of the part 102 to be measured under visible light or fluorescence illumination.

[0066] Furthermore, as Figure 2 shown, the visible light component, the fluorescence component, and the camera are fixedly connected to form a multi-illumination data collector 205. The visible light component includes four strip-shaped light sources 301, 302, 303, 304 and a ring-shaped light source 305; the camera 401 is arranged at the center of the ring-shaped light source 305, and the four strip-shaped light sources are arranged around the outer periphery of the ring-shaped light source 305. The fluorescence component includes four fluorescent lamps 501, which are preferably installed between the strip-shaped light sources and the ring-shaped light source and are arranged around the ring-shaped light source. Each light source can arbitrarily control the illumination intensity, and the fluorescent lamp can emit ultraviolet light. By adjusting the on-off and illumination intensity of each light source, different illumination conditions are realized.

[0067] Furthermore, the device further includes a control panel 001 and an adjustment structure. The control panel 001 can be manually controlled by a person to start and emergency stop, and visually display the current state of the device. The adjustment structure is used to adjust the overall position and orientation of the multi-light data collector 205. The adjustment structure specifically includes a horizontal adjuster 201, a vertical adjuster 203, and a pitch adjuster 204. The horizontal adjuster 201 can adjust the horizontal distance between the multi-light data collector and the part to be measured. The vertical adjuster 203 can adjust the vertical distance between the multi-light data collector and the part to be measured. The pitch adjuster 204 can adjust the inclination angle of the multi-light data collector. In this embodiment, the pitch adjuster 204 is connected to the vertical adjuster 203, and the vertical adjuster 203 can move up and down along the support rod 202, and the support rod 202 is connected to the horizontal adjuster 201.

[0068] Figure 3 The specific process of data collection by the device is as follows: First, place the part to be measured and make a fluorescent mark on the defective position of the part. The turntable selects and stops at a set angle, and then the multi-light data collector sequentially changes different lighting conditions, and the camera collects data. Then the turntable moves to the next position and repeats the above process until all the data collection at the set angles is completed.

[0069] Specifically, the execution relationship between the turntable angle and the light source and the camera is as Figure 5 shown. There are m angles and n visible light lighting conditions set in advance. In actual collection, the turntable first returns to zero, and then adjusts to angle 1. Subsequently, the lighting conditions are sequentially changed to lighting 1, lighting 2 until lighting n. After each lighting modification, it is accompanied by data collection by the camera. After the n lighting conditions are executed, the visible light source will be turned off, the fluorescent light source will be turned on, and the camera will be collected once again. Then the turntable adjusts to angle 2 and repeats the above process until all the collections at angle m are completed. The data collected under multi-view and multi-lighting is as Figure 6 shown; the defective area of the part to be measured marked with fluorescence will emit light under the fluorescent lamp. The images with defects collected under visible light and fluorescence are as Figure 4 shown.

[0070] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A complex parts defect data labeling method, characterized in that: The steps include: The images of the part to be tested are collected from different angles, and the defective area of ​​the part to be tested is fluorescently marked in advance; for each angle, several visible light images under different visible light illumination conditions and one fluorescent image under fluorescent illumination conditions are collected; The fluorescent image is processed to segment the defect image from it, and the defect image is used as the label of the visible light image at the corresponding angle, thereby forming a defect annotation data set and realizing defect data annotation.

2. The complex parts defect data labeling method according to claim 1, characterized in that: The fluorescent image is processed to segment the defect image, including: The region of interest (ROI) is extracted from the fluorescence image, and then the ROI is segmented using a binarization method to obtain a mask Mask_f. The mask Mask_f is fine-tuned using image morphological operations to obtain a defect image.

3. The complex parts defect data labeling method according to claim 2, characterized in that: Extract the region of interest (ROI) from the fluorescence image, including: The pre-trained deep learning model SAM is used to perform coarse segmentation on the fluorescence image to obtain the mask Mask_r, and then the region of interest ROI is obtained.

4. The complex parts defect data labeling method according to claim 2, characterized in that: The ROI is segmented using a binary method, including: Calculate the mean of all pixel values ​​in the ROI, and use the mean as the binarization threshold to segment the ROI to obtain the mask Mask_f.

5. The complex parts defect data labeling method according to any one of claims 2 to 4, characterized in that: The image morphological operations include dilation, erosion, opening and closing operations.

6. A defect detection method based on the complex parts defect data labeling method according to any one of claims 1 to 5, characterized in that: The steps include: The neural network is trained using the defect annotation data set, and the trained neural network is used as a defect detection model; defect detection of complex parts is achieved based on the defect detection model.

7. A multi-view and multi-illumination data acquisition device for implementing the complex parts defect data labeling method according to any one of claims 1 to 5, characterized in that: It includes a turntable, a visible light component, a fluorescent component and a camera, wherein: The turntable is used to adjust the angle of the part to be tested; The visible light component is used to provide different visible light illumination conditions, and the fluorescent component is used to provide fluorescent illumination conditions; The camera is used to obtain an image of the part to be tested under visible light or fluorescent light.

8. The multi-viewing and multi-illumination data acquisition device according to claim 7, characterized in that: The visible light component includes four strip light sources and one annular light source. The camera is arranged at the center of the annular light source, and the four strip light sources surround the outer side of the annular light source.

9. The multi-viewing and multi-illumination data acquisition device according to claim 7, characterized in that: It also includes an adjustment structure, wherein the visible light component, the fluorescent component and the camera are fixedly connected to form a multi-illumination data collector; the adjustment structure is used to adjust the overall posture of the multi-illumination data collector.

10. The multi-view multi-illumination data acquisition device according to claim 9, characterized in that: The adjustment structure includes a horizontal adjuster, a vertical adjuster and a pitch adjuster, wherein the horizontal adjuster and the vertical adjuster are respectively used to adjust the horizontal distance and the vertical distance between the multi-lighting data collector and the part to be measured, and the pitch adjuster is used to adjust the inclination angle of the multi-lighting data collector.

Images