A method for annotating defect data of complex parts, a method for defect detection, and a multi-view, multi-light data acquisition device.

Abstract

This invention belongs to the field of defect detection and specifically discloses a method for annotating defect data of complex parts, a defect detection method, and a multi-view, multi-illumination data acquisition device. In the data acquisition device, a turntable provides multiple viewpoints of the part under test, and the light source design provides multiple illumination conditions for the camera, ensuring the acquisition of information about the complete surface image of the part under test. For defect data, to avoid manual annotation of image pixels, fluorescent markings are applied to the defect locations beforehand. Therefore, under fluorescent illumination conditions during the acquisition process, clear defect locations can be obtained. A three-level defect segmentation method is used to obtain high-precision pixel-level mask information of the defect locations in the image. This invention can realize data acquisition of complex parts under multiple viewpoints and illumination conditions, and can automatically annotate defect data.

Description

Technical Field

[0001] This invention belongs to the field of defect detection, and more specifically, relates to a method for annotating defect data of complex parts, a defect detection method, and a multi-view, multi-light data acquisition device. Background Technology

[0002] In today's industrial production, ensuring product quality is paramount, making the application of industrial vision inspection technology particularly important. Industrial vision inspection technology uses automated vision systems to inspect products, identifying defects, flaws, or non-conforming items to ensure product quality meets standards. In recent years, anomaly detection technology has been widely used in the surface defect quality inspection of industrial products. In industrial vision inspection, by analyzing product surface images, anomaly detection can identify various defects such as scratches, dents, and impurities, thus eliminating non-conforming products early in the production process.

[0003] However, due to the self-occlusion problem of complex parts, it is difficult to complete the inspection from a single viewpoint. Furthermore, the appearance and defects of a product can vary significantly under different viewing angles and lighting conditions. Many small defects in industry can only be observed under specific lighting and viewing angles. Therefore, multi-view, multi-lighting inspection can more comprehensively assess the product condition and reduce missed and false detections. In addition, industrial production environments are complex and variable; multi-angle and multi-lighting inspection can adapt to different production conditions, ensuring the stability and reliability of the inspection system. Currently, anomaly detection relies on training with large amounts of data, but multi-view, multi-lighting data is massive in volume, and manual annotation is extremely costly. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method for annotating defect data of complex parts, a defect detection method, and a multi-view multi-light data acquisition device. Its purpose is to realize the automated annotation of defect data of complex parts, thereby providing a large amount of data for defect detection training and improving the accuracy of defect detection.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for annotating defect data of complex parts is proposed, comprising the following steps:

[0006] Images of the part under test were acquired from different angles, and the defective areas of the part under test were pre-marked with fluorescence. For each angle, several visible light images under different visible light illumination conditions and one fluorescence image under fluorescence illumination conditions were acquired.

[0007] The fluorescence image is processed to segment out the defect image, and the defect image is used as a label for the visible light image at the corresponding angle, thereby forming a defect annotation dataset and realizing defect data annotation.

[0008] As a further preferred embodiment, the fluorescence image is processed to segment out the defect image, including:

[0009] The region of interest (ROI) is extracted from the fluorescence image, and then the ROI is segmented using a binarization method to obtain the mask_f. The mask_f is then fine-tuned using image morphology operations to obtain the defect image.

[0010] As a further preferred method, the region of interest (ROI) is extracted from the fluorescence image, including:

[0011] The pre-trained deep learning model SAM is used to coarsely segment the fluorescence image to obtain the mask Mask_r, and then the region of interest (ROI) is obtained.

[0012] As a further preferred method, a binarization method is used to segment the ROI, including:

[0013] Calculate the mean value of all pixel values ​​in the ROI, and use this mean value as a binarization threshold to segment the ROI to obtain the mask Mask_f.

[0014] As a further preferred embodiment, the image morphological operations include dilation, erosion, opening, and closing operations.

[0015] According to a second aspect of the present invention, a defect detection method based on the above-described complex part defect data annotation method is provided, characterized by comprising the following steps:

[0016] The neural network is trained using the defect annotation dataset, and the trained neural network is used as a defect detection model; based on this defect detection model, defect detection of complex parts is realized.

[0017] According to a third aspect of the present invention, a multi-view, multi-light data acquisition device is provided for implementing the above-described method for annotating defect data of complex parts, characterized in that it includes a turntable, a visible light component, a fluorescence component, and a camera, wherein:

[0018] The turntable is used to adjust the angle of the part to be tested.

[0019] The visible light component is used to provide different visible light illumination conditions, and the fluorescent component is used to provide fluorescent illumination conditions;

[0020] The camera is used to acquire images of the part under test under visible or fluorescent light.

[0021] As a further preferred embodiment, the visible light component includes four strip light sources and one ring light source, with the camera positioned at the center of the ring light source and the four strip light sources surrounding the outer perimeter of the ring light source.

[0022] As a further preferred embodiment, an adjustment structure is also included, wherein the visible light component, the fluorescence component, and the camera are fixedly connected to form a multi-illuminance data acquisition device; the adjustment structure is used to adjust the overall pose of the multi-illuminance data acquisition device.

[0023] As a further preferred embodiment, the adjustment structure includes a horizontal adjuster, a vertical adjuster, and a pitch adjuster, wherein the horizontal adjuster and the vertical adjuster are used to adjust the horizontal distance and vertical distance between the multi-light data acquisition device and the part to be measured, respectively, and the pitch adjuster is used to adjust the tilt angle of the multi-light data acquisition device.

[0024] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages:

[0025] 1. This invention performs data acquisition under multi-view and multi-light conditions. For defect data, fluorescent markings are applied to the defect locations in advance. Therefore, under fluorescent lighting conditions during the acquisition process, clear defect locations can be obtained. Thus, pixel-level automated annotation of all images with defective parts can be achieved without relying on manual annotation.

[0026] 2. This invention designs a three-level defect segmentation method, which can not only effectively locate and segment defects in images, but also ensure the accuracy and stability of the segmentation results, and obtain high-precision pixel-level mask information of the defect location in the image. This multi-level processing strategy enables accurate identification of defects even in complex and ever-changing image environments, providing a solid foundation for subsequent image analysis and defect repair.

[0027] 3. This invention designs a multi-view, multi-light data acquisition device. The turntable provides multiple viewpoints of the part under test, and the light source provides multiple lighting conditions for the camera, ensuring the acquisition of information of the complete surface image of the part under test. It can realize the acquisition of image data of the part under test from any angle and under multiple lighting conditions. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the structure of the multi-view, multi-illuminance data acquisition device according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the multi-light source structure according to an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of the part image data acquisition process according to an embodiment of the present invention;

[0031] Figure 4 Images of the parts under visible light and fluorescence according to embodiments of the present invention;

[0032] Figure 5 This is a flowchart illustrating the multi-view, multi-illuminance data acquisition process according to an embodiment of the present invention.

[0033] Figure 6 These are multi-view, multi-lighting component images from embodiments of the present invention;

[0034] Figure 7 This is a flowchart illustrating the defect labeling process according to an embodiment of the present invention.

[0035] Figure 8 This is a diagram showing the defect segmentation results of an embodiment of the present invention.

[0036] In all the accompanying drawings, the same reference numerals are used to denote the same elements or structures, wherein: 001-control panel, 101-turntable, 102-part to be measured, 201-level adjuster, 202-support rod, 203-vertical adjuster, 204-pitch adjuster, 205-multi-light data acquisition device, 301, 302, 303, 304-strip light source, 305-ring light source, 401-camera, 501-fluorescent lamp. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0038] An embodiment of the present invention provides a method for annotating defect data of complex parts, comprising the following steps:

[0039] S1. Pre-mark the defective areas of the parts with fluorescent markers; acquire images of the parts after fluorescent marking from different angles, including several visible light images under different visible light illumination conditions and one fluorescent image under a fluorescent light source for each angle.

[0040] S2. Process each fluorescence image separately to segment out the defect images; use the defect images as labels for the visible light images at the corresponding angles to form a defect annotation dataset and achieve defect data annotation.

[0041] Furthermore, in order to effectively identify and locate defects in the image, a step-by-step, multi-level processing strategy was adopted when processing the fluorescence image, such as... Figure 7 As shown, it specifically includes:

[0042] (1) First, focus on the preliminary coarse segmentation of the defect area. The purpose of this step is to quickly locate the approximate location of the defect and lay the foundation for subsequent fine processing.

[0043] In this stage, a pre-trained deep learning model, SAM (Segment Anything Model), is used for coarse segmentation of the input fluorescence image I. The SAM model is trained on large-scale image data and is capable of segmenting various objects in the image, performing well even with complex backgrounds and diverse shapes.

[0044] Mask_r = SAM(I)

[0045] Mask_r is a coarse segmentation mask predicted by the SAM model.

[0046] The obtained coarse segmentation mask Mask_r is processed to extract the region of interest (ROI):

[0047] ROI = Mask_r & I

[0048] The purpose of extracting the Region of Interest (ROI) is to reduce the computational load of subsequent processing and focus on the defect itself, thereby improving the accuracy of segmentation. The ROI extraction process is actually a pixel-by-pixel AND operation between the original image and the coarse segmentation mask to obtain a sub-image containing only the defect region.

[0049] (2) After the ROI is extracted, binarization is used to perform fine segmentation of the ROI. Binarization is a method to convert an image into an image containing only two colors, usually black and white, which can more clearly define the boundaries of defects. This invention calculates the pixel mean within the defect area and uses the mean as the binarization threshold Th to perform fine segmentation:

[0050] Th = Average(ROI)

[0051] Here, Average is the mean calculation function, which obtains the mean value of all pixels in the ROI.

[0052] The process of binarization to obtain the fine segmentation mask Mask_f is as follows:

[0053] Mask_f ij =1ifROI ij >Th

[0054] Mask_f ij =0if ROI ij ≤Th

[0055] Where i and j are the subscript indices of the image.

[0056] (3) Finally, to further improve the segmentation accuracy, image morphological operations are used to fine-tune the mask Mask_f obtained from the refined segmentation. Image morphological operations include dilation, erosion, opening, and closing operations, which are a series of transformations based on set theory used to change the shape of objects in an image. The purpose of these operations is to remove noise, smooth edges, and fill small holes in defective regions, thereby obtaining more accurate segmentation results. Morphological operations can be represented as:

[0057] Mask=Morphological_Operation(Mask_f)

[0058] Here, Morphological_Operation is the morphological operation operator.

[0059] Specifically, different image morphology operations can be used for different defect types: for highly reflective materials, such as metals, the fluorescent area causes edge collisions due to reflected light, so the etching operation is preferred; for rough surface materials, such as ceramics, the surface roughness causes uneven fluorescence distribution, which may result in local protrusions or isolated points, so the opening operation is preferred; for transparent or translucent materials, such as glass and plastics, light may penetrate or scatter, causing the boundaries of the fluorescent area to be blurred, so the closing operation is preferred.

[0060] Through the above steps, not only can defects in the image be effectively located and segmented, but the accuracy and stability of the segmentation results can also be ensured; the defect segmentation results are as follows: Figure 8 As shown, this multi-layered processing strategy enables accurate identification of defects even in complex and ever-changing image environments, providing a solid foundation for subsequent image analysis and defect repair.

[0061] Furthermore, defect detection of complex parts is achieved based on defect annotation datasets, including:

[0062] The neural network is trained using the defect labeling dataset, specifically with visible light images as input and defect mask labels as output; the trained neural network is used as a defect detection model, and defect detection of complex parts is achieved based on this defect detection model.

[0063] To achieve defect detection of complex parts based on a defect-annotated dataset, a semi-supervised anomaly detection method using a teacher-student framework can be preferred. The detection network employs two feature extraction models: a teacher model and a student model. The teacher model is a pre-trained feature extraction model on a public dataset, while the student model is untrained. In addition to the aforementioned defect-annotated data (anomaly data), defect-free visible light images (normal data) also need to be collected. During training, normal and anomaly data are input into the teacher and student models respectively, and the difference in features output by the teacher and student models is used as the output of the detection network. The training objective is to minimize the feature difference between the two models extracting normal data and maximize the feature difference between the two models extracting anomaly data, updating only the parameters of the student model. After training, the defect detection model is obtained. For any input visible light image to be detected, if the feature difference between the teacher and student models exceeds a set threshold, it is determined to be an anomaly, and the defect detection model outputs the corresponding defect mask label.

[0064] This invention provides a multi-view, multi-illuminance data acquisition device for implementing the aforementioned image data acquisition method, such as... Figure 1 As shown, it includes a turntable 101, a visible light component, a fluorescence component, and a camera 401, wherein:

[0065] The turntable 101 is used to place the part to be tested 102 and adjust its angle. Specifically, the turntable 101 is an electric turntable that can rotate and pause at any angle along the central axis. The part to be tested 102 is any complex industrial part. The visible light component is used to provide different visible light illumination conditions, and the fluorescent component is used to provide fluorescent illumination. The camera 401 is used to acquire images of the part to be tested 102 under visible light or fluorescent illumination.

[0066] Furthermore, such as Figure 2 As shown, a visible light component, a fluorescent component, and a camera are fixedly connected to form a multi-light data acquisition unit 205. The visible light component includes four strip light sources 301, 302, 303, and 304, and a ring light source 305; the camera 401 is positioned at the center of the ring light source 305, with the four strip light sources surrounding the outer perimeter of the ring light source 305. The fluorescent component includes four fluorescent lamps 501, which are preferably installed between the strip light sources and the ring light source, and arranged around the ring light source. The light intensity of each light source can be arbitrarily controlled, and the fluorescent lamps emit ultraviolet light. Different lighting conditions can be achieved by adjusting the brightness and intensity of each light source.

[0067] Furthermore, the device also includes a control panel 001 and an adjustment structure. The control panel 001 can be manually controlled for start-up and emergency stop, and visually displays the current status of the device. The adjustment structure is used to adjust the overall pose of the multi-light data acquisition unit 205. Specifically, the adjustment structure includes a horizontal adjuster 201, a vertical adjuster 203, and a pitch adjuster 204. The horizontal adjuster 201 can adjust the horizontal distance between the multi-light data acquisition unit and the part under test, the vertical adjuster 203 can adjust the vertical distance between the multi-light data acquisition unit and the part under test, and the pitch adjuster 204 can adjust the tilt angle of the multi-light data acquisition unit. In this embodiment, the pitch adjuster 204 is connected to the vertical adjuster 203, and the vertical adjuster 203 can move up and down along the support rod 202, which is connected to the horizontal adjuster 201.

[0068] Figure 3 The specific process for data acquisition by the device is as follows: First, place the part to be tested and mark the defect location of the part with fluorescent markers; the turntable selects and stops at the set angle, and then the multi-light data acquisition unit changes different lighting conditions in sequence, the camera collects data, and then the turntable moves to the next position and repeats the above process until data acquisition of all set angles is completed.

[0069] Specifically, the relationship between the turntable angle and the execution of the light source and camera is as follows: Figure 5 As shown, m angles and n visible light illumination conditions are preset. In actual data acquisition, the turntable first returns to zero, then adjusts to angle 1. Subsequently, the illumination conditions are sequentially changed to illumination 1, illumination 2, and so on up to illumination n. Each illumination change is accompanied by camera data acquisition. After n illumination conditions are executed, the visible light source is turned off, the fluorescent light source is turned on, and camera acquisition is performed again. Then the turntable is adjusted to angle 2, and the above process is repeated until all acquisitions at angle m are completed. The data acquired under multiple views and multiple illumination conditions is shown below. Figure 6 As shown; the defective area of ​​the part under test is marked with fluorescent light, which will glow under a fluorescent lamp. Images of the defect acquired under visible light and fluorescence are shown below. Figure 4 As shown.

[0070] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for annotating defect data of complex parts using a multi-view, multi-light data acquisition device, characterized in that, The multi-view, multi-light data acquisition device includes a turntable, a visible light component, a fluorescence component, and a camera, wherein: the turntable is used to adjust the angle of the part under test; the visible light component is used to provide different visible light illumination conditions; the fluorescence component is used to provide fluorescence illumination conditions; and the camera is used to acquire images of the part under test under visible light or fluorescence illumination. The method includes the following steps: Images of the part under test were acquired from different angles, and the defective areas of the part under test were pre-marked with fluorescence. For each angle, several visible light images under different visible light illumination conditions and one fluorescence image under fluorescence illumination conditions were acquired. The fluorescence image is processed to segment out the defect image, and the defect image is used as a label for the visible light image at the corresponding angle, thereby forming a defect annotation dataset and realizing defect data annotation. The fluorescence image is processed to segment out the defect image, including: using a pre-trained deep learning model SAM to coarsely segment the fluorescence image to obtain a mask Mask_r, and then obtaining the region of interest (ROI); then using a binarization method to segment the ROI to obtain a mask Mask_f; and then using image morphology operations to fine-tune the mask Mask_f to obtain the defect image: for metal materials, the erosion operation is selected; for ceramic materials, the opening operation is selected; and for glass and plastic materials, the closing operation is selected. The ROI is segmented using a binarization method, which includes: calculating the mean value of all pixel values ​​in the ROI, and using this mean value as a binarization threshold to segment the ROI, thereby obtaining the mask Mask_f.

2. The method for annotating defect data of complex parts using a multi-view, multi-light data acquisition device according to claim 1, characterized in that, The visible light component includes four strip light sources and one ring light source. The camera is located at the center of the ring light source, and the four strip light sources surround the outer perimeter of the ring light source.

3. The method for annotating defect data of complex parts using a multi-view, multi-light data acquisition device according to claim 2, characterized in that, It also includes an adjustment structure, wherein the visible light component, the fluorescence component and the camera are fixedly connected to form a multi-illuminance data acquisition device; the adjustment structure is used to adjust the overall pose of the multi-illuminance data acquisition device.

4. The method for annotating defect data of complex parts using a multi-view, multi-light data acquisition device according to claim 3, characterized in that, The adjustment structure includes a horizontal adjuster, a vertical adjuster, and a pitch adjuster. The horizontal and vertical adjusters are used to adjust the horizontal and vertical distances between the multi-light data acquisition device and the part under test, respectively, and the pitch adjuster is used to adjust the tilt angle of the multi-light data acquisition device.

5. A defect detection method based on the method for annotating defect data of complex parts using a multi-view, multi-light data acquisition device as described in any one of claims 1 to 4, characterized in that, Includes the following steps: The neural network is trained using the defect annotation dataset, and the trained neural network is used as a defect detection model; based on this defect detection model, defect detection of complex parts is realized.

Images